This page looks best with JavaScript enabled

Design of Kpt for microservice architecture Part 1 - Basics & Architecture

· ☕ 8 min read
🏷️
  • #kubernetes
  • #kpt
  • Hi guys. This is my first post of “Design of Kpt for microservice architecture”.

    Google has annouced Kpt, an OSS tool for building declarative workflows on top of resource configuration. Although, many features are under development, the design & concepts looked interesting.

    Therefore, I dived into Kpt and started to design how I can adopt this concepts in pratice. This is how my story began.

    What you will learn

    In the series of the post, you will learn these topics:

    1. Basics of Kpt and how to use in microservice architecture
    2. Developing Kpt functions
    3. Designing CI/CD deployment workflow with other Cloud Native technologies

    What’s kpt?

    As explained in the official document,

    kpt is a toolkit to help you manage, manipulate, customize, and apply Kubernetes Resource configuration data files.

    kpt is based on “Configuration as Data” principle which is a new concept compared to Kubernetes.

    Demo

    Here, I will describe it by in each usecases.

    Basics

    1. Publish a package

    As explained in the previous section, you will need an upstream configuration data.

    First, create an initial file by the command below:

    $ kpt pkg init <DIRNAME>
    

    Here, I will name my <DIRNAME> as service name which is navy. So the command will be like below:

    $ kpt pkg init navy
    writing navy/Kptfile
    writing navy/README.md
    

    Each directory will be your atomic configuration data which will contain metadata.

    And then release it as usual package.

    $ git add .
    $ git commit -m "initial commit for navy"
    $ git tag navy/v0.1.0
    $ git push origin navy/v0.1.0
    

    Note that the tag naming convention is <DIR>/<VERSION>. This is because the these <DIR> contained tags will be used for per-directory versioning. See details later in this post.

    2. Get upstream remote package

    This step is for package consumers.

    Fetch the navy blueprint by the command below:

    $ kpt pkg get git@github.com:KeisukeYamashita/kpt-blueprint.git@navy/v0.1.0 k8s
    

    Now, we have a directory structure below.

    ./k8s
    └── kpt-blueprint
        ├── Kptfile
        ├── README.md
        └── navy
            ├── README.md
            └── deployment.yaml
    

    The path will be <REPO_NAME>/<DIRNAME> inside the second argument you passed for kpt pkg get command.
    Usally, there will be other packages for application in the root directory. That’s why I named the directory k8s.

    In the k8s/kpt-blueprint/Kptfile, you will see that the upstream file is configured in .upstream.

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    
    apiVersion: kpt.dev/v1alpha1
    kind: Kptfile
    metadata:
        name: kpt-blueprint
    upstream:
        type: git
        git:
            commit: 6e8de26d8646e37f90170e039b537f18058d4fb8
            repo: git@github.com:KeisukeYamashita/kpt-blueprint
            directory: /
            ref: navy/v0.1.0
    

    Then you can apply it to as usual.

    $ kubectl apply -R -f k8s
    

    Don’t forget the -R to recursively apply the resources.

    Updating the blueprint and deployment

    Kpt takes an Merge solution.

    As described in the figure, there are two usecases for this.

    1. Team Copy has updated(Day 1)
    2. Upstream blueprint is updated(Day 2)

    Update the team copy

    I updated the spec.replicas from 1 to 3.

    1
    2
    3
    
    spec:
    -    replicas: 1
    +    replicas: 3
    

    Then apply your changes. This will be usual Kubernetes deployment workflow.

    $ kuebctl apply -R -f k8s
    

    Update upstream blueprint

    Next, let’s update the upstream remote blueprint. I will update the image tag in the .spec.template.spec.containers[0].image.

    1
    2
    3
    
    containers:
    - - image: nginx
    + - image: nginx:1.18.0
    

    Then release a new tag.

    $ git tag navy/v0.1.1
    $ git push origin navy/v0.1.1
    

    Update team package with upstream blueprint

    Run the command below:

    $ kpt pkg update k8s/kpt-blueprint@navy/v0.1.1
    ...
    Error: local package files have been modified: [navy/deployment.yaml].
      use a different update --strategy.
    

    Because I have updated the team package, it ran into error. The correct command will be this:

    $ kpt pkg update k8s/kpt-blueprint@navy/v0.1.1 --strategy=resource-merge
    

    Now I can see the update by git diff.

    1
    2
    3
    4
    5
    6
    7
    8
    9
    
     upstream:
         type: git
         git:
    -        commit: 6e8de26d8646e37f90170e039b537f18058d4fb8
    +        commit: 6aaad42af50e826f666d5ac7dc5c4bc6e86ed59e
             repo: git@github.com:KeisukeYamashita/kpt-blueprint
             directory: /
    -        ref: navy/v0.1.0
    +        ref: navy/v0.1.1
    
    1
    2
    3
    4
    5
    6
    7
    8
    9
    
    --- a/k8s/kpt-blueprint/navy/deployment.yaml
    +++ b/k8s/kpt-blueprint/navy/deployment.yaml
    @@ -16,5 +16,5 @@ spec:
             app: "nginx"
         spec:
           containers:
    -      - image: nginx
    +      - image: nginx:1.18.0
             name: nginx
    

    There are four types of strategy. You can test it by yourself because there are --dry-run option that you can use for debugging the behavours of each strategy.

    1. Resource Merge: Preform a structural comparision of the original

    This stategy is well-knowned by three way merge.
    It will update the team package by 3-way merging the …

    1. Original upstream commit
    2. team pcakage
    3. new upstream reference

    Think of it as same as kubectl apply.

    2. Fast forward: Like git’s fast forward(default)

    This is the default stategy. It will fail if the team’s package has been updated.
    Here, I will show the example of fast forward.

                       A <- upstream package(tag:navy/v0.1.0)
                      /
                     /
    D --- E ---F ---G(tag:navy/v0.1.0)
    

    B is the update release of the upstream package.

                       A ---B <- upstream package(tag:navy/v0.1.1)
                      /
                     /
    D --- E ---F ---G(tag:navy/v0.1.0) <- team package
    

    After merging it will be like this. H is the new

                                             B <- upstream package(tag:navy/v0.1.1)
                                            /
                                           /
    D --- E ---F ---G(tag:navy/v0.1.0) ---H(tag:navy/v0.1.1)
    

    If you have update in the team package like below…

                       A ---B <- upstream package(tag:navy/v0.1.1)
                      /
                     /
    D --- E ---F ---G(tag:navy/v0.1.0) ---H(tag:navy/v0.1.0) <- team package
    

    You will fail to mege because it is not a fast merge anymore. You have to decide how are you going to merge the B and H. Thats why there are other stategy options.

    3. Alpha Git path

    This stategy uses git format-patch and git am to appty the patches.

    team package  -->|patch| upstream remote package
    

    Note that this patch is send to team’s copy of the upstream remote package. This means that not PR will be created.

    I will bring the upstream’s remote commit to your team’s git.

    4. Force delete replace

    This may destroy your team’s changes.

                       A ---B <- upstream package(tag:navy/v0.1.1)
                      /
                     /
    D --- E ---F ---G(tag:navy/v0.1.0) ---H(tag:navy/v0.1.0) <- team package
    

    If you have update in the team package like below…

                       B <- upstream package(tag:navy/v0.1.1)
                      /
                     /
    D --- E ---F ---G(tag:navy/v0.1.1) <- team package
    

    Setters: Template-free setting

    Setters are API between the resources and the CLI.
    If you configured Setters, you will be able to configure variables manually.

    Setters are Key-Value fields that teams can configure by CLI.

    1. Configure Setters

    Let’s add these fields in Kptfile:

    1
    2
    3
    4
    5
    6
    7
    
    + openAPI:
    +  definitions:
    +    io.k8s.cli.setters.replicas:
    +     x-k8s-cli:
    +        setter:
    +          name: "replicas"
    +          value: "3"
    

    Or you can use the

    $ kpt cfg create-setter navy/ replicas 3
    

    Then, add these field in the resources.

    1
    2
    3
    4
    5
    6
    7
    
    spec:
    - replicas: 1
    + replicas: 1 # {"$ref":"#/definitions/io.k8s.cli.setters.replicas"}
    ``
    
    Then publish a new release.
    
    

    $ git tag navy/@v0.1.2
    $ git push origin navy/@v0.1.2

    
    You can now see the list of the Setters.
    
    ```console
    $ kpt cfg list-setters navy/ 
        NAME     VALUE   SET BY   DESCRIPTION   COUNT  
      replicas   3
    

    2. Configure variables

    To use the setters, you have to fetch the new release. This is an fast-forward merge, therefore, you don’t need to specify the strategy.

    $ kpt pkg update k8s/kpt-blueprint@navy/v0.1.2
    

    You can see the list of setters by executing the command below:

    $ kpt cfg list-setters k8s/kpt-blueprint/navy 
        NAME     VALUE   SET BY   DESCRIPTION   COUNT  
      replicas   3                              1   
    

    Then, configure your setters.

    $ kpt cfg set k8s/kpt-blueprint/navy replicas 5
    

    You can add --set-by "keke" and --description on your patch to store the histories.

    Now you learnt the basics of Kpt. Let’s go on to design the architecture.


    Architecture Design

    There are common architecture to store your configuration.

    1. Each application seperate instance

    I guess Kpt is designed for this architecture becasue Kpt uses git commits, tags, refs(branches) for specifing the upstream package as you saw in this post before.

    Things to consider:

    • Sync between the upstream package and the team instance. How should you design the sync mechanism?

    2. All-in-one instance

    Things to consider:

    • Merge battle. For big companies, there will be Update branch in many PRs.
    • Access control.

    Ecosystem & Comparision

    1. Helm

    Helm is a tool for managing Charts.

    Helm can be used for templating the Kpt package but IMO, that’s it. Don’t try to distribute any Kubernetes resource with Helm chart unless your company is GAFA or Uber, Lyft or whatever big company.
    Helm will only abstract the Kubernetes resources with variables and you should always consider if that worth managing the Helm components and the charts.

    2. Kustomize

    Kustomize lets you customize raw, template-free YAML files for multiple purposes, leaving the original YAML untouched and usable as is.

    IMO, Kpt is mainly for distributing the resoures with several API likes Setter, Substitutions by “packages” and configure the data in each team’s package.
    In that stage, Kustomize can be uses in two spaces:

    1. When creating a package
    2. Building in the team’s package

    There will be no benifit to creat a package by kusotmize and you should package the kustomize configurations.

    Next

    In this post, I explained the basics of Kpt and designed an architecture. In the next post, I am planning to explain the advanced features of Kpt.

    See you soon.

    Futher readings

    Share on

    Keisuke Yamashita
    WRITTEN BY
    Keisuke Yamashita
    Site Reliability Engineer