GitOps with Flux and Kustomize

On Windows 10 with minikube & different namespaces

So far, we have covered GitOps in theory. Now, we are going to put GitOps into practice using Weaveworks Flux and Kustomize.

We are going to set up our hello-world example so that every time you push a change to a particular branch in your git repository, those changes will sync to Kubernetes, or in our case minikube, and ensure your application is in the state that you want it to be in. We are going to use Kustomize to highlight how we could deploy this service to different namespaces with different configuration values.

A quick recap of what we will be doing.

1. You will need to select a GitHub repository of an application or service you want to sync. You will need access to save a deploy key in GitHub for that repo. My repo as an example: https://github.com/airwavetechio/hello-world
2. We are going to install & deploy Flux into a minikube cluster. This will point to your GitHub repository. You should have minikube up and running before starting this exercise.
3. Solidify the connection from Flux to the repository by adding a deploy key that is created after you install flux in your cluster.
4. Sync your repo with your cluster and watch Kustomize take care of applying those changes.

2. Install & Deploy Flux

We are going to skip step 1 and jump right into step 2.

Flux is an open-source tool created by Weaveworks. It gets deployed to your cluster and is responsible for watching/syncing changes from GitHub to your cluster.

When we install flux, we add command line parameters to properly configure it.

First, let’s install fluxctlon our local machine. Much like kubectl, this is your command-line tool to administer flux.

choco install fluxctl

for other platforms, follow https://docs.fluxcd.io/en/1.18.0/references/fluxctl.html#

Let’s create a namespace for flux.

kubectl create ns flux

Let’s install flux on your cluster and configure it.

fluxctl install --manifest-generation=true --git-user=<a git user> --git-email=<email used for git> --git-url=git@github.com:org/<repo> --git-branch <branch> --git-path=<path to your files, additional path> --namespace=<namespace> | kubectl apply -f -

--manifest-generation=true is going to allow us to leverage a bit of templating so we can reuse a base set of YAML files for Kubernetes and patch the changes with Kustomize. This will help us when deploying specific app configurations for each namespace.

--git_user=<a git user> --git-email=<email used for git> --git-url=git@github.com:org/<repo> --git-branch <branch>is the configuration for your git repo that you want to sync.

--git-path=<path to your files, additional path> is telling flux where to look for particular files after the repository is cloned (inside of the cluster). I will get into more detail about the path after this section.

--namespace=flux tells fluxctlwhere to install flux on Kubernetes.

| kubectl apply -f - this pipes the output of the fluxctlcommand and applies that output to a kubectl apply command. (Don’t forget | ).

fluxctl output

kubectl get pods -n flux

An explanation of git-path

--git-path=flux/releases/namespaces,flux/releases/airwave-stage

When flux is working, it will clone the specified git repo inside the cluster, and then try to run Kustomize to generate Kubernetes manifest files based on what is located inside the--git-path . Kustomize will then apply those manifests after they have generated.

In our case here, we have specified 2 locations for Kustomize to look, separated by a comma.

flux\releases\namespaces

and

flux\releases\airwave-stage

Let’s look at our directory structure. We have a baseset of files, along with a directory for each namespace, along with a namespaces directory. airwave-deploy will not be used in this example.

directory structure

In our case, flux\releases is our root dir. At that level of the tree, we have.flux.yaml . This is required by flux if you want to use a generator, or else flux will apply YAML files just as kubectl would. The flux\releases parent directory is something I just made up to house the files and can be whatever you want.

.flux.yaml

.flux.yamltells flux how it will generate files. You want to drop this at the root of where your synced files will live. That’s because, by default, flux will look in the directory that is specified in--git-path and the directory above it.

flux\releases\namespaces

I have added namespaces as an example of how to setup namespace the #GitOps way.

By default, Kustomize looks for the file name kustomization.yaml . In our --git-path we specified flux\releases\namespaces , so it will use the kustomization.yaml file in that directory.

namespaces\kustomization.yaml tells Kustomize which files to apply.

namespace\kustomization.yaml

namespace\airwave-stage.yaml is a YAML file you could also directly apply to Kubernetes to create a namespace. Remember, flux and Kustomize are syncing the files between GitHub and the cluster for you and applying them automatically, so any YAML files you are using now to configure your services can probably be reused.

namespaces\airwave-stage.yaml

flux\releases\base\

base contains all the base files. In this case, our deployment, service, and configmap YAML files. The files mentioned below can be found in our hello-word repo.

base\kustomization.yaml lets Kustomize know which files to apply.

base\kustomization.yaml

In our base\deployment.yaml file, we are NOT specifying a namespace.

excerpt of base\deployment.yaml

base\service.yaml also does NOT specify a namespace, but specifies certain ports to listen on.

base\service.yaml

base\configmap.yaml specifies a default AIRWAVE_HW_PORT of 4997 that the deployment will look for.

base\configmap.yaml

But what does basehave to do with our --git-path ? Let’s find out.

flux\releases\airwave-stage

In this use case, we want to deploy hello-world to the airwave-stagenamespace.

airwave-stage’s deployment needs to use port 4998, not 4997 as configured in the base. Our service then needs to serve on port 5000 and not 4997 as originally configured.

a visual assistant

airwave-stage\kustomization.yaml again tells Kustomize to use the relative path to the base directory. It directs flux to install this in the airwave-stage namespace, and then patch the whatever matches kind: name: . It then uses the path: to specify where the patch file is.

airwave-stage\kustomization.yaml

airwave-stage\configmap.yaml is a patch file. In this case, it looks almost identical to base\configmap.yaml except for the port. Because it’s a direct match (and short) with the original file, it will just overwrite it.

airwave-stage\configmap.yaml

airwave-stage\service.yaml is also a patch file, but it’s a little different. Because we need to replace something inline, it wouldn’t make sense to just copy, paste, and make our changes from the original to this file. Here we add $patch: delete to the fields we want to delete, and then follow it with the new values.

airwave-stage\service.yaml

I’m currently not too sure about the state of $patch because Kustomize v2.03 and v3 are floating around and that makes things a bit confusing. If you find more information on this, let me know but for now, this works.

You should be able to test all of this locally by using kustomize build <directory> but with these version mismatches, but at the time for writing this article, it doesn’t currently work with v2.0.3 on Windows 10 Pro.

Hopefully, this gives you a better idea of how --git-path works. Now back to the action.

3. Solidify the connection from Flux to the repository by adding a deploy key

fluxctl identity --k8s-fwd-ns flux will output an ssh public key that is generated by the Kubernetes cluster. The cluster itself has the private key.

Copy that ssh-key and paste it into your GitHub Repo > Settings > Deploy keys

Allow write access

4. Sync your repo with your cluster

fluxctl --k8s-fwd-ns flux sync to kick off the process. You can also wait for 5 minutes.

the git commit hash should match your latest commit in GitHub

If you did everything right, you will see your changes in the proper area. Let’s look at our diagram again to refresh our memory.

The state of things

The deployments works.

kubectl get pods -n airwave-stage

The configmap is showing port 4998 instead of 4997

kubectl describe configmap hello-world-configmap -n airwave-stage

Our service is now listening on port 5000 instead of 4997

kubectl describe service airwavetech-helloworld-svc -n airwave-stage

Conclusion

In this exercise, we have taken our hello-world service, configured flux and Kustomize in our minikube cluster to sync with our GitHub repo, and have watched an automated deployment of that service to a specific namespace with a specific configuration. Welcome to the future. “This is heavy.”

If you learned something new in this article, don’t forget to clap and follow!

Until next time.