A Taste of Argocd

02/20/2021 argocd k8s iac

What is ArgoCD

ArgoCD is an application that syncs what is in a git repo into kubernetes. This is also known as GitOps. The idea is that ArgoCD will know what is in the git repo compared to what is currently in the cluster. This differs from say a CI/CD pipeline that just runs kubectl apply -f ./manifests in that after the apply you no longer know what the state of the cluster is. Argocd keeps track of the state of the cluster or its “apps”.

Apps of Apps

ArgoCD has a nice model called “Apps of Apps”. ArgoCD provides a CRD of kind “Application”. This Application defines where where git repo it is going to sync from is the name of the Appliction what type of manifests it is (helm, kustomize, jsonnet), and syncing options. The basic idea is that you have an Application that looks back at the directory that contains more Applications. Then we are able to configure automatic syncing of new Applications and when Applications.

Below we can see the tree view of what an “Apps of Apps” directory looks like. This example is using it as a helm chart but it could also just be a directory with the manifests in it. The the argocd-apps.yaml reference back to the directory argocd-apps.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


argocd-apps
├── Chart.yaml
└── templates
    ├── argocd-apps.yaml
    ├── azurekv-csi.yaml
    ├── cert-issuers.yaml
    ├── coredns.yaml
    ├── ingress-external.yaml
    ├── ingress-internal.yaml
    ├── kube-state-metrics.yaml

Below is the app that is going to sync all of the other apps we want back. This is essentially updating itself where itself contains itself and other apps. Comments are from the defautl ArgoCD manifest for Application.

The things to notice here are the finalizer is commented out because it will never delete this app otherwise. There have been cases where you want to delete all of the apps. The source.path is the path in the git repo to this app. The destination.namespace is going to argocd. This is also the namespace that we put ArgoCD in. This is what is recommended by the docs but you could put it in a different namespace if you wanted to. The syncPolicy block defines if we want it to automatically sync and since this is the app syncing other apps back we do.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40


apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: argocd-apps
  # You'll usually want to add your resources to the argocd namespace.
  namespace: argocd
  # Add a this finalizer ONLY if you want these to cascade delete.
  #finalizers:
  #  - resources-finalizer.argocd.argoproj.io
spec:
  # The project the application belongs to.
  project: default

  # Source of the application manifests
  source:
    path: 'argocd-apps'
    targetRevision: HEAD
    repoURL: 'git@site:/argocd'

  # Destination cluster and namespace to deploy the application
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd

  # Sync policy
  syncPolicy:
    automated: # automated sync by default retries failed attempts 5 times with following delays between attempts ( 5s, 10s, 20s, 40s, 80s ); retry controlled using `retry` field.
      prune: true # Specifies if resources should be pruned during auto-syncing ( false by default ).
      selfHeal: true # Specifies if partial app sync should be executed when resources are changed only in target Kubernetes cluster and no git change detected ( false by default ).
      allowEmpty: false # Allows deleting all application resources during automatic syncing ( false by default ).
    syncOptions: # Sync options which modifies sync behavior
      - Validate=true # disables resource validation (equivalent to 'kubectl apply --validate=false') ( true by default ).
      - CreateNamespace=true # Namespace Auto-Creation ensures that namespace specified as the application destination exists in the destination cluster.
    # The retry feature is available since v1.7
    retry:
      limit: 5 # number of failed sync attempt retries; unlimited number of attempts if less than 0
      backoff:
        duration: 5s # the amount to back off. Default unit is seconds, but could also be a duration (e.g. "2m", "1h")
        factor: 2 # a factor to multiply the base duration after each failed retry
        maxDuration: 3m # the maximum amount of time allowed for the backoff strategy

Here is an exmaple of an app inside the argocd-apps directory. This deploying nginx-ingress via a helm chart dependency. The main difference are the name of the app, the finalizer isn’t commented out, the path has changed, and the destination.namespace has chagned.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40


apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: ingress-external
  # You'll usually want to add your resources to the argocd namespace.
  namespace: argocd
  # Add a this finalizer ONLY if you want these to cascade delete.
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  # The project the application belongs to.
  project: default

  # Source of the application manifests
  source:
    path: 'ingress-external'
    targetRevision: HEAD
    repoURL: 'git@site:/argocd'

  # Destination cluster and namespace to deploy the application
  destination:
    server: https://kubernetes.default.svc
    namespace: infrastructure

  # Sync policy
  syncPolicy:
    automated: # automated sync by default retries failed attempts 5 times with following delays between attempts ( 5s, 10s, 20s, 40s, 80s ); retry controlled using `retry` field.
      prune: true # Specifies if resources should be pruned during auto-syncing ( false by default ).
      selfHeal: true # Specifies if partial app sync should be executed when resources are changed only in target Kubernetes cluster and no git change detected ( false by default ).
      allowEmpty: false # Allows deleting all application resources during automatic syncing ( false by default ).
    syncOptions: # Sync options which modifies sync behavior
      - Validate=true # disables resource validation (equivalent to 'kubectl apply --validate=false') ( true by default ).
      - CreateNamespace=true # Namespace Auto-Creation ensures that namespace specified as the application destination exists in the destination cluster.
    # The retry feature is available since v1.7
    retry:
      limit: 5 # number of failed sync attempt retries; unlimited number of attempts if less than 0
      backoff:
        duration: 5s # the amount to back off. Default unit is seconds, but could also be a duration (e.g. "2m", "1h")
        factor: 2 # a factor to multiply the base duration after each failed retry
        maxDuration: 3m # the maximum amount of time allowed for the backoff strategy

An Application that uses helm values that differ between environments can be shown below. This is deploying externaldns with two values files for its helm chart. One of the values files contains values that do not change between environments. The second values file values-dev.yaml contains values that are specific to only the dev environment. We can have as many values files as we want here.

Notice that values.yaml is explicitly listed here. Helm will automatically use this values.yaml file even if it is not listed. It is listed here in ArgoCD to be explicit that it is being used.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45


apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: externaldns
  # You'll usually want to add your resources to the argocd namespace.
  namespace: argocd
  # Add a this finalizer ONLY if you want these to cascade delete.
  #finalizers:
  #  - resources-finalizer.argocd.argoproj.io
spec:
  # The project the application belongs to.
  project: default

  # Source of the application manifests
  source:
    path: 'externaldns'
    targetRevision: HEAD
    repoURL: 'git@ssh.dev.azure.com:v3/bhginc/BHG-IT/argocd'

    helm:
      valueFiles:
        - values.yaml
        - values-dev.yaml

  # Destination cluster and namespace to deploy the application
  destination:
    server: https://kubernetes.default.svc
    namespace: default

  # Sync policy
  syncPolicy:
    #automated: # automated sync by default retries failed attempts 5 times with following delays between attempts ( 5s, 10s, 20s, 40s, 80s ); retry controlled using `retry` field.
    #  prune: true # Specifies if resources should be pruned during auto-syncing ( false by default ).
    #  selfHeal: true # Specifies if partial app sync should be executed when resources are changed only in target Kubernetes cluster and no git change detected ( false by default ).
    #  allowEmpty: false # Allows deleting all application resources during automatic syncing ( false by default ).
    syncOptions: # Sync options which modifies sync behavior
      - Validate=true # disables resource validation (equivalent to 'kubectl apply --validate=false') ( true by default ).
      - CreateNamespace=true # Namespace Auto-Creation ensures that namespace specified as the application destination exists in the destination cluster.
    # The retry feature is available since v1.7
    #retry:
    #  limit: 5 # number of failed sync attempt retries; unlimited number of attempts if less than 0
    #  backoff:
    #    duration: 5s # the amount to back off. Default unit is seconds, but could also be a duration (e.g. "2m", "1h")
    #    factor: 2 # a factor to multiply the base duration after each failed retry
    #    maxDuration: 3m # the maximum amount of time allowed for the backoff strategy

We can see both environment yaml files below. Notice the only things that can between them are environment specific things. This is a very basic file that only has two values but you can imagine something with more complex values.

The default values.yaml is not shown because it is empty for this application. The same basic principle still applies. The default configuration that does not change between environments would be in the values.yaml with the environment specific yaml files overriding the defaults.

values-dev.yaml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


azurejson: |
  {
    "tenantId": "8888888-4444-4444-4444-121212121212",
    "subscriptionId": "8888888-4444-4444-4444-121212121212",
    "resourceGroup": "rg-dev",
    "useManagedIdentityExtension": true,
    "userAssignedIdentityID": "8888888-4444-4444-4444-121212121212"
  }

txtownerid: "devcluster"

values-prod.yaml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


azurejson: |
  {
    "tenantId": "8888888-4444-4444-4444-121212121212",
    "subscriptionId": "8888888-4444-4444-4444-121212121212",
    "resourceGroup": "rg-prod",
    "useManagedIdentityExtension": true,
    "userAssignedIdentityID": "8888888-4444-4444-4444-121212121212"
  }

txtownerid: "prodcluster"

Boostraping ArgoCD

install argocd into the cluster

1

kubectl create namespace argocd

1

kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

get the default password for the argocd cluster

1

kubectl get pods -n argocd -l app.kubernetes.io/name=argocd-server -o name | cut -d'/' -f 2

forward the argocd pod to localhost

This is used until the ingress for argocd is setup

1

kubectl port-forward svc/argocd-server -n argocd 8080:443

change the admin password

1

argocd login localhost:8080

1

argocd account update-password

add a git repo to argocd

1

argocd repo add git@site:/argocd --name argocd-repo-name --ssh-private-key-path ~/.ssh/argocd

add the argocd apps

1

kubectl apply -f argocd/argocd-apps/templates/argocd-apps.yaml

adding azure secret

If the secret is not already encoded via base64.

1

kubectl patch secret -n argocd argocd-secret -o json -p='{"stringData": {"oidc.azure.clientSecret": "YXNkZmxrM2oyM2xrNGI1bGtoajMgLW4K"}}'

If the secret is already encoded in base64.

1

kubectl patch secret -n argocd argocd-secret -o json -p='{"data": {"oidc.azure.clientSecret": "YXNkZmxrM2oyM2xrNGI1bGtoajMgLW4K"}}'

Kick the server to have it read the new secret right away.

1

kubectl delete pods argocd-server-7fd556c67c-6qmv7 -n argocd

adding new apps to argocd

To add a new app to ArgoCD you will create a new Application manifest under the directory that contains another Application that syncs itself. Using the exmaple above we would add a new manifest under the templates directory. The contents of the manifest will be close to the same as all of the others except for settings that change for that particular app. A new app called “random-app” has been added. This app will also have a coresponding directory where the rest of the app actually resides.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


argocd-apps
├── Chart.yaml
└── templates
    ├── argocd-apps.yaml
    ├── azurekv-csi.yaml
    ├── cert-issuers.yaml
    ├── coredns.yaml
    ├── ingress-external.yaml
    ├── ingress-internal.yaml
    ├── kube-state-metrics.yaml
    ├── random-app.yaml

when to use automated sync and prune options

The sync policies as shown below can be modified to fit an apps needs. For apps that can chnage automatically the settings below are good. This will automatically sync the app and prune any extra resources for you. ArgoCD will also create the namespace for you if you do not have a namespace manifest inside your app already.

The retry is going to try to sync your app multiple times. These are the default settings if they do not exist. Sometimes an app will not sync on the first try becuase something does not exist yet or is waiting on something else if one of those scenarios is true then it can try to sync the app again.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


spec:
  # Sync policy
  syncPolicy:
    automated: # automated sync by default retries failed attempts 5 times with following delays between attempts ( 5s, 10s, 20s, 40s, 80s ); retry controlled using `retry` field.
      prune: true # Specifies if resources should be pruned during auto-syncing ( false by default ).
      selfHeal: true # Specifies if partial app sync should be executed when resources are changed only in target Kubernetes cluster and no git change detected ( false by default ).
      allowEmpty: false # Allows deleting all application resources during automatic syncing ( false by default ).
    syncOptions: # Sync options which modifies sync behavior
      - Validate=true # disables resource validation (equivalent to 'kubectl apply --validate=false') ( true by default ).
      - CreateNamespace=true # Namespace Auto-Creation ensures that namespace specified as the application destination exists in the destination cluster.
    # The retry feature is available since v1.7
    retry:
      limit: 5 # number of failed sync attempt retries; unlimited number of attempts if less than 0
      backoff:
        duration: 5s # the amount to back off. Default unit is seconds, but could also be a duration (e.g. "2m", "1h")
        factor: 2 # a factor to multiply the base duration after each failed retry
        maxDuration: 3m # the maximum amount of time allowed for the backoff strategy

For some apps like operators or apps you do not want to automatically sync because they are infrastructure related apps then commenting out or removing the syncPolicy is an option. Operators will create new resources that are not part of the app as ArgoCD sees it and will try to prune the new resources created by the operator. This causes a fight between the operator and ArgoCD trying to create and destroy the same resources repeatedly. It is possible to just turn off pruning and allow the operator manifest to sync automatically or just leave everything off as shown below.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


spec:
  # Sync policy
  syncPolicy:
    #automated: # automated sync by default retries failed attempts 5 times with following delays between attempts ( 5s, 10s, 20s, 40s, 80s ); retry controlled using `retry` field.
    #  prune: true # Specifies if resources should be pruned during auto-syncing ( false by default ).
    #  selfHeal: true # Specifies if partial app sync should be executed when resources are changed only in target Kubernetes cluster and no git change detected ( false by default ).
    #  allowEmpty: false # Allows deleting all application resources during automatic syncing ( false by default ).
    syncOptions: # Sync options which modifies sync behavior
      - Validate=true # disables resource validation (equivalent to 'kubectl apply --validate=false') ( true by default ).
      - CreateNamespace=true # Namespace Auto-Creation ensures that namespace specified as the application destination exists in the destination cluster.
    # The retry feature is available since v1.7
    retry:
      limit: 5 # number of failed sync attempt retries; unlimited number of attempts if less than 0
      backoff:
        duration: 5s # the amount to back off. Default unit is seconds, but could also be a duration (e.g. "2m", "1h")
        factor: 2 # a factor to multiply the base duration after each failed retry
        maxDuration: 3m # the maximum amount of time allowed for the backoff strategy