Send argocd notifications to elasticsearch

argocd, my favorite gitops solution, allows to send notification about event (such as sync failed) on several targets:

• slack
• email
• HTTP aka webhook

On the other side, elasticsearch/kibana, my favorite data visualization solution.

Let’s connect them together!

Argo notification configuration

As usally with argo components, configuration is make via a config-map (you can mount secrets as env var to keep stuff secret):

Triggers

webhook config:

service.webhook.es: >
  url: http://client-es.es.svc:9200/argocd-notif/_doc/
  method: POST
  headers:
  - name: content-type
    value: application/json
  basicAuth: #optional username password
    username: XXXXX
    password: "XXXXX"

template config:

As we want send JSON, let’s build it. We omit useless information such as “last-applied-configuration” annotation etc… Here this is a golang template string:

template.es: |
  webhook:
    es:
      method: POST
      body: |
        {{- $timestamp := call .time.Now -}}
        {
          "name": "{{ .app.metadata.name }}",
          "context": {{ .context | toJson }},
          "labels": {{ .app.metadata.labels | toJson }},
          "annotations": {{ (omit .app.metadata.annotations "kubectl.kubernetes.io/last-applied-configuration" "notified.notifications.argoproj.io") | toJson }},
          "source": {{ (merge (omit .app.spec.source "helm") (dict "type" "helm")) | toJson }},
          "destination": {{ .app.spec.destination | toJson }},
          "syncPolicy": {{ .app.spec.syncPolicy | toJson }},
          "info": {{ .app.spec.info | toJson }},
          "status": {{ (merge (omit .app.status "history" "operationState" "resources" "sync") (dict "operationStatePhase" .app.status.operationState.phase )) | toJson }},
          "@timestamp": "{{ $timestamp.Format "2006-01-02T15:04:05+07:00" }}"
        }

trigger config:

We want a lot of data! dont be stingy

trigger.on-all: |
    - when: app.status.operationState.phase == 'Succeeded'
      send: [es]
    - when: app.status.operationState.phase == 'Unknown'
      send: [es]
    - when: app.status.operationState.phase == 'Error'
      send: [es]
    - when: app.status.operationState.phase == 'Failed'
      send: [es]
    - when: app.status.health.status == 'Degraded'
      send: [es]
    - when: app.status.operationState.phase == 'Running'
      send: [es]
    - when: app.status.health.status == 'Healthy'
      send: [es]
    - when: true
      send: [es]

the glue config:

subscriptions: |
  - recipients:
    - es
    triggers:
    - on-all

Kibana dashboard

Improvements

Here we use elasticsearch document index API, in case of huge notifications count, dont flood elasticsearch and use something like an event-bus, kafka or logstash.