Send argocd notifications to elasticsearch

Thomas Decaux
2 min readSep 18, 2023

--

argocd, my favorite gitops solution, allows to send notification about event (such as sync failed) on several targets:

  • slack
  • email
  • HTTP aka webhook

On the other side, elasticsearch/kibana, my favorite data visualization solution.

Let’s connect them together!

Argo notification configuration

As usally with argo components, configuration is make via a config-map (you can mount secrets as env var to keep stuff secret):

webhook config:

service.webhook.es: >
url: http://client-es.es.svc:9200/argocd-notif/_doc/
method: POST
headers:
- name: content-type
value: application/json
basicAuth: #optional username password
username: XXXXX
password: "XXXXX"

template config:

As we want send JSON, let’s build it. We omit useless information such as “last-applied-configuration” annotation etc… Here this is a golang template string:

template.es: |
webhook:
es:
method: POST
body: |
{{- $timestamp := call .time.Now -}}
{
"name": "{{ .app.metadata.name }}",
"context": {{ .context | toJson }},
"labels": {{ .app.metadata.labels | toJson }},
"annotations": {{ (omit .app.metadata.annotations "kubectl.kubernetes.io/last-applied-configuration" "notified.notifications.argoproj.io") | toJson }},
"source": {{ (merge (omit .app.spec.source "helm") (dict "type" "helm")) | toJson }},
"destination": {{ .app.spec.destination | toJson }},
"syncPolicy": {{ .app.spec.syncPolicy | toJson }},
"info": {{ .app.spec.info | toJson }},
"status": {{ (merge (omit .app.status "history" "operationState" "resources" "sync") (dict "operationStatePhase" .app.status.operationState.phase )) | toJson }},
"@timestamp": "{{ $timestamp.Format "2006-01-02T15:04:05+07:00" }}"
}

trigger config:

We want a lot of data! dont be stingy

trigger.on-all: |
- when: app.status.operationState.phase == 'Succeeded'
send: [es]
- when: app.status.operationState.phase == 'Unknown'
send: [es]
- when: app.status.operationState.phase == 'Error'
send: [es]
- when: app.status.operationState.phase == 'Failed'
send: [es]
- when: app.status.health.status == 'Degraded'
send: [es]
- when: app.status.operationState.phase == 'Running'
send: [es]
- when: app.status.health.status == 'Healthy'
send: [es]
- when: true
send: [es]

the glue config:

subscriptions: |
- recipients:
- es
triggers:
- on-all

Kibana dashboard

Improvements

Here we use elasticsearch document index API, in case of huge notifications count, dont flood elasticsearch and use something like an event-bus, kafka or logstash.

--

--

Thomas Decaux
Thomas Decaux

No responses yet