Send argocd notifications to elasticsearch
argocd, my favorite gitops solution, allows to send notification about event (such as sync failed) on several targets:
- slack
- HTTP aka webhook
On the other side, elasticsearch/kibana, my favorite data visualization solution.
Let’s connect them together!
Argo notification configuration
As usally with argo components, configuration is make via a config-map (you can mount secrets as env var to keep stuff secret):
webhook config:
service.webhook.es: >
url: http://client-es.es.svc:9200/argocd-notif/_doc/
method: POST
headers:
- name: content-type
value: application/json
basicAuth: #optional username password
username: XXXXX
password: "XXXXX"
template config:
As we want send JSON, let’s build it. We omit useless information such as “last-applied-configuration” annotation etc… Here this is a golang template string:
template.es: |
webhook:
es:
method: POST
body: |
{{- $timestamp := call .time.Now -}}
{
"name": "{{ .app.metadata.name }}",
"context": {{ .context | toJson }},
"labels": {{ .app.metadata.labels | toJson }},
"annotations": {{ (omit .app.metadata.annotations "kubectl.kubernetes.io/last-applied-configuration" "notified.notifications.argoproj.io") | toJson }},
"source": {{ (merge (omit .app.spec.source "helm") (dict "type" "helm")) | toJson }},
"destination": {{ .app.spec.destination | toJson }},
"syncPolicy": {{ .app.spec.syncPolicy | toJson }},
"info": {{ .app.spec.info | toJson }},
"status": {{ (merge (omit .app.status "history" "operationState" "resources" "sync") (dict "operationStatePhase" .app.status.operationState.phase )) | toJson }},
"@timestamp": "{{ $timestamp.Format "2006-01-02T15:04:05+07:00" }}"
}
trigger config:
We want a lot of data! dont be stingy
trigger.on-all: |
- when: app.status.operationState.phase == 'Succeeded'
send: [es]
- when: app.status.operationState.phase == 'Unknown'
send: [es]
- when: app.status.operationState.phase == 'Error'
send: [es]
- when: app.status.operationState.phase == 'Failed'
send: [es]
- when: app.status.health.status == 'Degraded'
send: [es]
- when: app.status.operationState.phase == 'Running'
send: [es]
- when: app.status.health.status == 'Healthy'
send: [es]
- when: true
send: [es]
the glue config:
subscriptions: |
- recipients:
- es
triggers:
- on-all
Kibana dashboard
Improvements
Here we use elasticsearch document index API, in case of huge notifications count, dont flood elasticsearch and use something like an event-bus, kafka or logstash.