Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions config/grafana/alerting/projects.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@ groups:
labels:
severity: critical
annotations:
runbook_url: https://github.com/CMLPlatform/monitoring/blob/main/docs/RUNBOOK.md#alert-catalog
summary: "No telemetry from {{ $labels.project }}/{{ $labels.env }} in 15m"
description: "Host down, Docker down, the agent down, tunnel down, token rotated wrong, or the collector is rejecting this project's data. Nothing on the project side can detect this on its own. Expect this ~20m after the last sample: absent() needs the series to go stale first."

Expand Down Expand Up @@ -94,5 +95,6 @@ groups:
labels:
severity: warning
annotations:
runbook_url: https://github.com/CMLPlatform/monitoring/blob/main/docs/RUNBOOK.md#alert-catalog
summary: "{{ $labels.project }}/{{ $labels.env }} is sending telemetry but has no alert rules"
description: "Telemetry is arriving for a project/environment bootstrap.sh was never run for, so nothing would notice if it went silent. Run ./bootstrap.sh {{ $labels.project }} {{ $labels.env }} on the monitoring host. A literal `unknown` means the sender set no project or env resource attribute at all: fix the sender (docs/ONBOARDING.md), do not bootstrap a project by that name."
16 changes: 16 additions & 0 deletions config/grafana/alerting/rules.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,7 @@ groups:
labels:
severity: none
annotations:
runbook_url: https://github.com/CMLPlatform/monitoring/blob/main/docs/RUNBOOK.md#alert-delivery
summary: "Alerting-pipeline heartbeat"
description: >-
Routed to a dead man's switch. Pings stop when Grafana cannot be scraped
Expand All @@ -81,6 +82,7 @@ groups:
labels:
severity: critical
annotations:
runbook_url: https://github.com/CMLPlatform/monitoring/blob/main/docs/RUNBOOK.md#alert-delivery
summary: "Grafana cannot deliver {{ $labels.integration }} notifications"
description: >-
Alerts are firing and going nowhere. Most likely an unset ALERT_WEBHOOK_URL,
Expand All @@ -103,6 +105,7 @@ groups:
labels:
severity: critical
annotations:
runbook_url: https://github.com/CMLPlatform/monitoring/blob/main/docs/RUNBOOK.md#alert-catalog
summary: "Scrape target {{ $labels.job }} is down"
description: "Prometheus cannot scrape {{ $labels.instance }} (job {{ $labels.job }}) for 2 minutes."
- uid: otel-export-failures
Expand All @@ -121,6 +124,7 @@ groups:
labels:
severity: warning
annotations:
runbook_url: https://github.com/CMLPlatform/monitoring/blob/main/docs/RUNBOOK.md#alert-catalog
summary: "OTel Collector failing to export via {{ $labels.exporter }}"
description: >-
The collector has been failing to deliver telemetry to a backend for
Expand Down Expand Up @@ -148,6 +152,7 @@ groups:
labels:
severity: warning
annotations:
runbook_url: https://github.com/CMLPlatform/monitoring/blob/main/docs/RUNBOOK.md#alert-catalog
# Spoke series carry host_name; the central node-exporter only instance.
summary: "Filesystem {{ $labels.mountpoint }} on {{ or $labels.host_name $labels.instance }} is over 80% full"
description: >-
Expand All @@ -170,6 +175,7 @@ groups:
labels:
severity: warning
annotations:
runbook_url: https://github.com/CMLPlatform/monitoring/blob/main/docs/RUNBOOK.md#alert-catalog
summary: >-
Filesystem {{ $labels.mountpoint }} on {{ or $labels.host_name $labels.instance }}
fills up within 3 days at the current rate
Expand All @@ -193,6 +199,7 @@ groups:
labels:
severity: warning
annotations:
runbook_url: https://github.com/CMLPlatform/monitoring/blob/main/docs/RUNBOOK.md#alert-catalog
summary: "Memory on {{ or $labels.host_name $labels.instance }} is over 90% used"
description: >-
Sustained for 15 minutes. The OOM killer picks by RSS, so the next
Expand All @@ -216,6 +223,7 @@ groups:
labels:
severity: warning
annotations:
runbook_url: https://github.com/CMLPlatform/monitoring/blob/main/docs/RUNBOOK.md#alert-catalog
summary: "Prometheus holds {{ $values.QUERY }} active series, over the 30k ceiling"
description: >-
Find the label that exploded with
Expand All @@ -241,6 +249,7 @@ groups:
labels:
severity: warning
annotations:
runbook_url: https://github.com/CMLPlatform/monitoring/blob/main/docs/RUNBOOK.md#alert-catalog
summary: "Prometheus gained {{ $values.QUERY }} active series in 30 minutes"
description: >-
Far faster than onboarding a spoke explains, so a label has most
Expand All @@ -267,6 +276,7 @@ groups:
labels:
severity: warning
annotations:
runbook_url: https://github.com/CMLPlatform/monitoring/blob/main/docs/RUNBOOK.md#alert-catalog
summary: "Container {{ $labels.name }} is crash-looping"
description: "{{ $labels.name }} on {{ $labels.host_name }} has restarted more than 3 times in the last hour."
- uid: container-oom-killed
Expand All @@ -282,6 +292,7 @@ groups:
labels:
severity: warning
annotations:
runbook_url: https://github.com/CMLPlatform/monitoring/blob/main/docs/RUNBOOK.md#alert-catalog
summary: "Container {{ $labels.name }} was OOM-killed"
description: "{{ $labels.name }} on {{ $labels.host_name }} hit an OOM kill in the last 5 minutes."
- orgId: 1
Expand All @@ -307,6 +318,7 @@ groups:
labels:
severity: warning
annotations:
runbook_url: https://github.com/CMLPlatform/monitoring/blob/main/docs/RUNBOOK.md#alert-catalog
summary: "{{ $labels.project }}/{{ $labels.env }} {{ $labels.job }} error rate above 5%"
description: >-
More than 5% of requests to {{ $labels.job }}
Expand All @@ -329,6 +341,7 @@ groups:
labels:
severity: warning
annotations:
runbook_url: https://github.com/CMLPlatform/monitoring/blob/main/docs/RUNBOOK.md#alert-catalog
summary: "{{ $labels.project }}/{{ $labels.env }} {{ $labels.job }} p99 latency above 2s"
description: >-
The slowest 1% of requests to {{ $labels.job }}
Expand All @@ -347,6 +360,7 @@ groups:
labels:
severity: critical
annotations:
runbook_url: https://github.com/CMLPlatform/monitoring/blob/main/docs/RUNBOOK.md#alert-catalog
summary: "Outside-in HTTP probe failed for {{ $labels.instance }}"
description: "Blackbox prober failed to reach {{ $labels.instance }} for 2 minutes."
# Google SRE multi-window burn rate alerts for a 99.5% availability SLO
Expand All @@ -373,6 +387,7 @@ groups:
labels:
severity: critical
annotations:
runbook_url: https://github.com/CMLPlatform/monitoring/blob/main/docs/RUNBOOK.md#alert-catalog
summary: "{{ $labels.project }}/{{ $labels.env }} {{ $labels.job }} 14.4x error budget burn (fast burn)"
description: >-
Consuming 2% of the 30-day 99.5% error budget within 1 hour.
Expand Down Expand Up @@ -400,6 +415,7 @@ groups:
labels:
severity: warning
annotations:
runbook_url: https://github.com/CMLPlatform/monitoring/blob/main/docs/RUNBOOK.md#alert-catalog
summary: "{{ $labels.project }}/{{ $labels.env }} {{ $labels.job }} 6x error budget burn (slow burn)"
description: >-
Consuming 5% of the 30-day 99.5% error budget within 6 hours.
Expand Down
55 changes: 43 additions & 12 deletions dashboards/logs.json
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,15 @@
"project"
],
"links": [
{
"title": "Service metrics",
"type": "link",
"icon": "dashboard",
"url": "/d/service-health/service-health",
"includeVars": true,
"keepTime": true,
"targetBlank": false
},
{
"asDropdown": false,
"icon": "dashboard",
Expand All @@ -20,6 +29,13 @@
"tooltip": "Switch to another dashboard in the CML monitoring stack",
"type": "dashboards",
"url": ""
},
{
"title": "Incident runbook",
"type": "link",
"icon": "book",
"url": "https://github.com/CMLPlatform/monitoring/blob/main/docs/RUNBOOK.md#incident-workflow",
"targetBlank": true
}
],
"time": {
Expand All @@ -42,7 +58,7 @@
},
"definition": "label_values(project)",
"query": "label_values(project)",
"refresh": 1,
"refresh": 2,
"sort": 1,
"current": {},
"options": [],
Expand All @@ -58,7 +74,7 @@
},
"definition": "label_values({project=\"$project\"}, env)",
"query": "label_values({project=\"$project\"}, env)",
"refresh": 1,
"refresh": 2,
"sort": 1,
"current": {},
"options": [],
Expand All @@ -74,7 +90,7 @@
},
"definition": "label_values({project=\"$project\", env=\"$env\"}, service_name)",
"query": "label_values({project=\"$project\", env=\"$env\"}, service_name)",
"refresh": 1,
"refresh": 2,
"sort": 1,
"current": {
"text": "All",
Expand All @@ -88,6 +104,21 @@
]
},
"panels": [
{
"id": 100,
"type": "text",
"title": "Incident workflow",
"gridPos": {
"h": 4,
"w": 24,
"x": 0,
"y": 0
},
"options": {
"mode": "markdown",
"content": "Compare errors with All Logs. Follow trace IDs into Tempo. Confirm recovery with fresh Service metrics; missing logs do not prove health."
}
},
{
"type": "timeseries",
"title": "Log Rate by Service",
Expand All @@ -96,7 +127,7 @@
"h": 8,
"w": 12,
"x": 0,
"y": 0
"y": 4
},
"id": 1,
"datasource": {
Expand All @@ -105,7 +136,7 @@
},
"targets": [
{
"expr": "sum by (service_name) (rate({project=\"$project\", env=\"$env\", service_name=~\"$service\"}[$__auto]))",
"expr": "sum by (service_name) (rate({project=\"$project\", env=\"$env\", service_name=~\"${service:regex}\"}[$__auto]))",
"legendFormat": "{{service_name}}",
"refId": "A"
}
Expand All @@ -124,12 +155,12 @@
{
"type": "timeseries",
"title": "Log Rate by Level",
"description": "A rising error line is the cheapest early warning here.",
"description": "Rate by Loki-detected level. Missing levels or an empty result are not proof that errors stopped; compare All Logs and service metrics.",
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 0
"y": 4
},
"id": 2,
"datasource": {
Expand All @@ -138,7 +169,7 @@
},
"targets": [
{
"expr": "sum by (detected_level) (rate({project=\"$project\", env=\"$env\", service_name=~\"$service\"}[$__auto]))",
"expr": "sum by (detected_level) (rate({project=\"$project\", env=\"$env\", service_name=~\"${service:regex}\"}[$__auto]))",
"legendFormat": "{{detected_level}}",
"refId": "A"
}
Expand All @@ -162,7 +193,7 @@
"h": 10,
"w": 24,
"x": 0,
"y": 8
"y": 12
},
"id": 3,
"datasource": {
Expand All @@ -171,7 +202,7 @@
},
"targets": [
{
"expr": "{project=\"$project\", env=\"$env\", service_name=~\"$service\"} | detected_level=~\"(?i)(error|critical|fatal|warn)\"",
"expr": "{project=\"$project\", env=\"$env\", service_name=~\"${service:regex}\"} | detected_level=~\"(?i)(error|critical|fatal|warn)\"",
"refId": "A"
}
],
Expand All @@ -190,7 +221,7 @@
"h": 10,
"w": 24,
"x": 0,
"y": 18
"y": 22
},
"id": 4,
"datasource": {
Expand All @@ -199,7 +230,7 @@
},
"targets": [
{
"expr": "{project=\"$project\", env=\"$env\", service_name=~\"$service\"}",
"expr": "{project=\"$project\", env=\"$env\", service_name=~\"${service:regex}\"}",
"refId": "A"
}
],
Expand Down
Loading
Loading