Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
49385dd
fix: scan linked worktree history and staged changes
simonvanlierde Nov 14, 2026
84531d6
ci: verify fresh demo setup and dependency licenses
simonvanlierde Nov 14, 2026
ca00a1a
chore(collector): use canonical processor and transform names
simonvanlierde Nov 14, 2026
0e9fb2d
fix(rum): report final LCP and session-window CLS
simonvanlierde Nov 14, 2026
12057ef
fix(rum): distinguish zero CLS from unsupported measurement
simonvanlierde Nov 14, 2026
f3ec1da
test(rum): run vitals regressions during lint
simonvanlierde Nov 14, 2026
84a6a62
fix(rum): check browser support before reporting CLS
simonvanlierde Nov 14, 2026
5850ad8
fix: reject missing heartbeat provisioning URL
simonvanlierde Nov 14, 2026
8fa362d
Merge remote-tracking branch 'origin/main' into fix/monitoring-reliab…
simonvanlierde Nov 14, 2026
d08ead4
fix(infra): reject failed optional DNS lookups
simonvanlierde Nov 14, 2026
8bcafa5
docs: correct credential rotation and onboarding procedures
simonvanlierde Nov 14, 2026
5237756
fix(rum): collect only technical error metadata
simonvanlierde Nov 14, 2026
0baa48b
fix(dashboard): show export failures for each signal
simonvanlierde Nov 14, 2026
c4c8f99
fix: unpause interrupted backups and verify restored volumes
simonvanlierde Nov 14, 2026
6b9df43
test: run backup recovery regressions during lint
simonvanlierde Nov 14, 2026
45bc7a7
fix(alerts): keep notification credentials outside grafana
simonvanlierde Nov 14, 2026
f19b181
test(smoke): verify fresh traces and target recovery
simonvanlierde Nov 14, 2026
aa62c78
fix(alerting): stop watchdog on notification failures
simonvanlierde Nov 14, 2026
293d7cc
test: verify notification and heartbeat safeguards
simonvanlierde Nov 14, 2026
646e277
test(rum): cover buffered performance history
simonvanlierde Nov 14, 2026
6d3c4ae
fix(rum): omit browser credentials and referrers
simonvanlierde Nov 14, 2026
25c3b8a
docs(checks): declare jq prerequisite and surface failures
simonvanlierde Nov 14, 2026
56d15fc
fix(dashboard): distinguish export failure signals
simonvanlierde Nov 14, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 5 additions & 4 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -22,12 +22,12 @@ GRAFANA_COOKIE_SECURE=false
# openssl rand -hex 32
OTLP_AUTH_TOKEN=local-dev-token

# Where Grafana delivers alert notifications (self-hosted ntfy, hosted ntfy, Slack, …).
# Where the internal relay delivers Grafana alert notifications (self-hosted ntfy, hosted ntfy, Slack, …).
# For self-hosted ntfy with basic auth publisher:
# ALERT_WEBHOOK_URL=http://grafana:password@ntfy:80/alerts?tpl=1&title={{or .commonAnnotations.summary .groupLabels.alertname}}&message={{or .commonAnnotations.description .commonAnnotations.summary "Alert firing"}}
# Not optional: an empty value drops every alert silently while the heartbeat
# below keeps reporting healthy. With the tunnel overlay active, `just up`
# refuses to start without it.
# Only the relay receives this secret; Grafana stores an internal URL.
# Empty disables delivery (the relay returns 503).
# With the tunnel overlay active, `just up` refuses to start without it.
ALERT_WEBHOOK_URL=

# Public URL Cloudflare Tunnel exposes ntfy on, e.g. https://ntfy.example.com
Expand All @@ -36,6 +36,7 @@ NTFY_BASE_URL=

# Dead man's switch ping target (e.g. https://hc-ping.com/<uuid>). The Watchdog
# alert posts here every 5m; alert externally when pings stop.
# Only the relay receives this secret. Empty disables heartbeat delivery.
HEARTBEAT_URL=

# Optional. Lets `./bootstrap.sh` create a project's healthchecks.io checks.
Expand Down
8 changes: 7 additions & 1 deletion .github/CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,13 +22,19 @@ request outside what the stack sets out to do is closed with a link to this sect

## Set up and check

You need Docker with the Compose plugin and [`just`](https://github.com/casey/just).
You need Docker with the Compose plugin, [`just`](https://github.com/casey/just),
and `jq` for the local regression checks.
`just hooks` installs the git hooks (with [prek](https://prek.j178.dev), which sets up the
Python environment for the citation hooks itself). `just check` runs the static checks and
validates every config, `just smoke` starts a throwaway copy of the stack and sends it
telemetry, and `just fmt` formats the YAML. CI runs `just lint`, `just validate` and
`just smoke`, plus `just infra-validate` when `infra/` changes. The
[Verification commands](../docs/RUNBOOK.md#verification-commands) table in the runbook lists every recipe.
CI also checks the README demo on a fresh runner and checks the demo dependency licences.

Add or update tests for each behaviour change, including bug fixes. Extend the assertions
in `just lint` or `scripts/smoke.sh` for the affected path. Follow the coding standards
checked by Ruff for Python, ShellCheck for shell scripts, and yamllint and yamlfmt for YAML.

## Write commits and pull requests

Expand Down
2 changes: 1 addition & 1 deletion .github/ISSUE_TEMPLATE/bug.yml
Original file line number Diff line number Diff line change
Expand Up @@ -31,4 +31,4 @@ body:
id: environment
attributes:
label: Environment
description: Host operating system, and the Docker and Compose versions (`docker version`, `docker compose version`).
description: Operating system and Docker and Compose versions (`docker version`, `docker compose version`).
3 changes: 3 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ jobs:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
# gitleaks scans git history; a shallow clone would pass vacuously.
with:
persist-credentials: false
fetch-depth: 0
- uses: extractions/setup-just@53165ef7e734c5c07cb06b3c8e7b647c5aa16db3 # v4.0.0
# 600, or the exposure guards refuse it as world-readable.
Expand All @@ -36,6 +37,8 @@ jobs:
timeout-minutes: 15
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: extractions/setup-just@53165ef7e734c5c07cb06b3c8e7b647c5aa16db3 # v4.0.0
# 600, or the exposure guards refuse it as world-readable.
- run: install -m 600 .env.example .env
Expand Down
41 changes: 41 additions & 0 deletions .github/workflows/cold-install.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
name: cold install

on:
pull_request:
schedule:
- cron: "47 5 * * 1"
workflow_dispatch:

permissions:
contents: read

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

jobs:
readme:
name: README demo, fresh runner
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: extractions/setup-just@53165ef7e734c5c07cb06b3c8e7b647c5aa16db3 # v4.0.0
# Docker and Compose are supplied by the fresh runner; no build cache is restored.
- name: Run the README quick start
run: |
cp .env.example .env
just demo
- name: Check the demo API and Grafana
run: |
docker compose -p monitoring-demo -f compose.yml -f compose.demo.yml -f compose.sandbox.yml \
exec -T demo-load curl --fail --retry 20 --retry-all-errors --retry-delay 2 demo-api:8000/openapi.json
curl --fail --retry 20 --retry-all-errors --retry-delay 2 http://localhost:3002/api/health
- if: failure()
run: >-
docker compose -p monitoring-demo -f compose.yml -f compose.demo.yml -f compose.sandbox.yml
logs --no-color --tail=200
- if: always()
run: just demo-destroy
2 changes: 2 additions & 0 deletions .github/workflows/infra.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@ jobs:
timeout-minutes: 10
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: extractions/setup-just@53165ef7e734c5c07cb06b3c8e7b647c5aa16db3 # v4.0.0
- run: install -m 600 .env.example .env # 600 or the exposure guards reject it
- run: just infra-validate
42 changes: 42 additions & 0 deletions .github/workflows/licenses.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
name: demo dependency licenses

on:
pull_request:
schedule:
- cron: "23 5 * * 1"
workflow_dispatch:

permissions:
contents: read

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

jobs:
licenses:
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: astral-sh/setup-uv@68559e99cc807c1eb7e9238fd43a15bba460eec2 # v10.3.3
with:
version: "0.13.6"
enable-cache: false
- id: python
name: Read the demo runtime
run: |
version=$(sed -n 's/^FROM python:\([0-9]*\.[0-9]*\).*/\1/p' demo/Dockerfile)
test -n "$version"
echo "version=$version" >> "$GITHUB_OUTPUT"
# Resolve the same manifest as the demo image, including transitive packages.
- run: >-
uv pip compile --python-version "${{ steps.python.outputs.version }}"
demo/pyproject.toml -o "$RUNNER_TEMP/requirements.txt"
- name: List licenses and reject incompatibility with the repository's MIT license
run: >-
uv run --no-project --python "${{ steps.python.outputs.version }}"
--with-requirements "$RUNNER_TEMP/requirements.txt"
--with licensecheck==2026.0.8 licensecheck --license MIT -r "$RUNNER_TEMP/requirements.txt" --zero
5 changes: 4 additions & 1 deletion .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,16 +11,19 @@ on:
tags: ["v*"]

permissions:
contents: write
contents: read

jobs:
release:
runs-on: ubuntu-latest
timeout-minutes: 15
permissions:
contents: write
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
# gitleaks scans git history; a shallow clone would pass vacuously.
with:
persist-credentials: false
fetch-depth: 0
- uses: extractions/setup-just@53165ef7e734c5c07cb06b3c8e7b647c5aa16db3 # v4.0.0
# The tag's section of CHANGELOG.md is the release body. No section, no
Expand Down
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,8 @@ This launches the core monitoring stack and a demo FastAPI service under simulat
- View **Dashboards → Service Health (RED)** for request rates, error rates, and latency traces.
- Stop and clean up: `just demo-destroy`.

The demo displays sample telemetry but does not deliver alert notifications.

![Service Health dashboard](docs/img/service-health.png)

## Operating the stack
Expand Down
6 changes: 6 additions & 0 deletions compose.demo.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,12 @@ x-demo-logging: &demo-logging
max-file: "3"

services:
notification-relay:
# Demo notifications must not reach a production webhook or dead man's switch.
environment:
ALERT_WEBHOOK_URL: https://demo.invalid/alerts
HEARTBEAT_URL: https://demo.invalid/heartbeat

demo-api:
build: ./demo
restart: on-failure
Expand Down
35 changes: 30 additions & 5 deletions compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -163,10 +163,35 @@ services:
- /:/host:ro
networks: [backend]

notification-relay:
image: python:3.15.0-slim@sha256:f5b92f2a669a4c347044f29367f31983280b648250ca94dea66f98c53bc87e58
restart: unless-stopped
user: "65534:65534"
read_only: true
command: ["python", "-B", "/app/relay.py"]
volumes:
- ./notification-relay/relay.py:/app/relay.py:ro
environment:
ALERT_WEBHOOK_URL: ${ALERT_WEBHOOK_URL:-}
HEARTBEAT_URL: ${HEARTBEAT_URL:-}
healthcheck:
!!merge <<: *healthcheck
test:
- CMD
- python
- -c
- "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8080/health', timeout=2).close()"
mem_limit: 64m
!!merge <<: *hardened
pids_limit: 32
networks: [notification-egress, notifications]

grafana:
image: grafana/grafana:13.3.0@sha256:488984b716c0862639f91a698adbe2862677a5a07bad42da5f03f3030883b205
restart: unless-stopped
depends_on:
notification-relay:
condition: service_healthy
loki:
condition: service_started
tempo:
Expand All @@ -183,9 +208,6 @@ services:
- grafana_data:/var/lib/grafana
environment:
GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_ADMIN_PASSWORD:?set GRAFANA_ADMIN_PASSWORD in .env}
# Expanded by Grafana into the provisioned contact points.
ALERT_WEBHOOK_URL: ${ALERT_WEBHOOK_URL:-}
HEARTBEAT_URL: ${HEARTBEAT_URL:-}
GF_SERVER_ROOT_URL: ${GRAFANA_ROOT_URL:-http://localhost:3000}
GF_DASHBOARDS_DEFAULT_HOME_DASHBOARD_PATH: /var/lib/grafana/dashboards/stack-health.json
# Secure cookies break plain-http localhost logins, so opt-in. With the
Expand All @@ -209,7 +231,7 @@ services:
test: ["CMD", "wget", "-q", "--spider", "http://localhost:3000/api/health"]
mem_limit: 1g
!!merge <<: *hardened
networks: [default, backend]
networks: [default, backend, notifications]

blackbox-exporter:
image: prom/blackbox-exporter:v0.29.1@sha256:ea1c7aa3ef3f307cd03e85f34abb604478ccb90b567fc690da0cd9776da76b13
Expand Down Expand Up @@ -241,13 +263,16 @@ services:
mem_limit: 128m
pids_limit: 128
!!merge <<: *hardened
networks: [default, backend]
networks: [default, backend, notifications]

# Loki, Prometheus and Tempo have no authentication of their own, so only
# Grafana and the collector can reach them. cloudflared stays on `default`: its
# ingress list comes from Cloudflare at runtime, so an edited ingress cannot
# publish a backend that skips the collector's token gate.
networks:
notification-egress:
notifications:
internal: true
backend:
internal: true

Expand Down
10 changes: 3 additions & 7 deletions config/grafana/alerting/contact-points.yaml
Original file line number Diff line number Diff line change
@@ -1,8 +1,4 @@
# Notification targets. Grafana expands $VAR from the environment.
#
# An unset variable leaves the URL empty and every notification fails silently:
# AlertDeliveryFailing routes to this same webhook, and the heartbeat stays
# green. The exposure guard in `just up` is the only control.
# Destination credentials live only in notification-relay, outside Grafana.

apiVersion: 1

Expand All @@ -13,14 +9,14 @@ contactPoints:
- uid: cp-webhook-default
type: webhook
settings:
url: $ALERT_WEBHOOK_URL
url: http://notification-relay:8080/alerts

- orgId: 1
name: heartbeat
receivers:
- uid: cp-heartbeat-dms
type: webhook
settings:
url: $HEARTBEAT_URL
url: http://notification-relay:8080/heartbeat
# A resolved-notification would ping the switch and mask an outage.
disableResolveMessage: true
16 changes: 11 additions & 5 deletions config/grafana/alerting/rules.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -44,8 +44,8 @@ groups:
folder: Stack alerts
interval: 1m
rules:
# Always firing. execErrState is Error so a broken query stops the
# heartbeat and the switch alarms.
# Stop on failed deliveries, lost Grafana scrapes, or query errors. A new
# counter may start at 1; retain that evidence even if it disappears.
- uid: watchdog-heartbeat
title: Watchdog
!!merge <<: *rule_defaults
Expand All @@ -54,13 +54,19 @@ groups:
- !!merge <<: *query_node
model:
!!merge <<: *query_model
expr: vector(1)
expr: >
(vector(1) and on() (min(up{job="grafana"}) == 1))
unless (sum(increase(grafana_alerting_notifications_failed_total{job="grafana"}[10m])) > 0
or sum(max_over_time(grafana_alerting_notifications_failed_total{job="grafana"}[10m])
unless grafana_alerting_notifications_failed_total{job="grafana"} offset 10m) > 0)
- *firing
labels:
severity: none
annotations:
summary: "Alerting-pipeline heartbeat (always firing)"
description: "Routed to a dead man's switch. Investigate if the pings stop."
summary: "Alerting-pipeline heartbeat"
description: >-
Routed to a dead man's switch. Pings stop when Grafana cannot be scraped
or recent notification failures appear in a 10-minute window. Investigate if the pings stop.
# Cannot page when the webhook itself is broken; the heartbeat covers that.
- uid: alert-delivery-failing
title: AlertDeliveryFailing
Expand Down
8 changes: 4 additions & 4 deletions config/otel-collector.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@ processors:
# service.instance.id on each restart adds one every time. The default is
# unbounded. This cap is a runaway guard, not a budget: 28 streams cost ~51 MiB
# heap against the 400 MiB limiter (measured before `interval` was added).
deltatocumulative:
delta_to_cumulative:
max_streams: 5000

# The count connector emits one datapoint per incoming batch, so 50 log batches
Expand All @@ -109,7 +109,7 @@ processors:
metric_statements:
- context: datapoint
statements:
- set(time, Now())
- set(datapoint.time, Now())

# Sized to the container's mem_limit (512m in compose.yml); keep the two in step.
memory_limiter:
Expand Down Expand Up @@ -170,10 +170,10 @@ service:
processors: [memory_limiter, resource/department, batch]
exporters: [otlp_http/prometheus, count]
# The counters themselves. No batcher: it reordered timestamps ~1/min, which
# deltatocumulative refused as out of order. `interval` does the collapsing.
# delta_to_cumulative refused as out of order. `interval` does the collapsing.
metrics/count:
receivers: [count]
processors: [memory_limiter, deltatocumulative, interval, transform/count_ingest_time]
processors: [memory_limiter, delta_to_cumulative, interval, transform/count_ingest_time]
exporters: [otlp_http/prometheus_count]
telemetry:
metrics:
Expand Down
Loading
Loading