Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,9 @@ displays them side by side.
The alert marker in `config/grafana/alerting/projects.yaml` guards active reporting.
The `ProjectsUncovered` rule fires if an unlisted spoke begins transmitting.

For uptime evidence beyond Prometheus's 30-day retention, run the
[independent probe and daily JSON reports](docs/UPTIME-EVIDENCE.md) on a second host.

## Connect your project

Connecting a service takes about five minutes without application code changes.
Expand Down
6 changes: 6 additions & 0 deletions config/uptime-targets.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
[
{"id": "r9lab-app", "url": "https://app.r9lab.io", "project": "r9lab", "env": "prod"},
{"id": "r9lab-docs", "url": "https://docs.r9lab.io", "project": "r9lab", "env": "prod"},
{"id": "r9lab-api", "url": "https://api.r9lab.io/health", "project": "r9lab", "env": "prod"},
{"id": "cml-gallery", "url": "https://cmlplatform.github.io/cml-gallery/", "project": "cml-gallery", "env": "prod"}
]
4 changes: 4 additions & 0 deletions docs/RUNBOOK.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,10 @@ Retention is only partially size-bounded:
| Loki chunks | 30d | none |
| Tempo blocks | 7d | none |

For evidence beyond raw metric retention, use the
[independent uptime reports](UPTIME-EVIDENCE.md). Their evidence directory lives
on a second host and needs its own backup; `just backup` does not include it.

Loki and Tempo cannot cap their total size, so the disk alert at 80% is the
backstop. Two warnings fire earlier. `HostDiskFilling` means a 6-hour linear
fit says a filesystem is full within 3 days. `PrometheusCardinalityHigh`
Expand Down
88 changes: 88 additions & 0 deletions docs/UPTIME-EVIDENCE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
# Retain independent uptime evidence

Prometheus keeps at most 30 days of raw metrics. Use the independent probe to
retain daily JSON reports beyond that window. Run it on a second host outside
the monitoring hub's failure domain. The repository does not install a schedule
or choose a storage host.

## Prepare the second host

The commands below require Linux, `uv`, Python 3.15, GNU `timeout`,
a checkout at `/opt/monitoring`, and a writable evidence directory at
`/var/lib/monitoring-uptime`.
Provision Python 3.15 with the host's normal tooling before scheduling.
Run the manual commands first so cron never needs to provision the runtime.
Use a dedicated, unprivileged account. Keep this directory private and outside
the checkout. The probe needs outbound HTTP/S access, but no monitoring credentials.

1. Review `config/uptime-targets.json`. This is the independent probe's inventory;
`just probe-add` only updates the hub's Blackbox inventory.
Use final URLs: every redirect counts as failure. A probe accepts 2xx response
headers without reading the body, ignores proxy environment variables, and
enforces a ten-second deadline per target.
2. Register the inventory with today's UTC date:

```sh
cd /opt/monitoring
uv run --no-project --python 3.15 scripts/uptime_evidence.py register --store /var/lib/monitoring-uptime --targets config/uptime-targets.json --vantage second-host --effective "$(date -u +%F)"
```

3. Run one probe manually:

```sh
uv run --no-project --python 3.15 scripts/uptime_evidence.py probe --store /var/lib/monitoring-uptime
```

4. Inspect the JSON output. Check that each target has an observation. Exit 1
means a target failed, exit 2 means invalid input, and exit 3 means the command
could not complete.

For inventory changes, register a new manifest effective on a future UTC day.
Historical membership comes from these manifests, rather than the current
configuration. The first registration day includes unknown slots before setup.

## Schedule collection and daily reports

Configure the second host's scheduler as the probe account. Use an absolute path
to its `uv` executable. The following crontab assumes `/usr/local/bin/uv` and
GNU `date`; adjust them before installation. `CRON_TZ=UTC` requires a cron
implementation that supports it. Otherwise configure equivalent UTC jobs in
the host's existing scheduler.

```cron
CRON_TZ=UTC
*/5 * * * * cd /opt/monitoring && timeout 240 /usr/local/bin/uv run --no-project --python 3.15 scripts/uptime_evidence.py probe --store /var/lib/monitoring-uptime
17 0 * * * cd /opt/monitoring && /usr/local/bin/uv run --no-project --python 3.15 scripts/uptime_evidence.py summarize --store /var/lib/monitoring-uptime --date "$(date -u -d yesterday +\%F)"
```

The four-minute outer timeout bounds a stalled probe job, including DNS lookup.
An interrupted probe can leave targets unobserved; those slots remain unknown.
Monitor nonzero job exits with the host's existing job monitoring. A missing
schedule produces unknown intervals, not successful probes. A failed summary
job can be rerun for its original date. Reports reject conflicting replacements.

Back up the **entire evidence directory** to a separate destination using the
host's existing backup system. Retain at least six months. Do not prune raw
observations or manifests until their reports and backup copies are verified.
The hub's `just backup` does not include this directory.

## Read a historical period

```sh
uv run --no-project --python 3.15 scripts/uptime_evidence.py report --store /var/lib/monitoring-uptime --start 2026-01-01 --end 2026-06-30
```

Use dates covered by registered inventories. Reports count expected five-minute
slots separately as successful, failed or unknown. Compare success against
expected slots to avoid hiding gaps; success among observed slots describes
only measured requests. Keep coverage alongside any percentage.

The result measures HTTP reachability from the named probe location. It does
not establish continuous availability between probes, application correctness,
or availability from every user's network. Keep it separate from hub probes
and request-based service indicators. An outage of the probe host creates
unknown evidence. It cannot retrospectively establish uptime before collection.

To prove recovery after hub replacement, restore the evidence directory on a
clean second host and rerun the historical report. Compare its JSON with the
original output. No connection to Prometheus should be necessary.
8 changes: 7 additions & 1 deletion justfile
Original file line number Diff line number Diff line change
Expand Up @@ -90,8 +90,14 @@ restart service: _guard-if-exposed
# Validate every config in the repo.
check: lint validate

# Independent uptime evidence: same pinned Python as the notification relay.
uptime-test:
docker run --rm --network none -v .:/repo:ro -w /repo {{ruff}} check --no-cache scripts/uptime_evidence.py tests/test_uptime_evidence.py
docker run --rm --network none -v .:/repo:ro -w /repo {{ruff}} format --check --no-cache scripts/uptime_evidence.py tests/test_uptime_evidence.py
docker run --rm --network none -v .:/repo:ro -w /repo $(just _image compose.yml notification-relay) python -B -m unittest discover -s tests -p test_uptime_evidence.py

# Static checks and regression tests in tool images.
lint:
lint: uptime-test
# Digest-pinned, so anything already local is current; only fetch what is missing.
@printf '%s\n' {{lint_images}} | xargs -P 8 -I{} sh -c 'docker image inspect {} >/dev/null 2>&1 || docker pull -q {} >/dev/null'
# Explicit -f, not the host's COMPOSE_FILE, so lint means the same here as in CI.
Expand Down
Loading
Loading