Rail RAG monitor

Status

The quality alert fires when the last 3 probe runs fail more often than the up to 14 runs before them, with a one-sided Fisher exact test at p < 0.01 and a fail-rate rise of at least δ = 0.083 (measured from 3 noise and scheduled runs).

Fisher’s exact test (p < 0.01) is the gate that decides. With these sample sizes the test already needs a rise of at least 3 failed questions of 24 (about 12.5 percentage points), so a δ at the 2-question floor never binds and a measured δ only matters above that.

Two more rules run alongside. Uptime needs no baseline: it fires after 2 consecutive runs in which any question has a status other than ok (HTTP errors, timeouts, rate limits, unusable replies). Latency fires when warm p95 stays above 2× the baseline median on 2 consecutive runs. Only the quality and latency rules wait for 7 baseline runs.

Probe pass rate

Share of the probe questions that passed in each run. A question passes when an answerable one is answered with citations from the right document, or an unanswerable one is refused. HTTP errors are not quality failures.

Question grid

Each row is one probe question, each column one run, oldest on the left.

Latency

Seconds per probe request. Warm requests are alerted; the cold start after the demo scales to zero is shown but not alerted.

Real traffic low volume: not alerted

Daily summary of real requests to the demo, probe traffic excluded. Volumes are small, so no rule alerts on them; read the rates with each day’s n in mind. When the demo is down, failed probe calls can also appear in the error counts, because the demo’s error replies carry no trace id to exclude.

Tokens and cost per request: not reported by the app.