Lab 14 • SRE & Incident Command
Production Incident Decision Room (SEV-1 Latency Spike)
Correlate APM metrics, DB slow queries, and distributed traces. Identify the root cause between DDoS, memory leak, or unindexed query, and advance the simulation clock to verify recovery.
Simulated Time
14:14:00 UTC
Current P99 Latency
2,850 ms
Alert: Critical Breached
PostgreSQL CPU
98%
Sequential Scan Saturation
Incident State
ACTIVE OUTAGE
Action Required
Incident Timeline & Slow Query Traces
14:02 UTC: P99 API Latency breached 2,500ms alert threshold (SEV-1)
14:06 UTC: Database CPU spiked to 98% across all primary connection pools
Top Slow Query:
SELECT * FROM orders WHERE account_id = $1 ORDER BY created_at DESC;Seq Scan on orders (cost=0.00..48291.20 rows=320140 width=148) • Execution: 2,410ms