Incident Response and Management: From On-Call to Postmortem

Outages cannot be prevented, so losses are reduced by response speed and recovery structure.

Overview

Outages are unavoidable in a production service, so what matters is response speed and recoverability. Site Reliability Engineering (SRE) reaches the same conclusion: recovering quickly from failure beats trying to prevent it. This post covers incident response end to end, from the incident lifecycle and the Incident Command System (ICS) through runbooks, postmortems, and chaos engineering.

As of 2026, AI-based incident management platforms are more common, and reports of reduced MTTR (mean time to recover) have followed. SolarWinds' 2025 State of ITSM Report measured a 17.8% drop in average incident handling time at organizations that adopted generative AI. That works out to 4.87 hours saved per ticket. The report analyzed more than 2,000 ITSM systems and roughly 60,000 data points.

That figure covers ITSM ticket handling time, and the size of the reduction varies with workload and implementation quality, so it does not generalize cleanly. AI does not replace human judgment; it narrows the search space and automates repetitive work to support the on-call engineer.

Incident management lifecycle

Detection
    ↓
Triage
    ↓
Response
    ↓
Mitigation
    ↓
Recovery
    ↓
Post-mortem
    ↓
Improvement

On-call and alert management

The on-call system

On-call is the arrangement under which a designated engineer is responsible for responding immediately to system failures during a defined window.

Principles of a healthy on-call rotation:

1. Rotation: at least eight engineers in the rotation, with weekly shifts recommended
2. Escalation path: primary → secondary → manager → executive
3. Compensation: on-call pay or compensatory time off
4. Alert quality: send only actionable alerts
5. Toil budget: if more than 50% of on-call time is repetitive work, it needs automation

Alert routing and severity classification

Preventing alert fatigue:

• Remove alerts that are not actionable
• Group related alerts automatically
• Configure suppression rules
• Mute alerts during maintenance windows

Incident Command System (ICS)

Concept

ICS is a framework for responding to large incidents efficiently through clear role assignment and a defined decision-making chain. It originated in disaster response such as wildfire suppression and was later adopted by technology companies.

Core roles

Incident response process

1. Alert received -> on-call engineer acknowledges (within 5 minutes)
2. Initial assessment -> severity classification (SEV1-4)
3. SEV1/SEV2 -> declare an incident -> open a war room
4. Assign roles -> IC, Tech Lead, Comms Lead, Scribe
5. Investigate and mitigate -> relieve symptoms first, even before root cause is known
6. Communicate -> regular status updates (every 15-30 minutes)
7. Resolve -> confirm the service is healthy -> declare the incident closed
8. Post-mortem -> write it up within 48-72 hours

Runbooks and playbooks

Runbook

A runbook documents the step-by-step response procedure for a specific alert or failure scenario.

# Runbook: DB connection pool exhaustion
 
## Alert condition
- Active DB connections > 90% of the maximum pool size
 
## Initial diagnosis
1. Check the current number of active connections: `SELECT count(*) FROM pg_stat_activity;`
2. Check long-running queries: `SELECT * FROM pg_stat_activity WHERE state = 'active' AND duration > interval '30 seconds';`
3. Check the connection wait queue
 
## Mitigation
1. Cancel long-running queries: `SELECT pg_cancel_backend(pid);`
2. Clean up idle connections if needed
3. Temporarily increase the connection pool size (PgBouncer configuration)
 
## Root cause investigation
- Analyze the slow query log
- Review recent deployment history
- Check for a traffic spike
 
## Escalation
- If unresolved within 30 minutes: page the DBA team

pg_stat_activity is the PostgreSQL view that exposes current connections and running queries; it is where active connection counts and long-running queries are checked. pg_cancel_backend(pid) forcibly cancels the query owned by that process ID, which is the mitigation that reclaims a held connection.

Playbook

A playbook defines response strategy at a higher level than a runbook. It covers how to respond by incident type, communication procedures, and escalation criteria.

Postmortems and root cause analysis (RCA)

Blameless culture

The core of a postmortem is asking "why did the system allow this?" rather than "who made the mistake?".

Principles of a blameless postmortem:

1. Focus on systems and processes, not people
2. Create an environment where people with information can share it safely
3. Mistakes are a learning opportunity, not grounds for punishment
4. Every postmortem is published to the whole organization

Postmortem template

# Postmortem: [incident title]
 
## Summary
- Date: YYYY-MM-DD
- Severity: SEV1
- Impact window: HH:MM - HH:MM (X minutes total)
- Impact scope: feature Z for Y% of users
 
## Timeline
| Time | Event |
|------|--------|
| 14:00 | Deployment of v2.3.1 starts |
| 14:05 | Error rate spikes from 1% to 15% |
| 14:07 | Alert fires, on-call engineer acknowledges |
| 14:10 | Incident declared (SEV1), IC assigned |
| 14:15 | Decision to roll back the deployment |
| 14:20 | Rollback complete, error rate back to normal |
 
## Root cause
The database migration script dropped an index, which increased the
response time of key queries tenfold and exhausted the connection pool
 
## Contributing factors
- No review of the migration script
- Full deployment without a canary stage
- No alert configured on DB connection pool usage
 
## Lessons
- What went well: fast rollback decision (within 10 minutes)
- What to improve: migration review process, canary deployment
 
## Action items
| # | Action | Owner | Due | Priority |
|------|------|--------|------|----------|
| 1 | Write a DB migration review checklist | DBA team | 1 week | P1 |
| 2 | Make canary deployment mandatory | DevOps | 2 weeks | P1 |
| 3 | Add DB connection pool alerts | SRE | 1 week | P2 |

Root cause analysis techniques

5 Whys:

Why did the service go down?
  -> Because the database connection pool was exhausted
Why was the connection pool exhausted?
  -> Because query response time increased tenfold and connections were never returned
Why did query response time increase?
  -> Because an index was dropped, causing full table scans
Why was the index dropped?
  -> Because the migration script contained a DROP INDEX statement
Why was that not caught in review?
  -> Because there was no review process for migration scripts (root cause)

Fishbone (Ishikawa) diagram: Visualizes causes grouped by category: people, process, technology, environment

Reliability metrics

MTBF = MTTF + MTTR
 
Availability = MTTF / MTBF = MTTF / (MTTF + MTTR)
 
Example: MTTF = 720 hours, MTTR = 0.5 hours
Availability = 720 / 720.5 = 99.93%

Chaos engineering and Game Days

Game Day

A Game Day is a planned failure simulation in which the team rehearses a real outage and validates its response capability.

How a Game Day runs:

1. Design the scenario: "what if the primary database stops responding for 5 minutes?"
2. Limit the blast radius: non-production environment or restricted traffic
3. Station observers: watch monitoring dashboards and the alerting system
4. Run the experiment: inject the failure
5. Evaluate the response: assess ICS activation, runbook usage, and communication
6. Retrospective: extract the weaknesses found and the improvements needed

Pre-mortem analysis

A pre-mortem imagines, before anything fails, "if this project had failed, why did it fail?" as a way to surface latent risks early.

Premise: "Six months from now, this system suffered a major outage."
Question: "What went wrong?"
 
Team members write failure scenarios independently, then share them:
- "Imbalance across DB shards created a hotspot"
- "A third-party payment API was down for an extended period"
- "A cache stampede overloaded the DB"

Incident management tooling (2026)

Major platforms

AI SRE agents (a 2026 trend)

An AI SRE acts as an automated site reliability engineer: it investigates incidents autonomously, identifies root causes, and either proposes or applies fixes.

Multi-agent AI approach (current in 2026): Several AI agents investigate alongside the on-call engineer instead of replacing them:

• Narrow the search space
• Automate repetitive steps
• Defer judgment calls to humans

How AIOps changes the SRE role: AI-driven IT operations (AIOps) automate incident response, alert handling, and monitoring. As a result, SRE time shifts away from repetitive operations toward reliability design and automation work.

SRE culture and practice

Google SRE core principles

1. Eliminating toil: automate manual, repetitive work
2. Error budget: balance innovation against stability → monitoring and observability
3. Progressive rollout: never deploy everything at once → deployment strategy
4. Postmortems: run one for every SEV1 and SEV2 incident
5. Pursue simplicity: complexity is a source of failure

How SRE relates to DevOps

DevOps = culture and philosophy (what should be done)
SRE    = concrete implementation (how it should be done)
 
"SRE is a specific implementation of the DevOps philosophy" - Google

Summary

The goal of incident response is not to eliminate failure but to make the path from detection through recovery to post-analysis short and repeatable. On-call rotations and severity classification decide who responds, when, and how far. ICS separates roles and decision authority during large incidents. Runbooks and playbooks move the response out of individual memory and into documents, which makes it reproducible.

Blameless postmortems and reliability metrics such as MTTR give the evidence needed to fix systems and processes so the same failure does not recur. Chaos engineering and Game Days expose weaknesses in the response structure before a real outage does.

The next post in this series covers test automation.

References

• Site Reliability Engineering - Betsy Beyer, Chris Jones, Jennifer Petoff, Niall Murphy (Google)
• Incident Management for Operations - Rob Schnepp, Ron Vidal, Chris Hawley
• The Phoenix Project - Gene Kim, Kevin Behr, George Spafford
• PagerDuty Incident Response Guide: https://response.pagerduty.com