# Incident Triage, Root Cause Analysis, and Postmortems

> Chapter 13 of the Courseiva GOOGLE-PCDOE curriculum — https://courseiva.com//learn/google-pcdoe/incident-triage-and-postmortems

**Official objective:** 3.2 — Conduct incident triage, perform root cause analysis using logs and metrics, and run blameless postmortems.

## Introduction

The Incident Triage, Root Cause Analysis, and Postmortems domain of the PCDOE exam tests your ability to manage chaos in a live production system. When a service goes down, your job is not to panic or start guessing at fixes, but to follow a structured, repeatable process that minimises harm and uncovers the true cause of the failure. This chapter gives you the mental model and the technical vocabulary to handle incidents like a seasoned Site Reliability Engineer (SRE).

## The Restaurant Kitchen Crisis Analogy

A busy restaurant kitchen is the scene. During a Friday night rush, orders are piling up and the head chef notices that every single steak being sent to the pass is overcooked and tough. The chef immediately shouts, 'Stop all steak orders!', diverting the cook to grill chicken instead and calling the supplier to check the meat delivery. This is the triage: stopping the bleeding and stabilising the situation so the restaurant can keep serving other dishes.

Once the rush is over, the chef and the team gather in the walk-in cooler. They look at the temperature logs for the grill, the order tickets that came in, and the delivery receipts. The chef sees that a new batch of steaks arrived that morning, thinner than usual, and the grill cook had not adjusted the cooking time. The root cause analysis reveals that the issue was not a bad batch of meat or a lazy cook, but a miscommunication between the supplier and the receiving clerk about the required steak thickness.

Finally, the chef writes a postmortem report. It does not blame the cook or the clerk. Instead, it describes the facts: the thinner steaks, the unadjusted cooking time, the lack of a check-in step for new ingredients. The report suggests adding a simple visual inspection of all key ingredients before service. The entire team agrees that this process change will prevent the same incident from ruining a future Friday night. The kitchen is now better and more resilient, all because they treated the problem honestly and without blame.

## Core explanation

Incidents are inevitable in any software system. A server crashes, a database becomes slow, or a new software update accidentally deletes user data. When this happens, you need a systematic way to respond. The process is broken into three distinct phases: Incident Triage, Root Cause Analysis (RCA), and Postmortems.

**Incident Triage** is the immediate, high-pressure phase. 'Triage' comes from the French word for sorting, and it is exactly that: you sort incoming problems by their severity. The goal is not to fix the root cause yet. The goal is to contain the damage and restore service to users as quickly as possible. In Google Cloud (GCP), this might mean rolling back a bad deployment, failing traffic over to a secondary region, or scaling up more virtual machines to handle a sudden flood of requests. You use tools like Cloud Monitoring (for alerts) and Cloud Logging (to see error messages in real time) to understand the scope of the problem. You do not aim for a perfect fix; you aim for a good enough fix that stops the bleeding. Once the service is stable, you move to the next phase.

**Root Cause Analysis (RCA)** is the detective work that happens after the immediate crisis is over. You gather data from the incident: timestamps of when the problem started, logs from the affected services, metrics that spiked or dropped, and any changes made to the system right before the incident (like a code deployment or a configuration change). The goal is to identify the one underlying technical failure that caused the incident. This could be a bug in the code, a missing firewall rule, a full disk, or an exhausted database connection pool. An RCA is written as a formal document, often with a timeline of events, the data analysed, and the single root cause identified. It is critical that this analysis is honest and data-driven. If the cause was a human error, the RCA describes what in the system allowed that human error to have such a large impact. A good RCA always asks 'Why?' five times (the 'Five Whys' technique) to drill down from the surface symptom to the real system flaw.

**Postmortems** are the final, and arguably most important, phase. A postmortem is a blameless meeting and document where the entire team reviews the incident and the RCA. The key word is 'blameless'. The culture is designed to encourage full transparency. Engineers must feel safe admitting they made a mistake or missed a warning sign, because that honesty is how the organisation learns. In a blameless postmortem, the question is never 'Who did this?', but rather 'What in our system allowed this to happen, and what can we change to prevent it from happening again?'. The output of a postmortem is a list of action items: specific, concrete tasks like 'Add a monitoring alert for disk space usage above 80%' or 'Update the deployment script to require a manual approval before a production rollout'. These action items are tracked in a system like a bug tracker and followed up on.

Why does this three-phase process matter? Without it, teams react chaotically to every outage. They might fix the symptom (like restarting a server) but never fix the underlying code bug, so the same outage happens again next week. Or, they might blame a junior engineer who made a typo, creating a culture of fear where people hide their mistakes, leading to worse problems later. The SRE approach, codified by Google and tested in the PCDOE exam, is about building resilient systems through disciplined incident response. Every major cloud provider, including Google Cloud, has built-in tooling to support this: Cloud Incident Response, Error Reporting, and Cloud Monitoring all help you triage, analyse, and learn from incidents.

The PCDOE exam expects you to know the exact order: triage first, then root cause analysis, then postmortem. It also tests your understanding of the principles: blameless culture, the 'Five Whys', and the difference between a symptom and a root cause. For example, 'server is slow' is a symptom. The root cause could be 'a single query in the database that is not using an index'. The postmortem action item would be 'add an index to the users table'. You also need to know that postmortems are not formally required for every single small hiccup, but they are mandatory for major incidents (severity 1 and 2 incidents in Google Cloud terms).

Finally, a word on the 'Five Whys' technique. It is a simple but powerful method. You start with the symptom and ask 'Why?', then ask 'Why?' again based on the answer, and continue until you reach a human-process or system-root cause. For example:
- Why did the website go down? (The database server crashed.)
- Why did the database server crash? (It ran out of memory.)
- Why did it run out of memory? (A background job was processing too much data.)
- Why was the background job processing too much data? (The job's code had a bug that did not limit the data set size.)
- Why did the bug get into production? (There was no code review for that component.)
The root cause is the lack of a code review process, not the database crash. The action item is to implement mandatory code reviews for all production changes.

In the real world, a single incident might have multiple contributing factors. The RCA will identify the primary cause, and the postmortem will list all contributing factors. The PCDOE exam tests this as well: they want you to understand that incidents are rarely caused by a single 'stupid mistake', but by a combination of weaknesses in the system that lined up.

## Real-world context

Imagine you are the on-call Site Reliability Engineer for an e-commerce website running on Google Kubernetes Engine (GKE), a service that runs your application in containers. At 2:00 PM on a Monday, your phone buzzes with a high-priority alert from Cloud Monitoring: 'Error rate on checkout service is above 5% threshold'. You open the Cloud Monitoring dashboard and see the error rate is at 15% and climbing. Real customers are failing to complete their purchases.

Here is exactly what you do, step by step:
- **Triage (first 5 minutes):** You acknowledge the alert. You quickly check Cloud Logging for the error messages. You see a '500 Internal Server Error' from the checkout service. The error message says 'Connection refused' for the payment processing service. You know the payment service is a separate microservice. You suspect the payment service is down. Your first action is to roll back the latest change to the checkout service (a new deployment that happened 10 minutes ago). You use the `kubectl rollout undo` command to revert to the previous version. Within 2 minutes, the error rate drops to 0%. Users can check out again. You have triaged the issue: the immediate harm is stopped. However, you still do not know why the payment service was refusing connections.
- **Root Cause Analysis (the following hour):** You now have time to investigate. You pull up the deployment records for the payment service. You see that a configuration change was applied to the payment service 30 minutes before the incident. You compare the old configuration with the new one. You see that the new configuration changed the database connection string; it had a typo in the hostname. The payment service could not connect to its database, so it could not process any payments, and when the checkout service tried to call it, the checkout service got an error. You also check the deployment pipeline logs and see that the configuration change was deployed without a manual approval step. The root cause is a configuration error (the typo) combined with a missing approval gate in the deployment process.
- **Postmortem (the next day):** You schedule a one-hour meeting with the team (developers, operations, QA). The meeting starts with a recap of the timeline (all data from your logs). You state clearly: 'There is no blame here. A typo introduced a configuration error, and our deployment process allowed that change to go straight to production without review. We need to fix the process.' The team brainstorms action items: 1) Add a validation step in the CI/CD pipeline that checks database connection strings before deploying. 2) Require a second person to approve all configuration changes to production. 3) Write a runbook for the on-call engineer about how to handle similar checkout failures. Each action item gets an owner and a due date. The postmortem document is shared with the wider engineering organisation.

This scenario shows how triage (rollback), root cause analysis (finding the typo and the missing approval), and a blameless postmortem (process improvements) all work together. Without the triage, revenue would have been lost. Without the RCA, you might have just restarted the payment service, only to see the error return when the bad configuration was reapplied. Without the blameless postmortem, the engineer who made the typo would feel ashamed and hide future mistakes. The business benefits from faster recovery and fewer repeat incidents.

In the PCDOE exam, you will be asked what to do in the first 5 minutes of an incident. The correct answer is always triage: roll back, scale up, or redirect traffic. Do not start debugging the root cause until the service is stable. Know your Google Cloud tools: Cloud Monitoring for metrics and alerts, Cloud Logging for log data, Error Reporting for aggregated error views, and Service Infrastructure for tracing requests between services. The exam might give you a log snippet and ask you to identify the root cause. Practise reading log entries for common patterns like 'connection refused', 'permission denied', or 'timeout'.

## Exam focus

The PCDOE exam devotes significant weight to the incident lifecycle. You must know the exact definitions and sequence of triage, RCA, and postmortem. The exam is not about theory; it is about applying the correct process in a given scenario.

**What they test specifically:**
- The definition of 'blameless postmortem' and why it is important.
- The 'Five Whys' technique and how to apply it to a given log or metric.
- The order of operations: triage always comes before root cause analysis.
- The difference between a symptom (what users see) and a root cause (the underlying technical failure).
- The difference between a contributing factor (something that made the incident worse) and a root cause (the thing that started it).
- The purpose of a postmortem: to produce action items that prevent recurrence, not to assign blame.
- The types of data you use for RCA: metrics, logs, traces, deployment history, and configuration changes.
- The concept of 'incident commander' and 'operations lead' roles during an incident response in Google Cloud.
- The specific Google Cloud tools: Cloud Incident Response, Cloud Monitoring, Cloud Logging, Error Reporting, and Trace.

**Traps they set:**
- They give you a scenario where a team does a postmortem without doing an RCA first. That is wrong. You cannot write a useful postmortem without first identifying the root cause.
- They give you a scenario where the team blames a person. The correct answer is to change the process, not blame the person.
- They give you a list of actions and ask which is the first step during an incident. The trap is an option that says 'investigate logs to find root cause'. The correct first step is triage (stop the bleeding).
- They describe a postmortem that only lists what went wrong, without any action items. That is incomplete. A good postmortem ends with concrete, tracked action items.
- They confuse 'postmortem' with 'incident report'. A postmortem is specifically the blameless review and action planning phase. An incident report is just the raw log of what happened.

**Correct answer pattern:**
- If a question asks about the immediate response to a critical incident, look for the answer that involves containment or rollback.
- If a question asks about the goal of a postmortem, the answer that says 'to learn and prevent recurrence' is correct. Answers about 'to find who made the mistake' are wrong.
- If a question asks about the 'Five Whys', the answer that demonstrates drilling down from symptom to process/system cause is correct.
- If a question asks about tools, know that Cloud Logging and Cloud Monitoring are the primary tools for RCA.

**Key definitions to memorise:**
- Incident: An event that disrupts service availability, latency, or correctness.
- Triage: The process of assessing and prioritising incidents to minimise immediate impact.
- Root Cause Analysis (RCA): A systematic investigation to identify the single underlying technical failure.
- Postmortem: A blameless review and documentation of an incident to plan improvements.
- Blameless culture: The principle that incidents are viewed as opportunities to improve the system, not to punish individuals.
- Five Whys: A technique of repeatedly asking 'why' to move from symptom to root cause.
- Action item: A specific, assigned task that comes out of a postmortem to prevent a recurrence.
- Severity 1: A critical incident that causes a complete service outage for all users.
- Severity 2: A major incident that causes partial outage or significant degradation for many users.

## Step by step

1. **Acknowledge and Classify the Incident** — The on-call engineer receives an alert from Cloud Monitoring. The first step is to acknowledge the alert and classify its severity (S1, S2, S3, or S4). This determines the speed of response and who needs to be notified. For an S1 (complete outage), the entire incident response team is paged immediately.
2. **Triage: Contain the Damage** — The engineer takes the first safe action to stop the worsening of the incident. Options include rolling back a recent deployment, redirecting traffic to a healthy region, or scaling up resources. The goal is to restore service to users as fast as possible, even if the fix is temporary.
3. **Gather Data for Root Cause Analysis** — Once the service is stable, the engineer collects all relevant data: timestamps of the alert, log entries from the affected services, metrics that deviated, and a list of any recent changes (deployments, config updates, infrastructure changes). This data forms the evidence base for the RCA.
4. **Perform Root Cause Analysis Using Five Whys** — The engineer applies the Five Whys technique to the data. They start with the symptom (e.g., 'users see errors') and ask 'why' repeatedly until they reach the underlying technical failure or process gap. The result is a single root cause statement, such as 'a missing index caused a database timeout'.
5. **Write and Conduct a Blameless Postmortem** — The team meets to review the incident timeline, the root cause, and the contributing factors. The meeting is explicitly blameless. The group brainstorms action items to prevent recurrence. Each action item is assigned to an owner with a deadline. The postmortem document is saved and shared for organisational learning.
6. **Track and Close Action Items** — This is often overlooked but critical. The action items from the postmortem must be tracked in a system (like a bug tracker or a project management tool). The team follows up until each item is completed. Only then is the incident truly resolved from a learning perspective.

## Comparisons

### Symptom vs Root Cause

**Symptom:**
- What the user sees (e.g., 'page loads slowly')
- Often a metric or alert (e.g., latency spike)
- Treated immediately with triage actions

**Root Cause:**
- The underlying technical failure (e.g., missing database index)
- Discovered through log and metric analysis (RCA)
- Requires a code or configuration fix

### Triage vs Root Cause Analysis

**Triage:**
- Happens during the incident (first minutes)
- Goal is to stop the bleeding and restore service
- Uses immediate actions like rollback or traffic redirect

**Root Cause Analysis:**
- Happens after the incident is contained (next hours)
- Goal is to find the single underlying technical failure
- Uses data from logs, metrics, and deployment history

### Blameless Postmortem vs Traditional Post-Incident Review

**Blameless Postmortem:**
- Focuses on system and process failures
- Encourages full transparency and honesty
- Produces action items for system improvement

**Traditional Post-Incident Review:**
- Often focuses on who made a mistake
- Creates fear of punishment, hiding errors
- Rarely produces useful follow-up actions

### Five Whys vs Fishbone (Ishikawa) Diagram

**Five Whys:**
- Simple, linear, iterative questioning technique
- Drills straight down to a single root cause
- Takes 10-20 minutes for a single incident

**Fishbone (Ishikawa) Diagram:**
- Visual diagram showing multiple cause categories
- Used for complex incidents with many contributing factors
- Takes longer but provides a broader view

## Common misconceptions

- **Misconception:** A postmortem is optional; you only do it if someone demands it. **Reality:** Postmortems are a standard, mandatory practice for all significant incidents (S1 and S2). They are the primary way teams learn and prevent future outages. (Beginners often see postmortems as bureaucratic overhead. In reality, they are a disciplined quality improvement tool required by the SRE model and tested on the exam.)
- **Misconception:** Root cause analysis means finding the person who caused the incident. **Reality:** RCA is about finding the technical failure and the system weaknesses that allowed it. Blame is irrelevant; the goal is to find the process or code defect that caused the outage. (This is a very common confusion from non-IT backgrounds, where performance reviews focus on individual mistakes. The exam explicitly tests blameless culture.)
- **Misconception:** Triage and root cause analysis happen at the same time. **Reality:** Triage must happen first. You stop the bleeding before you start the detective work. Doing RCA while the service is down delays recovery and makes the outage worse. (Beginners think multitasking is efficient. In incident response, it is dangerous. The exam loves to test this order of operations with scenario questions.)
- **Misconception:** The 'Five Whys' always finds five distinct causes. **Reality:** The 'Five Whys' is a heuristic, not a fixed number. You stop asking 'why' when you reach a process or system root cause, which might happen after three questions or seven. The number is a guideline, not a rule. (People take 'Five Whys' literally. The exam expects you to understand it as a methodology, not a rigid count.)
- **Misconception:** A postmortem is just a meeting to talk about what went wrong. **Reality:** A postmortem must produce written action items with owners and deadlines. Without follow-through, the postmortem is just a chat. The exam tests that action items are the key output. (Beginners think meetings are enough. The SRE culture values measurable, tracked improvements.)

## Key takeaways

- Incident triage must always come first; never start root cause analysis until the immediate service disruption is contained.
- A blameless postmortem focuses on system changes, not individual mistakes, to prevent the same incident from recurring.
- The Five Whys technique helps you drill from a surface symptom to a genuine underlying system or process root cause.
- Cloud Monitoring and Cloud Logging are the two primary Google Cloud tools used for root cause analysis of incidents.
- Action items from a postmortem must be specific, assigned to an owner, and tracked to completion or they have no real value.
- The difference between a symptom and a root cause is critical: a symptom is what the user experiences (e.g., a timeout), while a root cause is the technical defect (e.g., a misconfigured firewall).

## FAQ

**What is the difference between a root cause and a contributing factor in an incident?**

A root cause is the single underlying technical failure that directly caused the incident. A contributing factor is any condition that made the incident worse or helped it happen, but was not the primary trigger. For example, a bug is a root cause; the lack of a monitoring alert for that bug is a contributing factor.

**Do I have to do a postmortem for every single small error or alert?**

No. Postmortems are typically reserved for significant incidents, such as severity 1 (complete outage) and severity 2 (major degradation). Minor incidents with low impact may be handled with a simpler review or no formal postmortem.

**What happens if an engineer is blamed during a postmortem?**

That would violate the principle of a blameless postmortem. In a properly run postmortem, the focus is on the system and process failures that allowed a human to cause the error. Blame undermines the psychological safety needed for honest reporting and learning.

**What tools in Google Cloud do I use for incident triage and root cause analysis?**

The primary tools are Cloud Monitoring (for metrics and alerts), Cloud Logging (for searching and analysing log data), Error Reporting (for aggregated error stats), and Cloud Trace (for latencies and request flows). All these are integrated into the Google Cloud console.

**Is the 'Five Whys' technique always five questions?**

No. The number is a guideline. You ask 'why' repeatedly until you reach a root cause that is a system or process issue, not a human mistake. Sometimes it takes three questions, sometimes seven. The technique is about depth, not a fixed count.

**What is an 'incident commander' and what do they do during an incident?**

The incident commander is the person who leads the response during an active incident. They coordinate the team, make decisions about triage actions, and communicate status to stakeholders. They do not necessarily fix the technical problem themselves; they manage the response process.

---

Interactive version with quiz and diagrams: https://courseiva.com//learn/google-pcdoe/incident-triage-and-postmortems