Standard templates miss the interesting questions
Standard SRE post-mortem templates work well for “the service went down.” They have rich sections for timeline, root cause, contributing factors, action items. They were designed for a world where the failure was structural — a configuration error, an infra outage, a code bug — and the diagnostic question was “what broke.”
Agent failures are different. The interesting question is rarely “what broke.” It’s often “what did the agent decide, and why.” Standard templates don’t have a place for that question, so post-mortems for agent failures end up either incomplete or off-shape.
This post is a template specifically designed for autonomous agent incidents. It borrows from standard SRE templates and adds the sections that matter for agent failures.
The template
A complete post-mortem for an agent incident covers ten sections. Each should be brief. The whole document should fit on one page, two at most.
1. Summary
One paragraph. What happened, when, who was affected, what was done about it. The version a board member could read.
2. Impact
Concrete impact. Number of users affected, duration, financial impact if quantifiable, regulatory exposure if any. This is the “how bad was it” section.
3. Timeline
Chronological. When the agent started misbehaving. When it was detected. When it was contained. When normal service was restored. Times in a single timezone, accurate to the minute where possible.
4. What the agent did, specifically
Here’s the section that differs from standard templates. List the specific actions the agent took during the incident:
- What inputs it received.
- What decisions it made.
- What tools it called.
- What outputs it produced.
This section needs the trace data. If you can’t reconstruct what the agent did, your observability stack failed and that’s its own action item.
5. Why the agent decided what it did
The next section that differs. For each meaningfully wrong decision, explain — to the extent possible from logs and reasoning traces — why the agent went that way.
This section is genuinely hard. Sometimes the answer is “we can’t fully reconstruct the reasoning.” That’s an acceptable answer if true. Document the uncertainty.
Common sub-answers:
- The prompt was ambiguous in this case.
- The toolset included an overlap that misled the agent.
- The agent’s confidence signal said the action was safe; the signal was wrong.
- The workflow document was wrong for this case.
- The model behaved differently than it had in prior testing.
6. Root cause (or “contributing factors” if there’s no single root)
Standard section. What was the most upstream cause. For agent failures, the root cause is often a combination — a prompt ambiguity that interacted with a model behavior shift that wasn’t caught by the eval suite. Acknowledge multi-factor causes when they apply.
7. Detection
How did you find out about the incident? Was it user complaint, automated alert, periodic review? How long after the incident started?
A good post-mortem highlights any detection gap. If users complained before the system alerted, that’s an observability problem to fix.
8. Containment
What stopped the incident? Was it the kill switch, a configuration change, a circuit breaker, a manual rollback? How long did containment take?
Containment time minus detection time is the metric to track.
9. Action items
Specific, owned, time-bound. Standard structure. For agent failures, common action item types:
- Add to eval suite (specific test cases that would have caught this).
- Adjust the prompt or workflow (specific changes).
- Tighten the toolset (remove overlapping or unsafe tools).
- Improve observability (specific signals or alerts).
- Improve detection (specific monitoring changes).
- Reduce blast radius (specific architectural changes to limit future damage).
Each item has an owner and a target date. Items without those are wishes, not action items.
10. What we would have wanted to know one week before
The most underused post-mortem question. With perfect hindsight, what signal would have predicted this incident? Why didn’t we have that signal? What would it take to have it next time?
This question is the source of the most valuable preventative work. Most agent incidents are predictable in retrospect; the predictive signal existed but wasn’t being watched.
What this template surfaces
Used consistently, the template produces a few distinctive patterns:
Most agent incidents have prompt or workflow issues at root. The agent did exactly what the prompt described; the prompt described the wrong thing for this case. Action items focus on the prompt and workflow as much as on the model or the infrastructure.
Detection is often the longest segment of the timeline. Agents can misbehave for hours before anyone notices. The fix is rarely “make agents better”; it’s “make our observability better.”
The “what we would have wanted to know” section produces the most leverage. Predictive signals are usually available — they just aren’t being watched. Adding them to the dashboard is small work with large payoff.
Action items cluster. Across many incidents, similar action items recur — same kinds of prompt gaps, same kinds of detection gaps. The pattern is worth tracking; it suggests systemic improvements.
What goes wrong without this structure
Two specific patterns we see when teams use a standard SRE template for agent incidents:
The “what did the agent do” reconstruction gets shoved into root cause. It doesn’t fit. The narrative becomes “the model made a wrong choice” without unpacking what choice or why. The action items end up vague.
The “what would have predicted this” question never gets asked. Standard templates don’t have a section for it. The team learns the lesson of the specific incident without extracting the broader preventative signal. The next incident is a different surprise.
The structure of the document shapes what gets discussed. The right structure surfaces the right questions.
The cultural piece
Post-mortems only work if the culture supports them. The non-template pieces:
Blameless. Standard rule. The post-mortem is about systems, not about individuals. People who feel blamed don’t contribute candid information.
Within 72 hours of containment. Recent enough that memory is fresh. Long enough that emotion has cooled. Waiting weeks produces sanitized documents.
Reviewed in a follow-up cadence. Action items must close. A monthly review of “what’s outstanding from prior post-mortems” prevents action items from rotting.
Shared widely. Other teams learn from incidents they didn’t have. The shared library of post-mortems is the institutional memory of how the agent program has actually evolved.
The take
Standard SRE post-mortem templates don’t fit agent failures. The structure above — ten sections, with specific space for what the agent did, why it decided what it did, and what would have predicted the incident — captures what matters. Used consistently across a program’s incidents, the template produces compounding learning. Used inconsistently or skipped, the same kinds of incidents repeat. The cost of the template is fifteen minutes per incident to follow the structure. The benefit is everything you don’t have to relearn.
Incident response patterns are part of how we run Custom AI Builds. If you’ve had a recent agent incident and want a second look at the post-mortem, schedule a call.
