It's 2 AM. Your support agent just refunded an order it should never have touched, and the customer is threatening legal action. The question you'll be asked — by your boss, the customer, possibly a regulator — is not "what do your logs say?" It's "prove it." This is the playbook.
Minute 0–15: freeze the evidence
Before you touch anything, preserve the state of the record:
- Export the ledger. Download the full CSV of the agent's actions. Each row carries
prev_hashandhash— the export itself is evidence, because any later tampering with the live ledger won't match your exported copy. - Mint a Tamper-Seal certificate. This captures the action count and chain tip at this moment, signed. It is your timestamped "the tape looked like this at 2:14 AM" artifact.
- Run chain verification. One click, server-side recomputation of every hash. Screenshot the PASS stamp with the timestamp. If it reports FAIL with a break point, you have a second incident — someone altered the record — and that changes everything about your response.
Do this before restarting the agent, before rolling back, before the post-mortem. Evidence first, remediation second — you can always fix the system later, but you can never re-create the moment.
Minute 15–60: reconstruct the sequence
Now walk the tape. An incident replay view — stepping through actions in order with timestamps — beats staring at a log dump, because incidents are sequences, not events. For each step, note:
- What tool was called, with what input summary and what output summary.
- Whether the action was caused by another agent's action (in a multi-agent setup, the refund-bot is rarely the true culprit — follow the causal links upstream).
- Any anomaly alerts that fired before the incident: a first-seen tool alert 20 minutes earlier is often the actual root cause.
Write the timeline in plain language as you go: "02:03:11 — orchestrator issued plan step #4 (seq #118). 02:03:44 — refund-bot called issue_refund, caused by seq #118." Cite every claim with seq numbers and hash prefixes. This document becomes the core of every conversation that follows.
The questions you must be able to answer
Every stakeholder asks a variant of the same five questions. Prepare answers with citations:
- What exactly did the agent do? — the cited action sequence.
- Why did it do that? — the decision records and inputs preceding the action. (If your agent doesn't log decision points, start — "the model said so" is not an answer.)
- Was the ledger tampered with? — the verification result and certificate.
- Could we have caught it earlier? — the alert history. An honest "yes, the burst alert fired and nobody was watching the inbox" is painful but credible; "we don't know" is not.
- What stops it happening again? — the remediation, which belongs in the post-mortem, not the evidence phase.
Talking to the customer
Customers don't want your hash chain explained. They want three things: acknowledgment, the facts, and the fix. A message that works: "Here's exactly what our agent did at 02:03 UTC, step by step [timeline]. We've verified our records are complete and unaltered [certificate link]. Here's what we're changing so it can't recur [remediation]." The public integrity badge and certificate links do quiet work here — they let the customer verify your claims without trusting your word.
Talking to auditors and regulators
This is where tamper-evidence earns its keep. An auditor's job is to distrust your systems and verify independently. Hand them: the CSV export (self-verifying via the hash links), the Tamper-Seal certificate (signed, timestamped), and the printable audit report (scope, methodology, findings, integrity statement). The report's methodology section should describe the hash-chain construction plainly — auditors evaluate process, and "each record commits SHA-256 of the previous record's hash plus its payload; verification recomputes all hashes server-side" is a process they can assess.
After: the post-mortem that prevents repeats
Evidence answers what happened. The post-mortem answers why the system allowed it. The most valuable artifact of the whole incident is usually one new logged decision point ("we now log the policy check before every refund-class tool call") or one new alert rule. Feed the incident back into the tape: the recorder that proved what happened should also be the instrument that catches it next time.
Checklist (print this)
- ☐ Export the full ledger CSV before touching anything
- ☐ Mint a Tamper-Seal certificate (timestamped evidence)
- ☐ Run chain verification; screenshot the PASS/FAIL stamp
- ☐ Walk the incident replay; write the cited timeline
- ☐ Follow causal links upstream in multi-agent setups
- ☐ Review alerts that fired before the incident
- ☐ Prepare the five answers (what / why / tampered? / earlier? / next)
- ☐ Customer message: facts + certificate + remediation
- ☐ Auditor pack: CSV + certificate + audit report
- ☐ Post-mortem: one new logged decision point or alert rule