Single-agent incidents are whodunits with one suspect. Multi-agent incidents are conspiracy cases: the refund-bot acted, but the orchestrator told it to, acting on a classifier's output, which misread a ticket. Proving what happened means proving causation across agents — and timing correlation isn't proof.
Why "it happened right after" isn't evidence
Observability tools infer relationships from timing and trace nesting: action B followed action A, so A probably caused B. That's useful for debugging and worthless as evidence. Clocks skew, queues delay, retries duplicate — in a busy system, "B happened 40ms after A" proves nothing about whether A caused B. An auditor, quite rightly, will not accept timing inference as a causal claim.
Real causal evidence has to be declared, not inferred: at the moment agent B acts because of agent A's action, the system records an explicit link — this action (agent B, seq #45) was caused by that action (agent A, seq #12). Declared at write time, validated against the ledger, pinned to the exact records.
Designing a caused_by convention
The link needs three properties to be evidence-grade:
- Declared by the infrastructure, not the agent. The orchestrator code that dispatches work knows the causal parent; the agent's own log message doesn't. Capture
caused_bywhere the dispatch happens — e.g.,{"agent_id": "orchestrator", "seq": 12}attached to the child action at ingest. - Validated at write time. The target must name a real agent and a real record that already exists in the ledger. A dangling causal pointer is worse than none — it looks like evidence and isn't. Reject invalid targets with an error, loudly, at ingest.
- Pinned to immutable records. Store the target's hash alongside the sequence number. Sequence numbers can theoretically be re-anchored; the hash pins the exact bytes of the causal parent. Since both records live in hash-chained ledgers, the causal claim inherits tamper-evidence from both ends.
What the graph gives you in an incident
With declared links, a multi-agent incident becomes a directed acyclic graph you can walk backwards: start at the harmful action, follow each caused_by edge to its parent, and stop at the root — the first action with no declared cause. That root is where your remediation belongs. Without the graph, teams fix the symptom (constrain the refund-bot) while the actual defect (the orchestrator's misclassification rule) keeps firing.
Render the graph with both endpoints cited — agent names, sequence numbers, hash prefixes, timestamps. An incident report that says "refund-bot seq #45 was caused by orchestrator seq #12 (hash 3c07…)" is a different document from "the refund seems related to the orchestration step." One of them ends arguments.
Patterns that work in practice
- Dispatch-time linking: the orchestrator attaches the parent's seq to every child action it triggers. Covers 90% of cases; one code change in the dispatcher.
- Message-envelope linking: when agents communicate via a bus or queue, the envelope carries the causal parent's (agent_id, seq, hash). The consumer records it at ingest. Works across service boundaries.
- Human-in-the-loop as a node: when a human approves an action, record the approval as a causal parent (agent_id "human-operator", with the approval record's hash). Human approvals are often the most important causal nodes — and the least logged.
- Depth limits: cap causal chains at a sane depth for rendering (a runaway recursive agent can generate deep chains). The ledger keeps everything; the graph view truncates gracefully.
What causality can't do
Declared links capture intended causation — what the system believes caused what. They don't capture emergent causation (two agents independently reacting to the same external event) or counterfactuals (would B have happened without A?). For those, you still need the full timeline and human judgment. The graph narrows the search; it doesn't replace the investigation.
And like all ledger properties, causal links are only as trustworthy as the write path: if a compromised orchestrator declares false parents, the graph faithfully records the lie. Defense in depth — independent verification, anomaly detection on link patterns (an agent suddenly causing ten times its usual downstream actions is worth an alert) — still applies.
The rule of thumb
If your system has more than one agent and any of them can affect the world, declare your causal links at write time and validate them against the ledger. Timing tells you what happened near something. Causality tells you what happened because of something. Only the second one holds up after the incident.