What we log when an agent fails

Debugging an agent run is a forensic exercise. Something happened over several minutes, across several model calls and tool invocations, and the only evidence is whatever you chose to record while it was happening. Choose badly and the evidence is a stack trace from the last step, which tells you where it died and nothing about why.

This is the record we keep for every step of every run. It has been shaped by incidents, and each field is here because its absence once cost us an afternoon.

Run-level fields

Recorded once when the run starts and finalised when it ends.

run_id. A ULID, so it sorts by time and is unique across machines.
workflow and version. Which workflow definition ran and which revision. Behaviour changes with prompts, and prompts change with deploys.
trigger. What started the run and the identity behind it. A user action, a schedule, a webhook.
budget. The caps the run started with: maximum steps, maximum tokens, maximum duration.
outcome. One of completed, escalated, failed, or cancelled, with a reason code. Never a free-text string alone, because you will want to count these.

Step-level fields

Recorded for every model call and every tool action.

step_id and parent_step_id. Steps form a tree when a plan fans out. Flat logs lose that.
kind. Model call, tool proposal, tool execution, validation, or escalation.
input, verbatim. For a model call, the full rendered prompt including system text and every message. Not the template. The template plus variables is not what the model saw, and the difference is where bugs live.
output, verbatim. The full model response before any parsing.
parse_result. Whether the output parsed into the expected type, and if not, the typed error. This one field separates "the model was wrong" from "our schema was wrong" instantly.
action and idempotency_key. For tool steps, the typed action that was executed and the key it was executed under. When a downstream system says it saw something twice, this is how you prove which run sent it.
result. The tool's response, or the error, with the status code and latency.
tokens and cost. Prompt tokens, completion tokens, and the price at the time. Budgets are enforced from these, and finance asks about them later.
model and parameters. Exact model identifier, temperature, and any sampling settings. Behaviour drifts across model versions in ways that are otherwise invisible.
timing. Start, end, and the time spent waiting on the provider versus on us.

What we deliberately leave out

Secrets never enter the record, including anything that looks like a bearer token in a tool response. Tool responses are redacted through the same typed parser that the workflow uses, so a field that is not in the schema does not get logged.

We also do not log the vector embeddings. They are large, they are derivable, and nobody has ever needed one during an incident.

Storage and retention

Each run is a small append-only log. We write steps as they happen rather than at the end, because the runs you most want to inspect are the ones that never reached the end. Prompt and response bodies are stored separately from the structured fields and referenced by hash, which keeps the queryable index small and deduplicates identical prompts across runs. Retention is thirty days for bodies and a year for structured fields.

Replay

The reason to be this thorough is that a complete record makes replay possible. Given the verbatim inputs and the recorded tool results, you can re-run a workflow deterministically without touching the real model or the real tools, and watch exactly where it diverged from what you expected. That is the core of threadlock, and it is the tool we reach for first when a run goes wrong.

The broader argument for this level of discipline is in Designing reliable AI workflows. The short version is that an agent is a distributed system that happens to include a language model, and it deserves the same observability as any other.