Chapter 6 — Observability, failure taxonomies, and production → eval → repair feedback loops¶
The previous chapters built an evaluation system from the inside out: choose the evaluation boundary, design offline sets, calibrate judges, inspect trajectories, and expose changes progressively in production.
One piece remains: close the loop.
Real systems produce signals no static eval can know in advance: new inputs, rare tool combinations, slow dependencies, partial state, abuse, distribution shifts, and failures that appear only after thousands of sessions. Production is indispensable evidence. But telemetry is not an evaluation verdict, and an incident should not automatically become a test.
This chapter asks:
How do we turn a production signal into reproducible evidence that can guide a repair and prevent regression without training ourselves to memorize the incident?
A production failure enters the eval only when its mechanism is reproducible
The signal detects; the trace reconstructs; the taxonomy proposes a cause; the minimal case makes it falsifiable; the regression gate validates the repair; production confirms whether the failure family actually stopped recurring.
How to read it: a metric or trace starts an investigation; it does not pass or fail the system by itself. If you cannot reproduce the mechanism, improve instrumentation. If you can, minimize and redact the case, protect the equivalence class with neighbors, version the verifier, and close the loop only after the repair returns to production and the failure family stops recurring under trustworthy observation.
Observability and evaluation answer different questions¶
Observability tries to reconstruct what happened in a running system.
It can include:
- aggregate metrics such as error rate, timeouts, tail latency, cost, and abandonment
- event and decision logs
- traces and spans connecting a request to models, retrieval, tools, guardrails, and handoffs
- dependency state
- user feedback
- support tickets, escalations, and incidents
Evaluation asks a different question: under a declared contract, does the system satisfy the behavior we want to measure?
That distinction matters. A spike in tool_error_rate proves that a signal changed. It does not by itself identify whether the cause was an invalid agent argument, a degraded API, an expired token, a network timeout, or a bug in the instrument itself.
Likewise, a complete trace can explain one execution without telling us whether that execution was correct. OpenAI Agents SDK, for example, documents end-to-end tracing with spans for generations, function tools, guardrails, handoffs, and audio. That improves workflow reconstruction; it does not turn the trace into a grader.2
The base rule is:
telemetry → hypothesis; reproduction + criterion → eval evidence.
Preserve a reproducible system identity first¶
A failure that cannot be tied to what was actually deployed is difficult to explain.
For each relevant session or task, record enough identity to reconstruct the executed package:
candidate_id
model + snapshot
prompt / policy version
retrieval / index version
tool schema + implementation version
runtime / harness version
routing / fallback policy
feature flags
relevant environment version
These fields do not have to live in one trace, but there must be a stable correlation key.
This prevents the anti-pattern:
when Tuesday also introduced a new index, a new tool version, and a regional fallback policy.
A useful trace preserves causality, not every byte¶
More data is not automatically better evidence.
Current OpenTelemetry GenAI semantic conventions standardize names for operations and attributes, while warning that input/output messages, retrieval queries, and tool arguments/results may contain sensitive information or PII.3 OpenAI Agents SDK gives a similar warning: generation and function spans may store sensitive inputs and outputs and their capture can be disabled.2
An observability contract should therefore answer at least:
which fields are required for diagnosis
which data is redacted or tokenized
who may access it
how long it is retained
which identifiers preserve correlation
which content must NOT persist
Observability does not authorize copying production conversations into an eval set. A session must pass the applicable privacy, permission, redaction, and retention policy before it becomes evaluation material.
OpenTelemetry semantic conventions also evolve over time. Standardized names improve interoperability, but not every GenAI attribute should be treated as a forever-stable application contract. Instrumentation version belongs in the evidence record.4
From anomaly to failure taxonomy¶
A taxonomy is not useful because it makes a dashboard tidy. It is useful because failures with the same mechanism can land together even when their surface text differs.
An operational taxonomy can separate at least:
| Family | Example symptom | Diagnostic question |
|---|---|---|
| Input / distribution | new format or language | Did an input assumption stop holding? |
| Retrieval / context | stale answer | Did selection, freshness, ranking, or assembly fail? |
| Model / decision | incorrect reasoning | Does the error appear before any tool action? |
| Tool selection / arguments | right tool, invalid parameters | Did policy choose the wrong action or build the call incorrectly? |
| Side effect / state | duplicate or partial write | Does external state match the intended action? |
| Orchestration / recovery | wrong retry or timeout reconciliation | Did recovery preserve invariants? |
| Policy / safety | unauthorized action | Was a non-compensatory boundary violated? |
| UX / handoff | abandonment or wrong escalation | Did the workflow technically finish but fail the experience? |
| Infrastructure / dependency | slow or unavailable external API | Is system behavior correct under a degraded dependency? |
| Observability | incomplete trace or corrupt metric | Can we trust the collected evidence? |
| Eval / grader | valid behavior marked FAIL | Is the oracle or task wrong? |
The classification should store symptom and cause separately.
symptom = duplicate_refund
cause = retry_after_ambiguous_timeout_without_state_reconciliation
If we store only duplicate_refund, the repair can target the wrong layer.
The NIST AI RMF Playbook recommends post-deployment monitoring, feedback mechanisms, incident response, recovery, and change management, together with documenting errors, near misses, and attack patterns.5 It also recommends comparing pre- and post-deployment behavior and using operational cases for testing and monitoring.6 This is risk-management guidance, not a universal taxonomy. Product boundaries still determine the useful categories.
An incident is not an eval case yet¶
When production exposes a failure, the wrong path is:
incident
→ copy transcript
→ expected = "do not let this happen again"
→ add to regression suite
That mixes private information, accidental context, and an ambiguous criterion.
The useful path is to build a minimal reproducible case that preserves the relevant mechanism.
An illustrative receipt — not a standard — could look like:
failure_case_id: support-refund-retry-017
observed_at: 2026-09-13T01:22:04Z
candidate_id: voice-support-4f2c
trace_id: redacted:8e4...
source: production_incident
symptom: duplicate_write
failure_family: orchestration_recovery
root_cause_hypothesis: retry_after_ambiguous_timeout
impact: financial_side_effect
reproduction:
environment: sandbox-refunds-v12
precondition: refund_not_yet_reconciled
stimulus: provider_timeout_after_write
expected_invariant:
- at_most_one_refund
- reconcile_before_retry
verifier:
type: deterministic_state_check
version: refund-invariants-v3
provenance:
user_content_retained: false
redaction_policy: prod-to-eval-v2
The goal is not to archive the entire incident. It is to preserve the condition that makes the repair falsifiable.
Reproducibility decides which gate you need¶
Not every production failure belongs in the same eval suite.
Case A — reproducible system behavior¶
Suppose that, in a particular state, an agent repeats a write after an ambiguous timeout.
That can become a trajectory eval with a state verifier:
The repair must show not only that final wording changed but that at_most_one_write remains true.
Case B — infrastructure or capacity failure¶
Suppose an external provider takes 40 seconds and exhausts the connection pool.
Forcing that incident into a prompt eval may be the wrong abstraction. A reliability, load, or fault-injection gate that reproduces the degraded dependency is more appropriate.
The principle is:
move the failure into the narrowest harness that can reproduce its mechanism, then confirm it at the boundary where the risk lives.
Case C — insufficient evidence¶
If the trace is missing, external state cannot be reconstructed, or the event occurred under broken instrumentation, do not invent a cause just to create a test.
Mark the case INSUFFICIENT_EVIDENCE, improve instrumentation, and retain the incident as a signal. A test built around an imagined cause can institutionalize the wrong diagnosis.
Do not overfit the regression suite to individual incidents¶
Anthropic recommends turning real failures and support tickets into eval cases, prioritized by user impact.1 That does not mean storing every incident as a unique independent example.
If twenty tickets represent the same mechanism, twenty nearly identical tests inflate apparent coverage.
Before adding a case:
- deduplicate by mechanism, not just text.
- define the equivalence class you want to protect.
- add at least one neighbor case where correct behavior differs.
- add a hard negative when needed so the repair does not become a universal rule.
- keep regression and capability sets distinct.
For example, a bug summarized as “never retry on a timeout” can lead to a naive repair that removes legitimate retries.
A better regression packet covers:
ambiguous timeout after possible write → reconcile, then decide
confirmed pre-write timeout → retry may be valid
read-only idempotent request → retry policy can differ
explicit provider failure/no write → safe recovery path
This protects the invariant instead of memorizing the incident.
The grader can also be the failure¶
A growing regression suite accumulates defects of its own.
When a production-derived case still fails after a repair, inspect:
Anthropic emphasizes reading transcripts and grades because a low score can come from an ambiguous task, a broken harness, or a grader rejecting valid behavior.1
Each new case should therefore version:
- the task or fixture
- the environment
- the verifier or rubric
- the system candidate
- relevant external data
Changing a grader changes the oracle. Do not present that as an agent improvement until the two effects are separated.
Repair loop: test the cause, not only the disappearance of the symptom¶
A robust repair loop can be written as:
1. detect signal
2. reconstruct execution
3. classify symptom + likely cause
4. reproduce the mechanism
5. create/update the versioned eval
6. implement repair
7. run targeted regression
8. run neighbor / broader regression suite
9. use shadow/canary when real-world risk changes
10. monitor recurrence after deployment
Step 8 prevents the classic “fixed this ticket and broke the neighboring class” failure.
Step 10 prevents another mistake: closing the loop in CI. A repair is not fully confirmed until production stops showing the same failure family without that disappearance being caused by new instrumentation or routing hiding it.
Useful loop metrics¶
There is no universal number that proves a system is repaired. But measures with explicit denominators are useful:
failure_family_rate
reproduction_success_rate
regression_pass_rate
recurrence_after_fix
mean_time_to_detect
mean_time_to_reproduce
mean_time_to_verified_repair
unknown / unclassified failure rate
These metrics measure the process as well as the model. A lower unknown_failure_rate, for example, can mean better instrumentation and taxonomy rather than better agent behavior.
NIST recommends measuring response and repair times and assessing whether existing metrics are sufficient for detecting emergent risks and guiding improvements.6
Three concrete examples¶
1. Correct answer, duplicate write¶
A support agent ends with “refund completed.” The user-facing response is correct, but the backend contains two refunds.
A success metric based only on final wording can pass this run.
The feedback loop uses the trace to expose:
Taxonomy: orchestration_recovery + side_effect.
New eval: external state plus trajectory, not text matching. Repair: idempotency or reconciliation before retry. Regression neighbors: timeout before the write, read-only tool, and idempotent tool.
2. Stale retrieval¶
Users report that the assistant answers with an old policy.
The feedback text is only the symptom. Trace correlation shows affected sessions used index_version=2026-08-31 after the policy changed.
Taxonomy: retrieval_context / freshness.
The reproduction should pin a corpus snapshot and freshness contract. The eval can check selection of the current version and behavior when conflicting documents coexist. The repair may be index invalidation rather than a prompt that says “use recent information.”
3. p99 rises, but the model is not the cause¶
After a release, p99 rises and abandonment increases.
The metric detects the regression. The trace shows that most of the time is spent in a CRM API rather than model generation.
Taxonomy: dependency / reliability.
The right gate may be fault injection with a latency budget and cancellation/recovery, followed by canary exposure. Turning this into a language-response eval would create false coverage.
The loop needs ownership and explicit states¶
An operating system needs to know what happens after a failure is classified.
A minimal state machine can be:
OBSERVED
→ TRIAGED
→ REPRODUCED | INSUFFICIENT_EVIDENCE
→ EVAL_ADDED | EXISTING_EVAL_COVERS
→ REPAIR_CANDIDATE
→ REGRESSION_PASS
→ RELEASED
→ VERIFIED_IN_PRODUCTION
REGRESSION_PASS is not VERIFIED_IN_PRODUCTION.
It is also useful to preserve states such as:
DUPLICATE_MECHANISM
NOT_PRODUCT_DEFECT
GRADER_DEFECT
INSTRUMENTATION_DEFECT
PRIVACY_BLOCKED
This prevents every case from being forced into a single “model bugs” queue.
What should stay attached to the case¶
For an auditable loop, retain enough provenance to reconstruct the decision:
production evidence reference
failure taxonomy version
redaction/privacy decision
minimal reproducible fixture
expected invariant
verifier + version
candidate/version that failed
repair candidate/version
eval-set version
release/deploy identity
post-deploy verification evidence
No specific platform is required to express this contract.
What matters is that a future reviewer can answer:
What did we observe, why do we think it failed, which test represents that mechanism, which change repaired it, and what evidence shows it did not return?
Production does not replace evals; evals do not replace production¶
Anthropic summarizes the operating pattern usefully: automated evals help before launch and in CI/CD, production monitoring finds drift and unanticipated failures, A/B tests validate significant changes, and human review fills gaps and calibrates criteria.1
NIST similarly treats post-deployment monitoring and continuing TEVV as complementary lifecycle activities.5
No layer is sufficient by itself:
offline eval
≠ production monitoring
≠ incident response
≠ randomized experiment
≠ human review
A reliable system connects these layers while preserving the question each one can answer.
Production rule¶
Do not turn telemetry into a score, an incident into a test, or a passing test into proof of root cause.
Close the loop only when you can traverse it in both directions:
production
→ evidence
→ reproducible mechanism
→ versioned eval
→ repair
→ regression gate
→ controlled deployment
→ production
If the same failure returns, do not reflexively add another identical case. First ask whether the repair, coverage, verifier, deployment, taxonomy, or observability failed.
That closes the series: evaluating AI systems in production is not about accumulating scores. It is about building evidence that survives system changes and can explain why a release decision is defensible.
-
Anthropic Engineering, Demystifying evals for AI agents, January 9, 2026. It separates evals, production monitoring, A/B testing, and human review; recommends turning real failures into eval cases, reading transcripts, and maintaining regression suites. ↩↩↩
-
OpenAI Agents SDK, Tracing. Documents traces/spans for generations, function tools, guardrails, and handoffs, including controls around sensitive captured content. ↩↩
-
OpenTelemetry, Gen AI semantic attributes. Defines GenAI semantic attributes and warns that messages, retrieval queries, and tool arguments/results may contain sensitive information or PII. ↩
-
OpenTelemetry, Semantic Conventions 1.44.0. Semantic conventions provide a common telemetry vocabulary; stability should be checked for the specific signal/area and version being used. ↩
-
NIST AI Resource Center, AI RMF Playbook — Manage, especially Manage 4.1–4.2 on post-deployment monitoring, incident response, recovery, feedback, and continual improvement. The Playbook is voluntary guidance, not a universal checklist. ↩↩
-
NIST AI Resource Center, AI RMF Playbook — Measure, especially Measure 2.4 and Measure 4 on production behavior, operational cases, feedback, emergent-risk metrics, and response/repair times. ↩↩