AI Demos Fail for a Boring Reason: Recovery

The agent writes the email and calls the send tool. The provider accepts it. Before the workflow records success, the worker crashes.

When the process restarts, should it call the tool again?

This is a boring question. It is also the point where an AI demo becomes a software product. The quality of the draft no longer matters; the system must know whether an external action already happened.

Most disappointing agent failures have this shape. They are not exotic reasoning errors. They are ordinary interruptions combined with missing execution truth.

Reliable AI is defined by what the workflow preserves when the happy path breaks.

The second run is the real demo#

Interactive walkthrough

A timeout is not proof that an action failed

The runtime checks committed truth before deciding what may happen next.

Step 1 of 5 · Plan

Prepare the action

The workflow validates the draft, policy, and approval state.

Runtime state

Ready to execute

Evidence

Approved action

Recovery starts by resolving uncertainty, not by retrying blindly.

The recovery path is less theatrical than the happy path. It is also the version users actually need.

Temporal’s durable execution documentation makes this point in traditional workflow language: a workflow execution is durable, reliable, and scalable; recovery uses event history so execution can resume from the latest recorded state.^{Temporal Workflow Execution}

AI workflows inherit all of that complexity and add nondeterministic model behavior on top.

That is why recovery cannot be an afterthought.

Ordinary failures create precise runtime questions#

Real workflows are interrupted by ordinary events:

Table

Scroll to compare

Interruption	Product symptom	Runtime question
API timeout	The workflow stalls or retries blindly.	What was the retry budget?
Rate limit	The agent keeps trying and increases cost.	Should the workflow back off, sleep, or reroute?
Worker crash	Progress disappears.	Which step was last committed?
Restart	The system repeats work.	Which side effects already happened?
Invalid external data	The model reasons from bad input.	Was validation performed before commit?
Human delay	The workflow resumes with stale state.	What changed while waiting?
Tool schema mismatch	The agent calls the right tool incorrectly.	Was parameter accuracy checked?
Partial side effect	The API succeeded but the local process failed.	How is idempotency enforced?

A runtime that treats an AI workflow like a temporary script will fail in exactly these moments.

A runtime that treats the workflow as durable can stop, record, resume, and continue.

That difference is the difference between a toy and a system.

Recovery is visible product behavior#

People often talk about reliability as if it were back-end plumbing.

For AI workflows, recovery is visible to the user.

A user notices when:

the same email is sent twice
the workflow starts over from scratch
a draft disappears after a restart
human approvals get lost
yesterday’s context overrides today’s instruction
long-running work silently dies
the system asks the user to explain everything again

These are not only engineering failures.

They are product failures.

MirrorNeuron treats recovery as part of the product promise: durable workflows, explicit state, retries, sleep and resume, and the ability to run workflows from a laptop to a cluster without changing the workflow idea.^{MirrorNeuron Home}^{MirrorNeuron Docs}

Measure recovery by breaking the system#

For customers, the recovery benchmark should be a hard number.

text

fault_recovery_rate =
  workflows_completed_correctly_after_injected_failures
  / workflows_with_injected_failures

A serious runtime should report this across a fault-injection suite, not just claim it abstractly.

MirrorNeuron's current internal benchmark result is:

text

fault recovery rate: 99.2%
benchmark base: 124 / 125 injected failures
target: 99.0%
fault classes covered: worker, tool, loop, and approval failures

That number should be read as a benchmark result for the current evaluation suite, not a universal guarantee across every possible failure mode.

But the principle is stable:

if recovery is not measured, reliability is mostly a story.

A useful fault suite#

A useful benchmark should break the system on purpose.

Table

Scroll to compare

Fault class	Example injection	Passing behavior
Worker failure	Kill the worker during an LLM call.	Resume from last committed step.
Tool timeout	Delay a tool response beyond timeout.	Retry within budget or pause cleanly.
Tool partial success	Tool succeeds but local process crashes before marking complete.	Detect committed side effect and avoid duplicate action.
Invalid output	Model returns malformed JSON.	Reject, repair, or route to verifier without corrupting state.
External data change	Source record changes while workflow waits.	Refresh or flag stale context before continuing.
Human approval delay	Approval arrives hours later.	Resume with current state and recorded approval.
Node loss	Cluster node disappears mid-run.	Fail over without losing workflow state.
Retry storm	Many workflows hit the same failing tool.	Apply backpressure and prevent runaway cost.

This is where a durable runtime has to prove its value.

Not in a perfect demo.

In a controlled disaster.

Three kinds of recovery#

Recovery is often discussed as if it were one thing.

It is not.

A serious AI runtime needs at least three recovery layers.

1. Execution recovery#

Execution recovery asks:

Can the workflow continue after process, machine, or network failure?

This requires persisted state, checkpoints, event logs, and resume semantics.

2. Semantic recovery#

Semantic recovery asks:

Can the agent recover from wrong, missing, stale, or malformed context?

This requires validation, context refresh, source provenance, memory boundaries, and sometimes human review.

3. Side-effect recovery#

Side-effect recovery asks:

Can the system avoid doing the dangerous thing twice?

This requires idempotency keys, commit boundaries, tool-call logs, approval state, and explicit records of external actions.

The third layer is where many agent demos quietly fail.

Generating a duplicate answer is annoying.

Sending a duplicate payment, message, ticket update, database mutation, or trade is a different category of problem.

The commit boundary matters#

A model response should not automatically become truth.

A tool call should not automatically become an approved state transition.

The runtime needs a commit boundary.

text

model proposes
↓
runtime validates
↓
policy checks
↓
side effects execute
↓
result is recorded
↓
state is committed

That boundary is where recovery becomes possible.

If state is committed before validation, the workflow can preserve the wrong thing.

If state is never committed, the workflow can lose progress.

If side effects are not recorded, retries become dangerous.

Recovery changes the economics#

Recovery is also a cost issue.

Every failed workflow has hidden cost:

text

wasted model calls
wasted tool calls
human repair time
duplicated work
lost trust
support burden
opportunity cost

The right economic metric is not raw token spend.

It is cost per successful workflow:

text

cost_per_successful_workflow =
  (model_cost + tool_cost + compute_cost + human_repair_cost)
  / successful_completed_workflows

A system with more careful runtime machinery can look slower or heavier on a single step, but be cheaper across the whole workflow because it avoids restarts, duplicate side effects, and human rescue.

This is the number customers and investors should care about.

The recovery scorecard#

A buyer evaluating an AI runtime should ask for a recovery scorecard that connects directly to the five hard metrics:

Table

Scroll to compare

Buyer metric	Recovery-specific question
Workflow Completion Rate	After normal variance and failures, how often does the workflow still finish correctly?
Fault Recovery Rate	After injected failures, how often does it resume from the right point?
Tool Execution Accuracy	Are retries and tool parameters correct after recovery?
Cost per Successful Workflow	How much cost is wasted on restarts, loops, and duplicate work?
Human Intervention Rate	How often does a person need to repair the workflow rather than approve it?

This is the practical distinction between “agent framework” and “AI workflow runtime.”

An agent framework helps you build behaviors.

A runtime helps those behaviors survive reality.

Recovery should feel boring#

A good recovery model should make AI feel calmer.

The user should not have to babysit every step.

They should be able to inspect progress, pause, resume, approve, retry, and understand what happened.

They should trust that if a machine sleeps, a tool fails, or a process restarts, the workflow does not lose its mind.

That is not magic.

It is runtime design.

The product begins where the happy path ends#

Recovery is not a footnote to intelligent behavior. It is the discipline that keeps one uncertain moment from corrupting the rest of the workflow.

The serious benchmark is not only whether an agent can reason. It is whether the workflow can run, fail, establish what is true, and continue without repeating or losing work.

That is where a demo becomes dependable.

References#

MirrorNeuron Home: MirrorNeuron product page. https://www.mirrorneuron.io/
MirrorNeuron Docs: “MirrorNeuron: durable multi-agent workflow runtime.” https://doc.mirrorneuron.io/
Temporal Workflow Execution: Temporal Docs. “Workflow Execution overview.” https://docs.temporal.io/workflow-execution
LangGraph Durable Execution: LangChain Docs. “LangGraph Overview.” https://docs.langchain.com/oss/python/langgraph/overview
AWS Agent Evaluation: AWS. “Evaluating AI agents: Real-world lessons from building agentic systems at Amazon.” 2026. https://aws.amazon.com/blogs/machine-learning/evaluating-ai-agents-real-world-lessons-from-building-agentic-systems-at-amazon/