Operations

Recover a failed run

Run recovery lets a failed run continue from its last completed checkpoint. Use it when the loop definition is still right and the thing that stopped the run can be fixed outside the already-completed work.

Mobius keeps completed step output, spend, turn counts, and the original loop version. The failed step runs again from the top, so check the side-effect warning before resuming a step that writes to another system.

Choose the recovery action

Start from the run's stop banner. The banner uses stop_reason and error_type to offer the safest next action.

Cause	Use	Why
`model_provider`, `timeout`, `worker_unavailable`	Resume run	The dependency may be healthy now. Mobius retries the failed step in place.
`budget_exceeded`	Raise limit & resume	Spend is cumulative. Raise the budget above `credit_spent`.
`turn_limit_reached`	Raise limit & resume	Turn count is cumulative. Raise `max_agent_turns` above `agent_turns_used`.
`wall_clock_exceeded`	Raise limit & resume	Add more wall-clock time from the recovery request time.
`progress_stalled`	Resume run	The run restarts from the next incomplete step.
`auth`	Reconnect integration, then Resume run	Fix the credential, then retry the same step with the same input.
`step_limit_reached`	Run again (upgrade plan)	The per-run step cap is a plan ceiling, not a per-run limit you can raise.
`check_failed`, `gate_rejected`, `cancelled`, `replaced`	Run again	These stops need a fresh run or a loop edit, not in-place recovery.

When in doubt, use Run again for deterministic configuration mistakes and Resume run for an outside-world problem that has cleared.

Recover from the app

In the app, open Runtime > Runs and select the failed run.
Read the stop banner and the latest timeline event near the failure.
For transient failures, click Resume run.
For budget, turn, or wall-clock stops, click Raise limit & resume.
Confirm the restart step, new headroom, and side-effect warning.
Click Resume run.

The run moves back to suspended, records run.resumed, and continues in the background. Open the timeline to watch the next step.started event.

Raise a limit

Guardrail recovery validates the new limit before the run moves:

Stop reason	Required field	Must be greater than
`budget_exceeded`	`credit_budget` or `budget_usd`	`credit_spent`
`turn_limit_reached`	`max_agent_turns`	`agent_turns_used`
`wall_clock_exceeded`	`wall_clock_extend_seconds`	`0`

The new limit is not a fresh allowance. If a run spent 50 credits and you raise the budget to 75 credits, the run has 25 credits of headroom.

Recover from the CLI

Resume a transient or fixed external failure:

mobius runs resume run_01... \
  --reason "provider recovered"

Retry is the same in-place recovery path, but records operator intent as a retry:

mobius runs retry run_01... \
  --reason "credential rotated"

Raise a budget and resume:

mobius runs resume run_01... \
  --credit-budget 750 \
  --reason "approved more budget"

Raise a turn cap or extend wall-clock time:

mobius runs resume run_01... \
  --max-agent-turns 40

mobius runs resume run_01... \
  --wall-clock-extend-seconds 1800

Use --file recovery.json when you want to keep the request body in version control:

{
  "reason": "approved more budget",
  "credit_budget": 750
}

Recover from the API

Resume and retry are project-scoped run operations:

POST /v1/projects/{project_handle}/runs/{run_id}/resume
POST /v1/projects/{project_handle}/runs/{run_id}/retry

The request body is optional for simple transient recovery:

{
  "reason": "provider recovered"
}

For a guardrail stop, include the limit you are raising:

{
  "reason": "approved more budget",
  "credit_budget": 750
}

budget_usd and credit_budget are mutually exclusive. Use credit_budget when your operator thinks in Mobius credits. Use budget_usd when your operator thinks in dollars.

Successful requests return 202 Accepted with the updated run. A version mismatch returns 409 loop_version_changed, which means the loop has changed since the run started. Start a new run on the latest version instead.

What is preserved

Recovery preserves:

Completed step outputs.
Run input at event.*, inputs.*, config.*, and meta.*.
The original loop_version_id.
Cumulative credit_spent and agent_turns_used.
The run id and timeline.

Recovery changes:

attempt increments by one.
Terminal fields such as completed_at, error_type, error_message, and stop_reason are cleared.
The first incomplete step resets to pending.
The timeline records step.retried and run.resumed.

The restarting step runs from its beginning. If it sent a message, opened a pull request, charged an account, or wrote to another service before failing, that side effect may happen again. Put destructive actions behind an interaction or make the action idempotent.

Automatic transient retry

Mobius also retries some failures before a run becomes failed. Agent steps get a small automatic retry budget for classified transient failures:

model_provider
timeout
worker_unavailable

These retries emit step.retried with retry_scope: transient. They do not consume the step's authored retry budget. Guardrail stops, auth failures, check failures, and deterministic step failures do not retry automatically.

Troubleshooting

Symptom	What to check
Resume run is not shown	The stop reason may require Run again, or the run may not be `failed`.
The API returns `loop_version_changed`	The loop was edited after this run started. Start a fresh run.
The API returns `limit_not_raised`	The new budget or turn cap is not greater than the amount already used.
The run fails again immediately	Open the restarting step. If the cause is deterministic, edit the loop and run again.
A side effect happened twice	The failed step likely performed the side effect before failing. Add idempotency or a human gate before retrying that step.

Read the run state model in runs.
Configure stop bounds in guardrails.
Watch retry and resume events in the event catalog.
Use the CLI from Mobius CLI.