Failure data becomes useful for robot learning when it records the state before failure, the failure mechanism, the severity, whether recovery is possible and the corrective action that follows. Simply mixing unsuccessful episodes into an expert demonstration set can confuse a policy; discarding all of them leaves the policy unprepared for deployment.
Most demonstration programs are optimized to produce clean successes. Operators restart after a slip. Reviewers reject an occluded grasp. Collection dashboards celebrate acceptance rate. This is reasonable for a first behavior-cloning dataset, but it creates a blind spot: the robot learns what nominal execution looks like and receives little evidence about what to do after its own small mistakes.
Why success-only data breaks in closed loop
During deployment, a learned policy chooses actions based on its own previous actions. Small deviations can accumulate until the robot reaches a state that was rare or absent in the expert set. The DAgger framework was developed around this problem of sequential prediction under distribution shift.
In a manipulation task, the off-distribution state may be mundane: the lid is half open, the cloth is folded along the wrong axis, the object shifted during contact, or one hand arrived before the other. A robust system needs at least one of three capabilities: recognize the state as uncertain, stop safely, or recover.
Do not use “failure” as one label
A useful taxonomy separates four classes:
- Recoverable deviation: the task is still achievable without a full reset—for example, a missed grasp with the object still reachable.
- Terminal task failure: the goal can no longer be reached within the episode rules, but the state is safe.
- Safety-critical failure: continuing could harm people, equipment or the environment; intervention is required.
- System or data failure: camera dropout, tracking loss, controller fault or timestamp corruption. This is not a behavioral negative example.
These classes belong in different training and evaluation pipelines. A sensor dropout should not teach the model a recovery motion. A dangerous collision should not be treated as merely a low reward.
The before–event–after capture rule
A clipped failure moment is rarely enough. Preserve three windows:
- Before: the observations and actions that led toward the failure.
- Event: the transition—slip, collision, loss of visibility, dropped object or human stop.
- After: the recovery, reset, escalation or safe termination.
This structure supports different uses: failure prediction, unsafe-action filtering, recovery-policy learning and root-cause analysis.
Collect natural failures before scripted failures
The safest source of failure data is passive preservation of what already occurs during teleoperation and model evaluation. Add event buttons or retrospective review so operators can mark slips, hesitations, interventions and resets without interrupting every episode.
Scripted failure collection can fill gaps, but it needs a safety case. Do not ask operators to manufacture hazardous states simply to balance a label. Use simulation, controlled fixtures or conservative perturbations where physical risk is material.
Interventions are especially informative
An intervention contains two valuable signals: the policy state at which a human decided “continuing is no longer acceptable,” and the corrective action chosen by the human. Record:
- intervention start and end timestamps;
- reason code and severity;
- policy confidence or uncertainty where available;
- pre-intervention observations and actions;
- human corrective trajectory;
- whether autonomous control resumed;
- final task outcome.
Recent work such as Failure-Aware RL explicitly studies failures that require human intervention and recovery policies for real-world manipulation. The broader lesson for data operations is simple: the handoff between autonomy and human control is itself a label.
Negative data needs an intended consumer
Before collecting failures, decide which component will use them:
| Consumer | Useful labels | Important context |
|---|---|---|
| Failure detector | failure onset, type, severity | pre-event window and normal near-misses |
| Recovery policy | recoverability, corrective action, outcome | state after failure and valid recovery paths |
| Safety critic | unsafe state/action, intervention | conservative boundary cases |
| Evaluator | failure mode, terminal state | consistent task-specific definitions |
| Data operations | root cause, device, operator, task version | lineage and recurrence trends |
A negative example without a consumer becomes storage. A consumer without the right temporal context becomes a weak label.
Balance recovery with expert behavior
Recovery data should not overwhelm nominal behavior. Maintain separate partitions and explicit sampling policies. Many failure states admit multiple valid responses: retry, regrasp, clear the workspace, ask for help or abort. Rich reason codes and language can help distinguish them, but the permitted strategy must follow deployment policy.
Also collect hard negatives: situations that resemble a failure but are still acceptable. A momentary pause, an unusual but valid grasp, or a harmless contact should not automatically trigger intervention. Without hard negatives, a failure detector may become overly conservative.
A closed-loop collection cadence
- Train on a clean, accepted demonstration set.
- Run controlled evaluations across a defined condition grid.
- Cluster failures by mechanism rather than by task name alone.
- Collect targeted recoveries for the highest-value, safely recoverable clusters.
- Retrain and measure whether both task success and intervention rate improve.
- Version the new data separately so its contribution can be ablated.
This turns failure capture into a model-driven program instead of an archive of accidents.
What failure-ready data looks like
MovraWorks can incorporate structured failures, interventions and recoveries into teleoperation and human demonstration programs. The specification separates behavioral failures from recording faults, preserves the causal window and keeps high-risk scenarios behind explicit controls.
A robot that has seen only success is not necessarily confident. It may simply be unaware of the boundary between a recoverable mistake and a dangerous one. Failure data gives that boundary a shape.