Improving a Harness from Failures

A failure alone does not reveal which part of the harness to change. First identify the earliest divergence from the expected state, then form a causal hypothesis and a small intervention. The first divergence is a diagnostic starting point, not proof of the cause.

A Database-Query Example

For example, if a SQL syntax error follows selection of the wrong database table, fixing only the syntax leaves the upstream divergence intact. Changing the table-selection step and rerunning the task tests whether it caused the failure. Also inspect for still-earlier factors, such as incorrect schema metadata.

Figure 1 shows the process of finding the first divergence, changing only the hook immediately before it, and using a separate evaluator to compare the harness before and after the change.

Figure 1: Find the first divergence in a trajectory and change only the pre-action hook. Validate each candidate update against the target failure, existing successes, held-out tasks, and the tool policy.

Part II’s Execution Feedback and Failure Analysis explains which execution data to preserve and how to represent first divergences and recurring failures as signatures.

What to Update

There are at least three ways to use information from failures.

  • Update the memory read on the next attempt
  • Generate candidate workflow changes, evaluate them, and select one
  • Train the editor that generates candidate updates

Reflexion stores a reflection after a failure in memory, changing the context read on the next attempt within the same task (Shinn et al. 2023). It does not change the model weights, but it does not evaluate an updated artifact carried forward to an independent future task.

Harness-R1 places an editor alongside a fixed target model. The editor reads failure trajectories and changes only predefined hooks (Shao et al. 2026). It reruns the same task set before and after the change, then trains the editor from the difference.

The authors report that the average success rate across WebShop, ALFWorld, and DBBench rose from 44.3% to 53.6%, an increase of 9.3 percentage points. However, because the changes are learned from observed tasks, this result alone does not show that they also help on unseen tasks.

Figure 2: The authors applied changes derived from a small number of failures to held-out tasks. The Harness-R1-specific editor improved performance on average, while a larger general-purpose model used as the editor made it worse. Whiskers show the standard deviation across three seeds.

Source: The held-out-task analysis in Shao et al., “Harness-R1” (Shao et al. 2026).

Figure 2 shows that a larger editor is not necessarily better. However, the results cover only three seeds and a limited set of domains, so they do not show generalization to other environments or harness implementations.

Limiting the Scope of Changes

The editor does not need permission to rewrite the entire repository. Limit its scope as follows.

  • Fix the input and output formats for each hook
  • Prevent changes to the tool policy
  • Reject broken changes and restore the incumbent harness
  • Rerun existing successes as well as the target failure
  • Store each change together with its supporting trajectory and evaluation results

If narrow changes are insufficient, expand the verifiable scope one step at a time.

Key Points

To improve a harness from failures, preserve trajectories, form a causal hypothesis from the first divergence, and try to falsify it with the smallest change. Even when training an editor, keep the acceptance rule outside the editor.

References

Shao, Shuai, Kangning Zhang, Qingyao Li, et al. 2026. Harness-R1: Learning to Edit Executable Runtime Harnesses from Agent Failure Trajectories. https://arxiv.org/abs/2608.02276.
Shinn, Noah, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. “Reflexion: Language Agents with Verbal Reinforcement Learning.” Advances in Neural Information Processing Systems 36. https://arxiv.org/abs/2303.11366.