Overview
Even with the same model, an agent’s results change with its prompt, tool interface, workflow, memory, and recovery logic. Harness optimization is the problem of updating this surrounding execution system from execution records and evaluations, while keeping model weights fixed, and carrying the update forward to subsequent tasks.
Suppose an agent answers without checking a tool result. One could revise the instruction, add a verification step, or place a validator before the final answer. Producing such candidates is not enough to establish improvement. They must be compared with the current version under the same conditions, and an accepted change must persist to the next task.
What Counts as Harness Optimization
A harness can include the system prompt, context assembly, retrieval, durable memory, tool mediation, workflow, verification, retries, and rollback (Weng 2026). This book treats an update as harness optimization when it satisfies two conditions:
- Records or evaluations from execution drive the update
- The updated harness is saved as a version and persists to future tasks
This definition sets the scope for describing prior work. To accept an update as a trustworthy improvement, however, an evaluator separate from the optimizer must measure both the current version and the candidate, and a precommitted acceptance rule must select the next version. This book treats that process as the recommended evaluation protocol. Whether an existing method satisfies the protocol is evaluated separately from whether it falls within the definition of the method class.
Scope
The key boundary is whether the updated harness persists to future executions. Generating and discarding alternatives within a single response does not constitute harness optimization.
| Change | Treatment in This Book |
|---|---|
| Save an evaluated change to a prompt, tool policy, workflow, or memory policy | Included |
| Add ordinary execution records to memory | A state update; not called optimization by itself |
| Increase samples or reflection rounds for one task | Test-time scaling; excluded because no change persists to the next run |
| Change model weights through fine-tuning or reinforcement learning (RL) | Model optimization; treated separately |
Labels such as autonomous research or self-evolving agent do not determine inclusion. The system must contain a harness update loop that satisfies the two conditions above.
Reflexion uses a post-failure reflection in the next trial within the same task (Shinn et al. 2023). This is a boundary case involving task-local state updates and should be distinguished from a persistent harness update carried as a version into independent future tasks.
Harness-R1 keeps the target model fixed and uses a separate editor to generate update candidates. The editor is trained, but the harness is the final object being updated (Shao et al. 2026).
Accepting a Trustworthy Improvement
Let the harness currently in use be the incumbent \(H_t\) and a proposed update be a candidate \(H'_t\). In the target agent \(A_{\theta,H_t}\), \(\theta\) denotes fixed model parameters. Running this agent produces a trajectory \(\tau_t\) and evaluation result \(y_t\), which the optimizer \(O\) uses to generate a candidate.
\[ H'_t = O(H_t, \tau_t, y_t), \qquad H_{t+1} = \begin{cases} H'_t & V(H'_t, H_t)=\mathrm{accept},\\ H_t & \text{otherwise}. \end{cases} \]
Under the recommended evaluation protocol, acceptance rule \(V\) compares the external evaluator’s records for the incumbent and candidate. The optimizer may propose a candidate but may not change the evaluator or \(V\). The target model and environment are also fixed in principle; if either changes, treat it as a separate version.
This boundary is shown in Figure 1.
Comparing the Entire Update Loop
This book compares the entire update loop rather than individual models or agent frameworks:
\[ \text{Current harness} \rightarrow \text{Execution} \rightarrow \text{Evidence} \rightarrow \text{Candidate generation} \rightarrow \text{External evaluation} \rightarrow \text{Acceptance / rollback} \rightarrow \text{Next version}. \]
Two prompt optimizers support different strengths of improvement claims when one selects candidates on training examples alone and the other checks unused tasks. Likewise, a workflow editor that fixes only the targeted failure differs from one that also checks prior successes and safety.
To make an experiment comparable, fix five decisions before it begins:
- Edit scope: Files, hooks, prompts, and tool schemas that may change
- Fixed conditions: APIs, permissions, timeouts, and output schemas
- Persisted state: What remains across tasks
- Evaluation metrics: How to aggregate quality, cost, latency, and failure rate
- Acceptance rule: Conditions for accepting or rejecting a candidate and rolling back if needed
The boundary should be explicit enough to state, for example, that “the pre-action hook may change, but the evaluator and tool permissions may not.” If the boundary is ambiguous, one cannot tell whether a performance difference came from the harness update or from different experimental conditions.