Independent Optimizer Evaluation
Two methods that reach the same final score are not equivalent if one finds its result after 10 candidates and the other after 1,000. Differences in the starting harness, target model, editable scope, disclosed data, and search budget also prevent the final score from identifying the better optimizer.
The preceding three chapters evaluated each candidate on unused tasks, other models and environments, regressions, cost, and safety. The final step is to shift evaluation from the finished agent to how reliably and efficiently an optimizer improves a harness under matched conditions.
Factors Confounded in the Final Score
The final score of an optimization campaign depends not only on the optimizer, but also on the seed harness, target model, environment, development and validation data, budget, and feedback disclosure policy. To compare only optimizers, hold the other factors fixed or vary them one at a time in a factorial design.
The claim that “Method A’s agent scores higher than Method B’s” does not by itself show that A’s optimizer is better. A may have started from a stronger initial harness, seen more test data, or tried more candidates.
Conditions to Hold Fixed
| Condition | Confounder If Not Fixed |
|---|---|
| Target model and decoding | Differences in base capability and sampling |
| Seed harness | Differences in room for improvement and initial quality |
| Edit surface | Differences in the classes of failures that can be changed |
| Environment image | Differences in tools, latency, and state |
| Development / validation | Differences in feedback quantity and task difficulty |
| Evaluator | Differences in score definitions and judge bias |
| Disclosure policy | Differences in the information provided by traces, labels, and critiques |
| Budget | Differences in candidate count, case-runs, and tokens |
| Acceptance rule | Differences in promotion criteria and tolerated regressions |
| Randomness | Selection of a lucky campaign |
Fixing everything to one setting reduces external validity. Begin with a controlled comparison, then vary the target, seed, surface, and environment one at a time.
Metrics Beyond Final Improvement
Measure the optimizer’s update trajectory directly.
| Metric | Definition | What It Reveals |
|---|---|---|
| Proposal validity | Executable candidates / proposals | Whether the optimizer can handle the edit surface |
| Improvement hit rate | Fraction of candidates that beat the incumbent | Proposal quality |
| Regression rate | Fraction of candidates that break prior successes | Locality of changes |
| Sample efficiency | Validation gain per case-run | Budget efficiency |
| Best-so-far curve | Best validation score at budget \(b\) | Search speed |
| Promotion regret | Gap from the best candidate on validation | Selection quality |
| Attribution accuracy | Agreement with the true failure site or priority | Diagnostic ability |
| Calibration | Predicted improvement versus actual difference | Reliability of self-evaluation |
| Robustness | Variation across seeds, targets, and environments | Transfer stability |
Even when two optimizers reach the same final score, their efficiency differs if one requires 10 proposals and the other requires 1,000.
Separating Candidate Generation from Selection
An optimizer has at least two capabilities.
- Include good candidates in the generated set
- Select the good candidate from limited evaluations
Define selection regret as the gap between the candidate with the highest score on promotion validation and the candidate actually selected by the optimizer. This measures how well the optimizer selects from surrogate estimates or limited evaluation queries during search. Do not rerank the generated set on the sealed test; evaluate only the finalized candidate and incumbent there.
If the candidate set contains no good artifact, the problem lies in proposal generation. If it contains a good artifact that is not selected, the problem lies in the surrogate or validation procedure.
Separating Harness Updating from Harness Benefit
An optimizer’s ability to produce useful updates is not the same as a target agent’s ability to benefit from an updated harness. Harness Updating Is Not Harness Benefit defines updating capability as the average post-update gain across a set of anchor agents, and benefit capability as the average gain a target agent receives from updates produced by multiple evolvers (Lin et al. 2026).
Crossing evolvers with solvers distinguishes strong updaters from strong beneficiaries. Independent evaluation should report both axes rather than interpreting one end-to-end gain as the optimizer’s intrinsic diagnostic ability.
Evaluator and Acceptance Rule
The evaluator performs measurement, while the acceptance rule makes the decision. Asking a single LLM judge only whether a candidate improved hides the measurement, aggregation, and threshold. Return executable tests, per-task scores, costs, invalid and safety events, and human ratings separately, then apply thresholds fixed in advance.
Traditional optimizer research often lets the optimizer use development metrics and feedback directly, so separating the evaluator from acceptance is not yet an established standard. This separation is a recommended protocol for avoiding evaluation overfitting and unclear promotion rationales.
Judge Independence
Judge bias remains even when the target, optimizer, and evaluator use different models. The self-preference of large language model (LLM) evaluators for their own generations has been demonstrated empirically (Panickssery et al. 2024). Small changes in prompt wording have also been shown to cause large changes in model scores (Sclar et al. 2024).
When using an LLM judge, blind the candidate identity, randomize answer order, and fix the prompt and model version. Prefer executable verifiers, and report agreement with a human-verified subset and disagreement among multiple judges.
If judge explanations are returned to the optimizer, that judge is a development feedback generator. Keep it separate from the final acceptance evaluator.
Direct Evaluation Through Priority Ranking
Towards Direct Evaluation of Harness Optimizers via Priority Ranking is a recent preprint in which an optimizer ranks harness components by improvement priority (Ong et al. 2026). SHOR consists of 182 human-verified scenarios that cover prompts, tools, workflows, memory, and other components.
Instead of running full multi-step optimization every time, this evaluation measures the step-level decision of which component should be changed first. The paper reports a correlation between ranking performance and actual optimization ability, seeking to distinguish diagnostic ability from an optimizer that improves by chance through trial and error.
However, Priority Ranking does not directly measure the following.
- Correctness of a concrete edit
- Candidate executability
- Final agent score
- Cost and safety
- Cross-environment transfer
Because human priorities serve as the oracle, the benchmark also has difficulty handling interactions among multiple sites or improvements that humans overlook. Use it as a diagnostic rather than a replacement for end-to-end evaluation.
Controlled End-to-End Evaluation
VeRO is a benchmark harness that versions the optimization objective, environment, harness, feedback, and budget, then compares eight optimizer configurations on five tasks over three iterations each (Ursekar, Shanker, Chatrath, et al. 2026). Its main protocol fixes the target agent to GPT-4.1 mini and reports initial score, best score, gain, runtime, and run-to-run variance over 120 experiments. As work accepted at ICML 2026, it is a major prior example of direct evaluation of the optimizer’s update process.
HarnessOpt-Bench is a new preprint that compares optimizers on OfficeQA, BrowseComp-Plus, Terminal-Bench, and GAIA while fixing development, validation, hidden test, case-pass, and target-model token budgets (Ursekar, Shanker, Maurya, et al. 2026).
Its key design choices are as follows.
- Fix the target model, environment, and verifier
- Disclose case-level results and traces during development
- Return an aggregate score during validation
- Seal the hidden test from search
- Evaluate the final candidate and seed repeatedly on the test set
- Evaluate the optimizer configuration itself across multiple campaigns
VeRO and HarnessOpt-Bench both reduce confounding, but their comparisons remain within prepared optimization problems. Real-world permissions, secrets, long-term memory migrations, and environment updates require separate evaluation. HarnessOpt-Bench is an unreviewed preprint released in August 2026 and cannot yet be considered an established standard.
Providing Multiple Optimization Problems
Searching for a harness only within one task benchmark makes optimizer generalization difficult to measure. Prepare multiple optimization problems, each combining a seed harness, target, environment, task, and edit surface. Design the optimizer on some problems and evaluate it on held-out problems.
- Held-out seed harness
- Held-out edit surface
- Held-out target model
- Held-out environment
- Held-out failure mode
This separates the generalization of a single optimized harness from the generalization of the optimizer algorithm.
Placing Acceptance on the Benchmark Side
If a benchmark returns only the final score while each method selects candidates with its own procedure, the acceptance rule also becomes part of the method. Separate two types of tracks.
End-to-End Track
Evaluate the entire optimizer, including proposal, search, and selection. Record the flexibility of the acceptance rule as part of the algorithm.
Fixed-Acceptance Track
Submit a candidate set or a single candidate, and let a fixed promotion contract on the benchmark side decide whether to accept it. This reduces evaluator overfitting and makes proposal ability easier to compare.
Both tracks are necessary. Fixed acceptance alone does not measure actual optimizer operation, while end-to-end evaluation alone cannot separate candidate quality from selection.
Minimal Benchmark Specification
Optimizer-independent evaluation requires the following contract.
| Component | What to Fix or Report |
|---|---|
| Problem | Seed, target, environment, editable and forbidden surfaces |
| Feedback | Information returned during development and validation, query log |
| Budget | Case-runs, tokens, tools, wall time |
| Test | Execution point sealed from the optimizer and researchers |
| Baseline | Incumbent, Best-of-\(N\), test-time scaling |
| Diagnostics | Validity, hit rate, regressions, sample efficiency |
| Promotion | Evaluator, acceptance rule, safety gate |
| Artifact | Optimizer version, candidate lineage, diff, seed |
When using API models or dynamic websites, preserve timestamps, response caches, and environment snapshots, and state which parts cannot be reproduced completely.
What We Know Today
Prompt, workflow, and code search have multiple optimizer families supported by accepted primary research. VeRO implements direct evaluation in accepted work, while Priority Ranking, Rethinking Evaluation, HarnessOpt-Bench, and Harness Updating Is Not Harness Benefit are new 2026 preprints that extend diagnostics, generalization, and capability decomposition. A common problem formulation is beginning to emerge, but it does not yet constitute independent replication or cross-field consensus.
The protocol in Part III therefore is not a “standard consistently followed by existing research.” It is a recommendation that combines gaps in the evaluation of existing methods with recent work on direct evaluation.
Key Points
The final agent score confounds the optimizer, target, seed, surface, budget, and evaluator. Combine controlled end-to-end evaluation with step-level diagnostics, and separate candidate generation from selection. The evaluator measures outcomes, while the acceptance rule controls promotion. Fix both independently of the optimizer. Recent direct-evaluation preprints offer useful designs but should not be treated as established standards.