Task Splits
Part II examined how to improve a harness. Part III asks whether that improvement remains valid under unseen conditions. Even without training harness weights, the process of proposing revisions, running them on tasks, and selecting the next revision adapts to data. Once the result of a task affects a later proposal or candidate selection, that task cannot also serve as the final evaluation.
Separating Optimization from Final Evaluation
Divide the task set into at least three partitions.
| Partition | What the Optimizer Can See | Purpose |
|---|---|---|
| Development \(\mathcal{D}_{\mathrm{dev}}\) | Tasks, trajectories, errors, scores | Failure analysis and candidate generation |
| Validation \(\mathcal{D}_{\mathrm{val}}\) | Limited scores, or aggregate values if needed | Search, hyperparameters, stopping decisions |
| Sealed test \(\mathcal{D}_{\mathrm{test}}\) | Nothing until the candidate is finalized | Final generalization evaluation |
The important distinction is information flow, not folder names. If the optimizer can read test scores, rerun the test, or put test failures into the next prompt, then that set has become validation.
Treat the entire outer loop that creates the harness as a learning algorithm, and measure generalization on a test that the loop did not access.
Even if individual candidates did not see the test, the research campaign as a whole has adapted to it if the optimizer designer viewed test results and adjusted the algorithm, prompt, or budget.
What Was Held Out
The phrase “held-out task” alone does not specify what was unknown.
| Level | Unknown Element | Example |
|---|---|---|
| Instance-held-out | Another instance in the same benchmark | A different MATH problem |
| Template-held-out | Another format or constraint in the same domain | An unseen instruction constraint |
| Task-family-held-out | Another task type | Search on math and evaluate on code |
| Target-held-out | Another model or model version | Search on GPT-family models and evaluate on Llama-family models |
| Environment-held-out | Another tool API, runtime, permission set, or website | A change to a browser API or database schema |
| Temporal-held-out | Tasks arising after the search | Future issues or an updated website |
A random split of the same benchmark is instance-held-out and does not eliminate benchmark-family leakage. Even a cross-model evaluation is cross-target, not cross-task, if it uses the same tasks. Report each axis separately.
Reusing the Same Tasks
This book uses transductive optimization to refer to settings in which the optimizer repeatedly reuses the same task set or benchmark distribution targeted for deployment. This is not necessarily improper. If the goal is to improve a fixed internal workflow using past logs from that workflow, the setup is reasonable.
The problem is reporting a transductive result as generalization to unknown tasks.
Harness-R1’s same-batch reward and DGM’s benchmark archive search are useful for local improvement and exploration, but they are transductive (Shao et al. 2026; Zhang et al. 2026). Read them separately from the held-out and cross-target results reported elsewhere.
A transductive study should state the following.
- How many tasks were reused and how many times
- Whether the optimizer saw only scores or also trajectories and expected answers
- Whether task-specific exception handling was allowed
- Which artifact will be deployed to production
- When and by whom future tasks will be evaluated
Overusing Validation Tasks
Because validation is used for selection, each iteration returns information to the optimizer. Selecting the validation maximum from candidates \(H_1,\ldots,H_N\) adapts to observation noise and validation-specific patterns.
Overfitting of the model-selection procedure itself has long been recognized (Cawley and Talbot 2010). For repeated adaptive submissions to a public leaderboard, designs such as Ladder have been proposed to return limited feedback (Blum and Hardt 2015). The same problem occurs in harness optimization.
Record the number of candidate-level queries, full passes, amount of information returned, use in stopping decisions, and times a human viewed the results and changed the method. If a person changes the search prompt after seeing a validation score, that person is also part of the optimizer.
Reading Splits in Papers
Read how data is used in candidate generation, selection, stopping, and figure creation, rather than relying on split names. For example, ProTeGi shows a test curve at every optimization step, and an OPRO analysis includes a table that selects the instruction with the highest test accuracy (Pryzant et al. 2023; Yang et al. 2024). These are useful for analyzing the search mechanism but should be distinguished from a final value on a sealed test.
Comparing Against Additional Inference-Time Computation
Harness search uses many target-agent runs. It should be compared with using the same amount of computation for test-time sampling without improving the harness.
For a budget \(B\), compare at least the following three conditions.
- Run the incumbent once
- Run the incumbent multiple times within \(B\) and select / vote
- Spend \(B\) on harness optimization, then run the finalized harness once
If the third condition does not outperform the second, the gain may come from additional attempts rather than a persistent artifact.
Rethinking the Evaluation of Harness Evolution compares harness evolution and test-time scaling under matched conditions on Terminal-Bench 2.1 and reports that improvements that appear large on the same tasks shrink under a disjoint split (Wang et al. 2026). This is early evidence from a single-benchmark preprint, but it demonstrates the need to distinguish persistent artifacts from additional attempts.
Separating Evaluation-Data Leakage
Leakage can occur through several paths.
| Path | What Leaks |
|---|---|
| Target-model contamination | Benchmark problems or answers seen during pretraining |
| Optimizer-model contamination | Known solutions or public agent implementations |
| Feedback leakage | Hidden answers, unit tests, judge rationales, repeated scores |
| Researcher adaptation | Selection of optimizer, prompt, seed, or budget after seeing the test |
| Temporal leakage | “Future tasks” that were public before the model’s training cutoff |
Do not collapse these paths into a single “contaminated/not contaminated” label. Even with a known benchmark, sealed instances, limited feedback, and temporal splits can be combined to reduce information flow.
Implementing a Sealed Test
A sealed test requires more than omitting its name from the optimizer prompt.
- Store test data in storage or a process separate from the optimizer
- Do not give the optimizer read permission
- Have the evaluator accept candidate artifacts and return only aggregate results
- Run it only once after finalizing the candidate
- Do not restart search based on the test result
- Pin and record the environment image and tool versions
- Hide candidate identity from human evaluators
- Prepare a new test set if reevaluation is necessary
If only a public benchmark is available, fix the instance split and its hash before the study begins and have a separate operator manage the test. Treat changes made after viewing the test score as a new study.
Key Points
Treat the entire harness optimizer as an outer-loop learning algorithm, and separate development, validation, and sealed test data by information flow. Repeated improvement on the same benchmark is transductive and differs from generalization to unknown tasks. Report test-time scaling, validation overuse, and target/optimizer contamination explicitly, and decompose the levels of held-out evaluation.