Evaluating Harness Improvements
A higher evaluation score on familiar tasks does not guarantee better performance on unseen tasks. When evaluating harness improvements, examine improvements, generalization to unseen tasks, and regressions separately.
Break It Down into Three Questions
Ask the following three questions when evaluating a candidate.
- Did it improve performance on tasks the optimizer saw?
- Did it also improve performance on tasks the optimizer did not see?
- Did it avoid regressing existing successes, increasing cost or latency, or expanding permissions?
The first question alone is insufficient. If the same tasks are used repeatedly, the optimizer can tailor the harness to those tasks. This is adaptation to observed tasks, not improvement on unseen tasks.
Roadmap to the Later Chapters
The three questions become separate protocols in the later chapters.
| Question | Where It Is Developed |
|---|---|
| Did performance improve on observed tasks? | Part II’s Optimization Loops and Execution Feedback and Failure Analysis |
| Did performance improve on unobserved tasks? | Part III’s Task Splits and Target and Environment Generalization |
| Did the update preserve existing capabilities and operating constraints? | Part III’s Regressions, Cost, and Safety |
The governing principle here is simple: any information used for candidate generation, selection, or stopping is optimization data and must be separated from final evaluation under unseen conditions. Part III defines the development / validation / sealed-test information flow, search-budget accounting, promotion contract, and rollback protocol.
Key Points
Break improvement into three questions: gain on observed tasks, generalization to unseen conditions, and regression. Part III defines the concrete split and promotion protocol.