Target and Environment Generalization
The preceding chapter separated tasks used for improvement from tasks reserved for final evaluation. This chapter asks whether a harness improved for one model still works with another model or after a tool update. Its prompt may depend on model-specific instruction interpretation, its workflow may assume an old error format, or its validator may accept only an old schema. It distinguishes transferring the same harness from rerunning the same optimizer.
Separating What Changed
Consider harness performance as the following function.
\[ J(H;\theta,E,\mathcal{D},v), \]
Here, \(\theta\) is the target model, \(E\) is the environment including tools and runtime, \(\mathcal{D}\) is the task distribution, and \(v\) is the version of the harness interface. Distinguish evaluations that change only one factor from those that change several factors at once.
| Dimension | Example Change | Primary Failure |
|---|---|---|
| Target model | Model family, size, provider | Instruction interpretation, tool-call format, context limit |
| Model version | API update, checkpoint update | Sampling, refusal, schema adherence |
| Environment | Website, database, simulator | Observation, state transition, latency |
| Tool contract | Name, argument, return schema | Invalid call, silent misparsing |
| Permission | Read/write/network scope | Denial, privilege violation, faulty fallback |
| Task distribution | Domain, difficulty, language | Benchmark-specific shortcut |
| Harness interface | Hook, memory schema, runtime | Patch incompatibility |
A cross-target evaluation changes \(\theta\) while using the same \(H\). An experiment that compares \(H_\theta\) obtained by rerunning the search for each target is multi-target optimization, not artifact transfer.
What Was Transferred
Harness optimizer transfer takes three forms.
- Artifact transfer: Apply the same harness to a new target / environment
- Optimizer transfer: Use the same optimizer to search for a harness for a new target
- Design transfer: Use a module or pattern found in one campaign to initialize another campaign
Applying the same patch unchanged to another model is artifact transfer; rerunning the search for that model is optimizer transfer; using only the patch’s design principle as an initial proposal is design transfer.
Conflating these forms makes it impossible to determine what generalized. Even if a learned editor can generate a patch for a new target, the original patch itself has not necessarily transferred.
What Existing Research Transferred
A result that improves on multiple models or domains does not by itself reveal whether the same artifact transferred or the optimizer was rerun. The following table organizes representative studies by the unit of transfer.
| Study | What Was Transferred | Interpretation |
|---|---|---|
| RLPrompt (Deng et al. 2022) | Discrete prompt learned on the source model | Artifact transfer between RoBERTa and GPT-2; results differ by direction |
| ADAS (Hu et al. 2025) | Discovered agent design | Design transfer across domains and models; distinguish prompt or adapter retuning |
| DGM (J. Zhang et al. 2026) | Coding-agent code or search method | Distinguish artifact transfer between SWE-bench and Polyglot from a new search |
| Meta-Harness (Lee et al. 2026) | Coding proposer | Optimizer transfer that searches a separate harness for each domain |
| Harness-R1 (Shao et al. 2026) | Trained editor and common hooks | Cross-target within a shared runtime, not transfer to a new tool API |
| Self-Harness (H. Zhang et al. 2026) | Optimization recipe | Rebuilds a harness for each target model; distinct from transferring the discovered harness itself across models |
When a prompt optimizer is run with multiple models, generating new candidates for each model is not artifact transfer. Even if the same architecture is claimed, show in a diff which prompts, adapters, or validators could be changed at the destination. Code artifacts depend on tool paths, containers, test commands, and repository layouts, so an experiment that changes only the model does not measure runtime portability.
Changing the Environment
A cross-environment evaluation holds the target model fixed while changing the tools and state transitions.
Organize environment differences into levels.
| Level | Change | Example |
|---|---|---|
| Patch-level | Bug fix, minor version | Change in an error message |
| Contract-compatible | New field, ordering, latency | Optional schema field |
| Contract-breaking | Tool name, argument, authentication | API migration |
| Behavioral | Website policy, simulator dynamics | Change in the result of the same action |
| Adversarial | Prompt injection, malicious tool output | Untrusted observation |
Within the primary studies reviewed in this book, systematic evaluation of contract-breaking tool APIs or permission changes is limited. Cross-environment generalization remains an open problem.
Measuring Dependence on Target–Environment Combinations
The effects of a harness and target need not be additive. Evaluate the same candidate and incumbent over a target-by-environment matrix, and report the worst case and regression count in addition to the mean. If a harness improves greatly on one model but regresses on another, it is target-specific.
Robustness to Version Updates
Provider models and tool APIs can change even when their names remain the same. A cross-target study should record not only the model ID but also the checkpoint, API date, decoding parameters, and system policy.
On a new version, reevaluate the incumbent and candidate under matched conditions and break down output schema, tool calls, latency, and refusals. Evaluator quality may also change across languages or domains, so use a human-verified subset or a domain-specific executable verifier. Rather than treating a harness as a universal artifact, attach a compatibility matrix to its version metadata.
Minimal Transfer Evaluation
To evaluate the transfer of a single artifact \(H^*\), hold the following conditions fixed.
- Finalize \(H^*\) using only development data from the source target / environment
- Freeze the candidate source, history, and scores
- Measure zero-shot transfer without editing prompts or code at the destination
- If needed, measure adapted transfer using a small destination validation set separately
- Evaluate the incumbent \(H_0\) at the destination as well
- Report the mean, worst case, regression count, and invalid count
- Include a separate case in which the target model and environment change simultaneously
Do not place zero-shot transfer and adapted transfer in the same table column.
Key Points
Evaluate cross-target, cross-environment, and cross-task generalization as separate dimensions. Rerunning the same method for each target measures optimizer transfer, not the transfer of a single artifact. The primary studies reviewed here provide limited examples of cross-model and cross-domain transfer, and only limited cross-environment evidence involving new tool APIs, runtimes, or permissions.