Learning Candidate Generators
Asking a large language model (LLM) to write revisions is not the same as learning an optimizer. A method can use an LLM with fixed weights to produce revisions, or it can learn the candidate generator itself from the outcomes of repeated revisions. These cases must be distinguished.
The search methods in the preceding chapters update candidates and search history without updating the candidate generator’s weights. APE, OPRO, GEPA, AFlow, and Meta-Harness are examples of this case; RLPrompt and Harness-R1 learn the candidate generator itself. This book excludes updates to the target model’s own weights, but includes learning an optimizer separate from the target model. The boundary is whether the artifact ultimately distributed is a harness external to the model.
What Is Learned
| Category | Weights Updated During Campaign | Persistent Output | Examples |
|---|---|---|---|
| Fixed editor | None | Candidates and search history | Meta-Harness, GEPA |
| Learned candidate generator | Optimizer weights | Candidate-generation policy and selected harness | RLPrompt, Harness-R1 |
| Target-model training | Target-model weights | Updated target model | Outside this book’s scope |
Meta-Harness’s coding proposer is a fixed editor. It reads source code, scores, and traces and generates new harness code, but the proposer weights do not change during search (Lee et al. 2026). GEPA’s reflective mutation is also a fixed editor, and what changes is the population of candidate prompts (Agrawal et al. 2026).
For a learned optimizer, separate the optimizer parameters from the target-model parameters, and evaluate the optimizer’s training performance separately from the held-out performance of the harnesses it generates.
RLPrompt: Learning a Discrete Prompt Generator
RLPrompt learns a policy that generates discrete-token prompts for a frozen target model using soft Q-learning, a form of reinforcement learning (RL) (Deng et al. 2022). It requires neither target-model gradients nor weight updates and returns task rewards to the policy.
Two distinct artifacts persist in this method: the policy weights that generate prompts and the selected discrete prompt. The fact that RLPrompt does not change target weights is not a reason to include weight learning in general within this book’s scope. The relevant question is whether the learned policy is the target agent itself or an optimizer that creates harness candidates.
The Boundary with Query-Specific Generators
FlowReasoner post-trains a meta-agent with a reward that combines execution quality, complexity, and efficiency, then generates a multi-agent system for each user query (Gao et al. 2025). It is relevant because execution feedback is used to learn an optimizer policy.
However, the generated workflow is specific to each query, and the selected workflow artifact is not carried forward as a version for the next independent task. The persistent improvement lies primarily in the meta-agent weights. This book therefore describes it as adjacent work on learned optimizers, but does not count it as a core example of persistent harness updates.
This boundary is determined not by whether a trained model exists, but by whether the generated artifact is carried forward as a version to independent future tasks. Per-query planning, test-time workflow synthesis, and sample-and-select fall under test-time scaling if their outputs are discarded.
Learning Runtime Patches
Harness-R1 trains a specialized harness engineer that generates patches for executable runtime hooks from failure trajectories (Shao et al. 2026). The target agent remains frozen, while the editor is post-trained with supervised fine-tuning and online reinforcement learning.
The edit surface consists of five hooks in the runtime lifecycle. Rather than freely rewriting the entire repository, the editor produces patches for limited interfaces such as context construction, tool mediation, action validation, and recovery. Candidates are compiled, and the same tasks are rerun in a sandbox to compute the reward.
The main paired reward is the difference obtained by rerunning the task batch that produced the failures before and after the patch, minus penalties for invalid patches and increased cost. This makes the local effect of a patch easy to measure, but it is transductive optimization on the same tasks. Read it separately from the paper’s held-out-task and cross-target results.
Harness-R1 is a limited recent example of a learned runtime editor, not the core of learned-optimizer research or an established standard. Its lineage includes earlier work on learning prompt generators, such as RLPrompt, and query-level meta-agents, such as FlowReasoner. Its novelty is expanding the edit surface to runtime code.
Separating Reward from Acceptance
A learned optimizer’s training reward can use dense signals for task deltas, parsing and compilation, cost, preservation, and safety. Combining them as a weighted sum, however, turns the coefficients into an implicit acceptance rule. At deployment, measure quality, regressions, cost, and safety separately, and do not promote a candidate based only on the trained editor’s self-evaluation.
Patch Representations
The output format of a learned editor affects both performance and safety.
| Representation | Advantage | Limitation |
|---|---|---|
| Full file | Highly expressive | Large diffs that can easily remove existing logic |
| Unified diff | Makes changes from the parent explicit | Patch context mismatch |
| AST edit | More likely to preserve syntactic validity | Language- and schema-dependent |
| Hook body | Fixes the interface | Cannot repair failures outside the hooks |
| DSL / operator | Easy to validate | Search space is constrained by the designer |
Harness-R1 constrains the surface with hooks and a patch compiler. Meta-Harness uses a coding agent to edit single-file harness code. DGM changes the entire codebase. When comparing editor capabilities, match not only the model but also the patch representation and permission scope.
The output of a learned editor should also be validated in order for static validity, interface compliance, sandbox execution, targeted reruns, preservation, held-out performance, and safety. Compilation guarantees only syntax and part of the interface, while a sandbox limits external effects but does not establish semantic safety. Keep the evaluator, sealed test, permissions, acceptance threshold, incumbent, and rollback mechanism outside the editor’s change authority.
This completes Part II’s classification of methods. A higher score during search, however, does not establish performance on unseen tasks, transfer to another model or environment, preservation of capability, cost and safety, or the optimizer’s own superiority. Part III separates these questions in that order.
Key Points
Distinguish methods that keep the candidate generator fixed, methods that learn the candidate generator, and methods that train the target model. Methods that hold the target model fixed while learning a separate optimizer are within this book’s scope, but methods that only generate per-query outputs without saving artifacts are outside its boundary. Harness-R1 is a very recent and limited example of a failure-conditioned runtime editor, not a replacement for external validation and acceptance.