Learned Optimizers
Can we generate suitable instructions or agent configurations for a new problem without testing many candidates every time? Here, the learned component is not the target model solving problems, but the generator of prompts or execution structures supplied to it. Even with a fixed large language model (LLM), the generator’s weights can be trained. This distinction makes harness optimization and model training compatible rather than mutually exclusive.
Training a generator does not, however, mean learning the entire optimizer, including search, evaluation, and acceptance. Checking generated code for executability, deciding which candidates to retain, and restarting search after failure often remain separate fixed procedures. This chapter first compares discrete-prompt policies with preference learning, then turns to question-specific multi-agent system generation. Its central studies are RLPrompt at EMNLP 2022, MePO at EACL 2026, MAS-GPT at ICML 2025, and MAS-Architect at ICML 2026. FlowReasoner, a workshop paper, and Harness-R1, a preprint, supplement them by illustrating differences in learning targets and evaluation.
From Searching for Candidates to Learning to Generate Them
For a mathematics problem, possible configurations include solving alone, having another agent check the result, or combining several solutions. In outer-loop search with a fixed LLM proposer, candidates are executed and their results returned to the history to inform the next proposal. The history and candidate collection change, but the proposing model’s weights do not. A trained generator instead changes the probability of its next configuration using past problem–configuration pairs or rewards from executing configurations.
Let \(q\) be a problem, \(h\) the generated instruction, configuration, or code, \(G_\theta(h\mid q)\) the generator, and \(T_\phi\) the fixed target model. With execution reward \(R\), reinforcement learning (RL) of the generator conceptually takes the following form.
\[ \max_\theta\;\mathbb{E}_{q\sim D,\,h\sim G_\theta(\cdot\mid q)} \left[R\bigl(\operatorname{Exec}(T_\phi,h,q)\bigr)\right], \qquad \phi\text{ is fixed}. \]
This equation does not equate the papers’ individual losses. Supervised learning imitates token sequences for good configurations. Preference learning adjusts the relative probabilities of preferred and dispreferred rewrites. RL updates generation probabilities from execution rewards. None necessarily requires numerical gradients through the target. Even when outputs are discrete code or strings, gradients can be computed for the weights of the policy generating them.
Learning a prompt shared by all problems also differs from generating a configuration per question. The former can distribute a string once learned. The latter requires a generator call for every question, and the execution costs of the resulting agents remain. Table 1 compares these output units and supervision signals.
| Method | Learned Component and Output | Training Data and Objective | Work Remaining on New Problems |
|---|---|---|---|
| RLPrompt | Discrete tokens generated through a small policy module | Soft Q-learning with rewards from few-shot labels or style evaluation | Classification uses a learned shared prompt. Generation tasks also include input-conditioned generation |
| MePO | A 7B prompt-rewriting model | Direct preference optimization on preference pairs derived from Alpaca and BPO | Rewrite the question and pass it to a fixed answering model |
| MAS-GPT | A 32B question-conditioned system generator | Supervised learning on question–code pairs constructed through execution evaluation and consistent mappings | Execute the system obtained in one generation |
| MAS-Architect | A model generating declarative structures and node implementations | Initialization on correct teacher code, followed by RL using validity, cheating checks, and correctness rewards | Generate and execute a question-specific structure and implementation |
Learning Discrete Prompts from Rewards or Preferences
Instead of directly replacing individual prompt words, we can train a policy that selects the next word. RLPrompt treats previously generated prompt tokens as the state, the next token as the action, and the target LLM’s result with the completed prompt as the reward. The target model is fixed, as is the body of a small pretrained generator model. Only a small multilayer perceptron (MLP) transforming its internal representations is trained, and the original vocabulary output layer produces the next-token distribution. The paper uses on-policy soft Q-learning. (Deng et al. 2022, secs. 2.2–2.3)
The white model components in Figure 1 remain fixed, while only the small red policy module is updated from rewards. The two target models shown are a masked model and a left-to-right generative model, illustrating how the same token-generation mechanism connects to different targets.
For classification, a piecewise reward uses the gap between the correct label’s probability and the strongest incorrect label’s probability, placing greater weight on correct classifications. Maximizing only the correct-label probability could reward prompts that ignore the input and assign high probability to one class. For style transfer, rewards assess content preservation and target-style strength. Rewards for several prompts on the same input are normalized by their mean and standard deviation to distinguish input difficulty from prompt quality. (Deng et al. 2022, sec. 2.4)
After training, classification can use a greedily generated discrete string without rerunning the generator each time. Input-conditioned style transfer, by contrast, includes prompt generation per input. The strings need not be natural instructions and may contain ungrammatical word sequences. Discrete representation does not imply clear human-readable meaning.
Experiments likewise require separating a fixed target model from inexpensive inference. On Yelp sentiment style transfer with GPT-2-xl, a five-token RLPrompt scores 61.4 on the main metric \(J\), combining content, style, and fluency, versus 53.4 for a handwritten prompt. Both conditions, however, generate 32 output candidates and select the one with the highest reward. This supports prompt learning under that setup, not equivalent performance from a single output. (Deng et al. 2022, Table 3, §3.2)
Another approach collects preference pairs of good rewrites in advance rather than repeatedly executing the target for rewards. MePO specifies four properties: clarity, precision, concise reasoning instructions, and preservation of the original information. A lightweight model revises prompts accordingly. Starting from Alpaca and BPO, it retains pairs where DeepSeek-R1 rates the revised answer above the original and the pair meets specified quality thresholds. The dataset contains 40,151 examples, including deliberately degraded inputs simulating imperfect user instructions. (Zhu et al. 2026, secs. 2–3, Table 1)
Figure 2 shows three stages: discovering properties, constructing revision data, and training the rewriter. The left-hand bars show property frequencies, not accuracy or the effect of removing each property. The learned component on the right generates rewritten input instructions, not answers themselves.
Training starts from Qwen2.5-7B-Instruct and uses direct preference optimization (DPO) to make revised prompts more likely than their originals. Answers provide evidence for selecting data. The answering model itself is not trained with DPO. At evaluation, only the test-question portion is rewritten, while demonstrations and other material remain fixed. Unlike RLPrompt’s continual rewards from a particular target, this design learns offline to rewrite diverse questions.
The design allows a local rewriter without an external optimization API at deployment. Data filtering and property discovery still use large evaluation models. Local operation therefore does not imply development without external models. Nor are the four properties universal optimality conditions for every task.
MePO’s Table 4 also compares a training-free condition where static property instructions ask the answering model itself to rewrite the question. Averaged over five task families, Qwen3-8B scores 76.28 without training and 76.88 with MePO. LLaMA2-7B-chat scores 37.13 and 39.66, respectively. Training consistently adds value in these comparisons, but its magnitude depends on the target. Table 5’s property ablations use the training-free template setting. They do not remove individual properties from DPO training data. (Zhu et al. 2026, secs. 4.4–4.5)
Moving Question-Specific Systems from Imitation to Search
When one string poorly represents multi-agent roles and communication, executable code can provide supervision. MAS-GPT represents a multi-agent system (MAS) as a Python function that takes a question and returns a final answer. Variables correspond to role-specific prompts, function calls to LLM execution, and string concatenation to information transfer between agents. The generator outputs this executable system rather than the answer itself. (Ye et al. 2025, sec. 3)
Teacher-data construction is crucial. First, verifiable questions and more than 40 basic configurations are prepared. Question–configuration combinations are executed and labeled for correctness. Indiscriminately assigning many different correct configurations to one question would make supervision ambiguous. Similar questions are therefore grouped, and a configuration with strong aggregate performance within each group is selected as its representative. A stronger LLM then adjusts roles for each question and adds an explanation of why the configuration fits. Revised configurations are rerun and accepted when performance does not worsen.
The resulting 11,442 question–explanation–code examples are used for supervised fine-tuning of Qwen2.5-Coder-32B-Instruct. A new question needs one generation to create a system, but that does not reduce all subsequent agent calls to one. Although trained from initial configuration families, it generates code rather than acting only as a classifier choosing one member of that collection at inference. (Ye et al. 2025, Tables 1-2)
With Llama-3-70B-Instruct as the execution model, Table 1 reports an eight-benchmark average of 59.83 for the standalone model and 65.47 for MAS-GPT. On HumanEval and SciBench, treated as outside the training domains, scores rise from 79.01 to 80.25 and from 21.05 to 24.21, respectively. On GSM8K, however, self-consistency scores 94.99, above MAS-GPT’s 93.39. Training the generator does not surpass existing configurations on every task. Out-of-domain scores measure transfer by the system combining a trained generator and execution model. They do not establish complete novelty relative to the foundation models’ pretraining.
Imitating teacher configurations alone provides no active improvement signal for structures the teacher did not demonstrate. MAS-Architect changes both the representation and learning method. Its declarative structure layer specifies nodes, conditional edges, loops, and shared-state types. Its implementation layer realizes each node’s roles, tools, and reasoning procedures. This separates generation of connections from generation of computations more explicitly than embedding both in a single sequence of imperative code. (Huang et al. 2026, secs. 3.1–3.2)
Training has two stages. First, declarative code from a large teacher is executed, and only correct examples are retained for supervised fine-tuning. Then multiple candidates are generated for the same question and checked for compilation and execution, cheating such as embedded answers, and final-answer correctness. RL updates the configuration-generation policy from these rewards. Cheating or invalid candidates receive 0, valid but incorrect candidates 0.1, and correct candidates 1. The updated component is the configuration generator, not the target model driving generated agents. Supervised training uses MMLU and HotpotQA, while RL uses those datasets plus GSM8K. (Huang et al. 2026, sec. 3.3, Eq. 8, §4.1)
With Qwen3-4B-Instruct-2507 for execution, Table 2 reports MATH accuracy of 82.1 for the standalone condition, 83.3 after supervised training, and 89.5 after RL. This demonstrates improvement beyond imitation without including MATH in the training data. It is a within-method stage comparison involving both declarative representation and RL, not an isolated measurement of representation alone. GPQA differs between 63.8 in the main results of Table 1 and 64.1 in the stage comparison of Table 2, so these must not be mixed into one numerical series. (Huang et al. 2026, Tables 1-2)
FlowReasoner provides another example of further training through external execution. We treat it as a workshop paper and do not place the cited arXiv results on the same footing as main-conference publications. It initializes by distilling multiple rounds of configuration reasoning from DeepSeek-R1, then updates its generator with group relative policy optimization (GRPO) using generated-code execution feedback. The reward combines test pass rates, abstract-syntax-tree-based complexity, and diversity. Inference also includes multiple rounds of configuration refinement using external tests, so its inference budget differs from MAS-GPT’s one-shot generation. In the 14B condition of Table 2, Overall across three coding tasks rises from 81.50 after supervised learning to 81.89 after RL. This does not support an invariably large additional effect from RL. (Gao et al. 2025, secs. 4.1–4.3, Table 2) The three stages in Figure 5 distinguish imitating teacher reasoning from subsequently improving the generator using execution rewards.
Separating Reusable Procedure Generation from Execution-Policy Learning
Learned generators need not create only new question-specific systems. MetaFlowLLM takes two existing tasks and their procedures and generates a higher-level procedure that retains common parts as static operations while delegating differing parts to dynamic agents. MetaFlowGen is initialized with teacher examples whose merged procedure passes tests for every source task. It then uses GRPO with a reward combining the fraction of source tasks solved correctly and the fraction of static operations retained. The latter is relative to the smaller static-operation count of the two input procedures. It is not direct minimization of LLM call counts. (Fan et al. 2025, sec. 3.2, Eqs. 3-5)
Here, training generator weights coexists with updating an experience tree that merges and retrieves generated procedures. Adding nodes to the tree is not itself weight training. Experience Reuse covers the memory unit and retrieval and execution details.
Research training a model to plan during execution should also remain in the comparison. AgentFlow connects a planner, tool executor, verifier, and answer generator through shared memory. The planner selects subgoals and tools. Flow-GRPO distributes final-answer correctness to individual turns and updates the planner from actual multi-turn executions. It generates actions executed immediately, not a configuration later applied to another fixed target. It therefore learns the target-side execution policy’s weights rather than a harness generator for a fixed target. (Li et al. 2026, secs. 3.1–3.2) The trained/frozen distinction in Figure 6 locates learning in the planner rather than in an outer generator of reusable system code.
CoMAS likewise does not learn an outer code generator. Agents exchange answers, critiques, and scores, with complementary rewards assigned to answers and critiques from the scores. A highly rated answer rewards its author, while discovering a fatal error rewards the critic. Invalid scoring formats are penalized, and REINFORCE++ updates the participating agents’ own weights. Interaction provides learning signals without requiring external correct answers every time, but internal scoring may still diverge from actual correctness. Even under the common aim of improving agents, we must specify whose weights change. (Xue et al. 2026, secs. 3.1–3.3) The transition from discussion to reward formulation in Figure 7 shows why the scoring interaction itself becomes part of the learning signal.
Evaluating Learned Runtime Patches
Extending outputs from system configurations to runtime patches makes the gap between plausible and effective changes more consequential. Harness-R1, a preprint, trains a specialized model to generate code intervening at episode start, before decisions, before actions, and after feedback, using failure records from a fixed target. Patches supplement input context, normalize or reject actions, and recover from stalled states without changing target-model weights. (Shao et al. 2026, sec. 3)
Initialization uses supervised learning on valid teacher patches that do not worsen performance. During RL, eight candidates are generated from one failure summary. Each is independently installed and the fixed target rerun. The reward is the difference in mean reward over the same complete batch, including originally successful cases as well as failures. Invalid candidates, candidates with no substantive change, and incomplete evaluations receive 0. Measuring changes that break previously successful tasks is important.
This training reward measures improvement on the same batch, not generalization to unseen tasks itself. A separate experiment generates one patch from only 10 failures per environment and applies it to tasks absent from that evidence. Figure 8 shows this evaluation of patch reuse.
This pools the remaining 1,270 tasks across WebShop, ALFWorld, and DBBench. It differs from a separate aggregate that weights environments equally. The changes are +8.9 ± 1.5, −4.3 ± 2.5, and −0.4 ± 3.6 percentage points, respectively. With matched targets and failure evidence, this illustrates a difference between a trained patcher and large fixed patchers. It does not establish independent replication under the same conditions or a general standard evaluation. (Shao et al. 2026, sec. 4.4, Table F.1)
Shifting Costs and Generalization Requirements Through Training
Training a generator shifts search on each new problem into training-time data collection and execution evaluation. Even when generation becomes one-shot, as in MAS-GPT, teacher question–configuration combinations must first be executed. RL requires target execution per candidate, while offline methods such as MePO still incur preference-data and evaluation-model costs. Whether training costs can be amortized across many uses depends not only on usage volume, but also on the stability of target models and tool specifications.
Generalization has two levels. First, do generated instructions or configurations help on new tasks? Second, can the learned generator produce effective configurations for new target models, tools, and failure types? RLPrompt directly uses rewards from a particular target, MePO uses general rewriting pairs, and MAS-GPT and MAS-Architect learn executable structures. Their data and evaluators bear different generalization requirements. A single transfer score obscures these distinctions.
The remaining scientific question is when a trained generator outperforms outer-loop search with a fixed LLM under matched targets, usage counts, data, and total budgets including training and search. Current results demonstrate learnability and effectiveness in specific settings, not that training the optimizer always improves outcomes.
Figure Credits
Figure 1: (Deng et al. 2022). Original published version, Figure 1. CC BY 4.0. Cropped and rasterized.
Figure 4: (Zhu et al. 2026). Original published version, Figure 4. CC BY 4.0. Cropped and rasterized.
Figure 3: (Ye et al. 2025). Original published version, Figure 3. CC BY 4.0. Cropped and rasterized.
Figure 2: (Huang et al. 2026). Original published version, Figure 2. CC BY 4.0. Cropped and rasterized.
Figure 4a: (Shao et al. 2026). Original preprint, Figure 4a. CC BY 4.0. Relevant panel cropped.
Figure 3: (Gao et al. 2025). Original paper PDF, Figure 3. Cropped and rasterized.
Figure 2: (Li et al. 2026). Original paper PDF, Figure 2. Cropped and rasterized.
Figure 2: (Xue et al. 2026). Original paper PDF, Figure 2. Cropped and rasterized.