Experience Reuse
Must an agent reconsider a job from scratch after solving it once? Even without changing a large language model’s (LLM’s) weights, external records of past actions, lessons from failures, and reusable procedures can change subsequent decisions. Accumulating records does not, however, establish better future performance. This chapter examines what experience becomes, how it is updated and selected, and how its usefulness is tested.
Memory here means external state consulted in subsequent execution. It includes both facts saved from conversations and skills describing how to perform work. For harness optimization, the particularly important case is improvement to guidance or procedures reused across jobs, rather than individual answers. Information retrieval over long conversations is important adjacent research for understanding update and retrieval mechanisms. We distinguish its evaluation objectives explicitly.
Retrying the Same Task and Transferring Experience to Another
First distinguish passing an explanation of failure to the next attempt from accumulating knowledge across tasks. Reflexion converts environmental success or failure signals and execution outcomes into natural-language reflections that condition the next attempt. This is not reinforcement learning through model-weight updates. Representative evaluations involve multiple attempts at the same problem, so improvements include retry computation as well as reflection. Such within-task improvement is useful, but alone does not establish better shared procedures carried forward to unseen tasks. (Shinn et al. 2023)
The feedback loop in Figure 1 conditions another attempt at the same task rather than establishing transfer to unseen tasks.
ExpeL converts training-task experience into two forms used on other tasks. One is successful execution traces, retrieved as concrete examples of similar work. The other is insights: natural-language rules guiding behavior, extracted by contrasting successes and failures or examining multiple successes. Evaluation supplies a list of insights and the highest-ranked successful traces retrieved by task similarity. Recalling what was done and abstracting what deserves attention are separate routes. (Zhao et al. 2024)
Insights are not simply generated once and frozen. Add, edit, upvote, and downvote operations update the list, and entries whose importance counts reach zero are removed. Successful traces can contain unnecessary steps, so a rule extracted from one example is not preserved unconditionally forever. Insight extraction primarily uses GPT-4-0613, whereas the acting agent at evaluation uses GPT-3.5-turbo-0613. The model producing knowledge is distinct from the model using it. (Zhao et al. 2024)
Figure 2 separates these two routes from within-task retries. Passing only successful traces to later tasks provides example retrieval. Adding insights also transfers decision rules beyond specific examples.
The paper’s ALFWorld comparison makes this distinction concrete. With GPT-3.5-turbo-0613 and four-fold validation, success before retries is 40.3% for ReAct, 54.5% for ExpeL retrieving only successful traces, and 59.0% for ExpeL also using insights. After three Reflexion retries, the respective values are 54.4%, 60.4%, and 64.2%. Carrying retrieval and insights forward is compatible with retrying the same task, but they are not the same improvement. This is the preliminary retry experiment in Table 2, and its results should not be collapsed into one success rate that ignores attempt counts. (Zhao et al. 2024)
Comparing rows in Figure 3 changes how experience is supplied. Comparing columns changes additional attempts at the same job. Comparing only the R0 baseline with full ExpeL at R3 therefore cannot attribute the entire difference to insights.
Converting Traces into Parameterized Procedures
Concrete examples retain details but also carry values irrelevant to the next job, such as order numbers or screen elements. Conversely, a short insight such as “check carefully” loses the operation sequence. An intermediate representation is a reusable workflow that replaces specific values with variables.
Agent Workflow Memory (AWM, ICML 2025) extracts recurring subprocedures from successful action traces. The order lookup in Figure 4 generalizes operations on a particular order number into a procedure taking that number as a variable. Alongside a goal description, the procedure retains environmental states, decisions, and actions at each stage. It is neither a copy of a successful example nor a short summary of its goal alone. (Wang et al. 2025)
Extracted procedures are added to the agent’s context to guide generation of subsequent actions. Even when a procedure contains action code, it is not automatically executed as one new tool. The basic experiments also manage procedure collections by website to maintain relevance, rather than selecting one universal skill from all past experience on every occasion. (Wang et al. 2025)
AWM has two evaluation settings. Offline, procedures are constructed from training examples in advance and the same memory is used throughout testing. Online, test problems are processed sequentially, with procedures induced from executions judged successful by a model evaluator and passed to the next problem. The latter uses no reference labels, but mistaken success judgments can preserve unsuitable procedures. Later problems also benefit from knowledge acquired on earlier ones, so this assesses something different from independent testing with fixed memory. (Wang et al. 2025)
Comparisons within the paper do not support the simple claim that abstraction always raises success rates. On WebArena, rule-based extraction and deduplication of concrete traces and LLM-based abstraction of subprocedures have nearly identical success rates. Their difference appears mainly in step counts. On Mind2Web, better element selection can coexist with a decline in the F1 score measuring action type. Following a procedure can constrain the agent when the current screen calls for a different action. (Wang et al. 2025)
| Within-Paper Setting | Representations or Uses Compared | Measurements | Supported Interpretation |
|---|---|---|---|
| AWM, online WebArena, GPT-4o-2024-05-13 as specified in the text | Rule-based extraction / LLM abstraction | Success 35.6 / 35.5%, mean steps 6.3 / 5.9 | No success-rate advantage for abstraction, but a difference in steps |
| AWM, Mind2Web cross-task, GPT-4, offline | MindAct / AWM | Step success 36.2 / 45.1%, action F1 60.6 / 57.3 | Overall step success improves while action F1 declines |
| Dynamic Cheatsheet, 30 AIME 2024 problems, Claude 3.5 Sonnet, sequential adaptation | Empty memory with instructions and tools / cumulative memory / retrieval and synthesis | Accuracy 36.7 / 50.0 / 46.7% | Memory exceeds a strong memory-free baseline, but retrieval is not always best |
Selecting Experience Granularity and Retrieving Relations
Even parameterized procedures mix reusable parts with parts requiring fresh reasoning for the current task. MetaFlowLLM (NeurIPS 2025) incorporates this distinction into execution structure. A MetaFlow combines static steps that directly execute tool calls or code with dynamic steps delegating subtask solving to agents. Both read and write shared state. Compared with AWM’s in-context procedures, the reusable executable components are more explicit. (Fan et al. 2025)
In the experience tree built from past jobs, nodes nearer the leaves contain more concrete static steps, while nodes nearer the root delegate more work to dynamic solving. The tree is built by hierarchically merging MetaFlows, and a suitable node is retrieved for a new task. The method chooses a granularity that maintains relevance, rather than always using the most specific experience. AppWorld measures task goal completion requiring passage of all evaluation tests. WorkBench measures unintended side effects as well as correctness. Thus evaluation concerns execution outcomes, not just reuse efficiency. (Fan et al. 2025)
Because the method also trains the weights of its generator, MetaFlowGen, calling the entire system a memory method without weight changes would be inaccurate. Experience-tree representation and retrieval are distinct from training the generator that constructs tree nodes. Generator training is discussed in Learned Optimizers. The abstract’s average improvements of 32.3% and 6.2% are not used here to combine experiments with different units, and percentages must not be reinterpreted as percentage points. (Fan et al. 2025)
When experience spans long conversations or multi-agent collaboration, abstracting procedures is not the only challenge. The relations followed to reach relevant facts or collaboration failures also matter.
A-Mem (NeurIPS 2025) converts new conversations and interactions into notes containing content, timestamps, keywords, tags, and contextual descriptions. It finds nearby existing notes through embeddings and assesses their relationships. Besides creating links, it updates existing notes’ contextual representations and attributes. Retrieval uses this organized memory rather than raw history. The main evaluation is question answering over long conversations, not an experiment improving action procedures, but it demonstrates a write-time mechanism that reorganizes indexes and existing knowledge. (Xu et al. 2025)
The write-time paths in Figure 6 show why retrieved context depends on memory organization, not only similarity search.
In LoCoMo experiments with GPT-4o, A-Mem’s temporal-reasoning F1 is 39.41, versus 9.09 for the comparison LoCoMo method. On single-hop questions, however, F1 is 48.43 versus 61.56, so it does not dominate every question type. Input token lengths in the same Table 1 are 1,216 versus 16,910, but these are not total costs including writes and note updates. Retrieving necessary relationships in a short context is different from preserving every piece of historical information. (Xu et al. 2025)
G-Memory (NeurIPS 2025) separates collaborative multi-agent experience into three graph layers: insights, queries, and interactions. From a new query, it finds relevant past queries, traverses upward to general insights, and downward to necessary portions of interactions. All three layers are updated after completion, allowing role assignments and intermediate exchanges, not just conclusions, to inform later collaboration. Whereas A-Mem flexibly develops relationships among notes, G-Memory hierarchically organizes memories at different granularities for retrieval. (G. Zhang et al. 2025)
The upward and downward traversals in Figure 7 retrieve general insights and selected interaction details rather than the complete history.
The paper evaluates ALFWorld, SciWorld, Planning Domain Definition Language (PDDL) tasks, HotpotQA, and FEVER with multiple models and collaboration frameworks. For example, ALFWorld success with Qwen2.5-14B and MacNet rises from 58.21% without memory to 79.10% with G-Memory. This compares complete memory configurations, not the isolated effect of graph representation. Interaction compression, role-specific presentation, and insight extraction change together. (G. Zhang et al. 2025)
Caching Plans and Developing Problem-Solving Playbooks
Experience has value beyond accuracy. Avoiding repeated planning by an expensive model for similar jobs is also a clear optimization objective. Agentic Plan Caching (APC, NeurIPS 2025) stores execution-plan templates rather than final answers. It removes unnecessary reasoning and instance-specific values from successful executions and stores the result with keywords representing task intent. (Q. Zhang et al. 2025)
For a new request, a lightweight model extracts intent. A hit instantiates the template for current data and surroundings. A miss invokes a large planning model to create a new plan, and the completed experience returns to the cache. Unlike a cache returning a previous output based only on semantic similarity, plan reuse still leaves the work of obtaining an answer from the target data. (Q. Zhang et al. 2025)
Across five workloads, the authors report average cost reductions of 50.31% and latency reductions of 27.28%, retaining 96.61% of the paper’s optimal-accuracy baseline. 96.61% is relative accuracy retention, not absolute accuracy, and does not mean zero accuracy loss. Read the objective of memory use in relation to cost and latency, not success rate alone. (Q. Zhang et al. 2025)
Dynamic Cheatsheet (EACL 2026), by contrast, sequentially edits problem-solving strategies, code snippets, and cautions. Its cumulative variant, DC-Cu, answers using current memory, then curates and updates it. Its retrieval-and-synthesis variant, DC-RS, retrieves relevant examples from past inputs and outputs and updates memory for the current problem before answering. The curator decides what to retain, revise, or delete without seeing reference labels. This differs from preserving the complete raw history or simply retrieving previous examples unchanged. (Suzgun et al. 2026)
Besides the AIME results in Table 1, GPT-4o achieves 10.0% on Game of 24 with the basic prompt, 19.0% with empty memory plus instructions and tools, and 99.0% with DC-RS. The paper describes preserving and reusing a Python search procedure acquired during execution. Comparison with the empty-memory baseline is essential before assigning the gain to memory, and cumulative memory sometimes outperforms retrieval, as on AIME. Rather than uniformly reading expressions such as “9%” in the abstract as nine percentage points, check each table’s conditions and values. (Suzgun et al. 2026)
Agentic Context Engineering (ACE, ICLR 2026) accumulates this practical knowledge as a playbook containing detailed procedures and cautions. It separates a Generator that executes, a Reflector that analyzes successes and failures, and a Curator that integrates knowledge. Entries have identifiers and helpful/harmful counts and are updated incrementally. Instead of rewriting the whole playbook, it adds entries, makes local revisions, and removes duplicates. (Q. Zhang et al. 2026)
This design addresses context collapse, in which repeated shortening erases accumulated detail. Short summaries are not inherently bad. The problem is losing specific exceptions and tool-use details that should persist when regenerating the whole artifact. Dynamic Cheatsheet’s selective curation can also be more useful than retaining full text, but ACE further limits the update unit. (Q. Zhang et al. 2026)
The update mechanism alone cannot guarantee correct experience. In ACE’s online financial-analysis experiments with DeepSeek-V3.1, FiNER accuracy is 76.7% with reference labels and 67.3% without them, below the pre-adaptation 70.7%. Formula scores, in the same order, are 76.5% and 78.5%, versus 67.5% before adaptation. Table 2 shows that the same update framework can reverse its effect depending on task and feedback. Tasks with observable execution success should be distinguished from tasks where plausible but incorrect explanations can enter memory. (Q. Zhang et al. 2026)
Turning Writes into Validated Updates
These studies do not use a uniform notion of experience. ExpeL retains successful traces and insights, AWM parameterized subprocedures, MetaFlowLLM hierarchies of static and dynamic processing, A-Mem and G-Memory relational memories, APC plan templates, and Dynamic Cheatsheet and ACE editable collections of solving methods. The retained representation determines both what needs updating and the unit retrieved.
Distinguish three operations: writing to memory, evaluating the complete system including those writes, and optimizing by comparing individual changes and deciding whether to accept them. Changes to note attributes or additions of experience are not themselves evidence of measured improvement. Conversely, memory research evaluating usefulness on later tasks has value even without a validation gate for each update.
SkillOpt, a 2026 preprint, explicitly addresses acceptance. It proposes additions, deletions, and replacements to external skill documents for a fixed agent from execution records, limits edit size, and accepts a change only if its score strictly improves on a separately reserved validation set. Rejected edits are also retained as negative examples for future proposals. Writing a skill is thus separated from accepting it after measurement. This validation set is repeatedly used for selection, however, and must not be equated with a final unseen test. Read this as current author-reported experimentation, without generalizing with the same confidence as for a peer-reviewed conference paper. (Yang et al. 2026)
The gate in Figure 11 separates proposing edits from accepting measured improvements; its feedback still belongs to selection, not final testing.
The term test-time learning also requires an evaluation protocol. Dynamic Cheatsheet and online AWM change state while processing an evaluation sequence. Even without seeing correct answers, this differs from applying a fixed procedure to independent unseen problems. Problem order, initial memory, update counts, and additional curation and retrieval costs influence results. ACE likewise reports offline adaptation, online adaptation, and online updates after pre-adaptation separately. (Wang et al. 2025; Suzgun et al. 2026; Q. Zhang et al. 2026)
The remaining scientific question is not whether experience can accumulate, but which representations and update units maintain procedures useful on other jobs without carrying forward old mistakes or irrelevant specific values. Answering it requires distinguishing unseen evaluation with fixed memory from sequential adaptation, and measuring retrieval, curation, and retry costs as well as accuracy. Evaluation compares evaluation designs.
Figure Credits
Figure 1: (Zhao et al. 2024). Original preprint, Figure 1 (public arXiv version). CC BY 4.0. Reduced in size.
Table 2: (Zhao et al. 2024). Original preprint, Table 2 (public arXiv version). CC BY 4.0. Table body cropped and rasterized.
Figure 2: (Wang et al. 2025). Original published version, Figure 2. CC BY-SA 4.0. Cropped and rasterized. This adaptation is provided under the same license.
Figure 2: (Fan et al. 2025). Original published version, Figure 2. CC BY 4.0. Cropped and rasterized.
Figure 2: (Q. Zhang et al. 2025). Original published version, Figure 2. CC BY 4.0. Cropped and rasterized.
Figure 2: (Suzgun et al. 2026). Original published version, Figure 2. CC BY 4.0. Cropped and rasterized.
Figure 4: (Q. Zhang et al. 2026). Original published version, Figure 4. CC BY 4.0. Cropped and rasterized.
Figure 2: (Shinn et al. 2023). Original: source PDF, Figure 2. Figure region rasterized without changing its contents.
Figure 2: (Xu et al. 2025). Original: source PDF, Figure 2. Figure region rasterized without changing its contents. CC BY 4.0.
Figure 2: (G. Zhang et al. 2025). Original: source PDF, Figure 2. Figure region rasterized without changing its contents.
Figure 2: (Yang et al. 2026). Original: source PDF, Figure 2. Figure region rasterized without changing its contents.