ALTK-Evolve Calibrates Memory for DeepSeek and GPT Models

A new evaluation of the ALTK-Evolve framework across eight language models reveals that equipping AI agents with memory requires careful calibration based on model capability. Using the AppWorld benchmark, which features 585 multi-step tasks across nine simulated applications, researchers tested how different memory delivery methods affect Task Goal Completion (TGC) and Scenario Goal Completion (SGC). Rather than updating model weights, ALTK-Evolve distills reusable behavioral guidelines from past successful and unsuccessful runs and injects them back into the context window during inference.

The study shows that strong models with headroom benefit most from receiving the full guideline set. For example, DeepSeek-V3.2, a 671B mixture-of-experts (MoE) model, achieved a 9.5 percentage point increase in TGC and a 16.1 percentage point jump in SGC when given its full self-mined guideline set. Conversely, smaller or weaker models perform better with a curated retrieval strategy that combines a high-confidence core with task-relevant guidelines. The 117B MoE model gpt-oss-120b gained 16.1 percentage points in TGC using this selective approach, while the full guideline set yielded lower gains and increased token costs by approximately 50 percent. Meanwhile, saturated models like the 745B MoE GLM-5 showed no measurable improvement.

Even highly capable models near their performance ceiling saw reliability gains. GPT-5.5 and Opus improved their SGC scores by 7.2 and 7.1 percentage points, respectively, demonstrating that memory helps resolve rare edge cases. Crucially, curated retrieval kept inference costs low; gpt-oss-120b achieved its performance gains with only a 5 percent increase in tokens. For larger models using full guidelines, prompt caching can mitigate the cost of injecting guidelines at every step of the reasoning loop, as DeepSeek-V3.2 maintained an average of 18 to 19 ReAct steps with or without memory.

For practitioners building agentic workflows, these findings shift the focus from simply accumulating past trajectories to tailoring context-window delivery. Developers should deploy curated retrieval for smaller models to maximize accuracy and minimize token overhead, while reserving full guideline injection—supported by cache-aware prompt engineering—for frontier models.