The Harness Is the Point

For two years the story of LLM agents was a story about models — bigger context windows, better reasoning traces, more capable base weights. ICML 2026 tells a different story. A striking share of the strongest agent papers are not about the model at all. They are about the harness: the runtime, the tool layer, the orchestrator, the evaluation scaffold, and the control boundary that sits around the model and decides what actually happens. Call it harness engineering.

The thesis running through this literature is blunt: as base models converge in raw capability, the harness becomes the locus of both value and failure. AdaptOrch states it directly — "as models converge, orchestration becomes the differentiator". Recognize Your Orchestrator shows it empirically — across deep-research, coding, GUI, and RAG agents, the orchestrator, not the executors, is the primary failure source. And Base Models Know How to Reason, Thinking Models Learn When argues the capability is already latent; the control layer decides when it fires.

Below is the landscape — forty-five papers across five facets.^† Hover a node to see the paper; click to open it. Filter by facet with the legend.

Facet one

Scaffolding & runtimes: the harness as engineered infrastructure

The most literal reading of "harness engineering" is: treat the scaffold as a first-class artifact you compile, schedule, and debug — not a prompt you hand-tune.

SAGE is the purest expression of the idea, modeling agent pipelines as a compiled dataflow with explicit resource contracts, bounded-queue backpressure, and p99 latency guarantees — the harness as systems infrastructure, not a prompt. OpenSage goes further and synthesizes the scaffold itself: an agent-development kit where topology, toolset, and hierarchical memory are generated rather than hand-authored. MAS-Orchestra does the same for multi-agent systems, generating a whole coordination structure "at once" and shipping MASBENCH to measure when that structure actually helps — finding, importantly, that multi-agent benefit is task-dependent, not universal, at >10× efficiency over baselines. The direct predecessor is AFlow (ICLR 2025 Oral), which framed scaffold generation as MCTS over code-represented workflows.

Two papers treat the harness as debuggable infrastructure. HarnessFix (preprint) attributes failures in execution traces to specific harness components and applies targeted repairs for 6.3–18.4% gains. The Interplay of Harness Design and Post-Training (preprint) shows the harness is a variable you must co-design with training — neglect it and robustness collapses under environment shift. Rounding out the facet: DeepHA builds a hierarchical planner/executor whose Chain-of-Action cuts context ~75%; AutoTool learns dynamic tool selection via a Plackett–Luce ranking that generalizes to unseen tools; Gecko gives the tool-calling inner loop a stateful simulation to refine against, lifting GPT-4o on BFCLv3 from 76.9%→84.6%; and Agentic Proposing composes modular reasoning skills into verifiable trajectories, reaching 91.6% on AIME25.

Facet two

Tool use & function calling: the load-bearing layer

If the harness has a load-bearing wall, it is the tool interface. The most actionable finding of the cycle comes from AgentNoiseBench: across 25 models, corrupted tool feedback degrades agents more than ambiguous user requests. That single result reframes where a harness should spend its defensive budget — validate and sanitize what comes back from tools, not just what the user sends in.

Evaluation of tool use is shifting from outcomes to trajectories. TRACE ("Beyond the Final Answer") grades the call sequence — redundancy, hallucinated tool outputs — without ground-truth trajectories, using small open-source judges. ComplexMCP stress-tests the modern regime directly: 300+ tools across 7 stateful MCP sandboxes, where leading models reach ~60% versus ~90% for humans, and names the three bottlenecks a harness must engineer against — retrieval at scale, calibration, and recovery. AppWorld-UL adds the human-in-the-loop axis (top models: 38.2%). On the systems side, RealtimeTool attacks tool-call latency with parallel decoding for 3–6× (up to 9.6×) speedups; D-CORE incentivizes task decomposition for complex multi-step tool use; and SciAgentGym provides a multi-step scientific tool-use benchmark.

Facet three

Evaluation harnesses: you can't engineer what you can't measure

A remarkable amount of this year's work is harness-engineering about harness engineering — building the reproducible scaffolds that let us evaluate agents at all.

VeRO is the most on-the-nose: an evaluation harness for the "agent optimizes agent" task, with versioned snapshots, budget-controlled evaluation, and traces that separate deterministic code from stochastic LLM completions — exactly the reproducibility machinery the field has lacked. SWE-Bench Pro modernizes code-agent evaluation with a contamination-resistant public/private split over long-horizon, multi-file tasks, and daVinci-Dev (Oral) operationalizes executable-repo environments with real tool outputs as training substrate, hitting SOTA-open on SWE-Bench Verified at under half the tokens. SWE-Compass pushes toward unified coding-agent evaluation.

The most elegant idea in the facet is AutoWebWorld: generate intrinsically verifiable web environments from finite state machines, so ground truth is programmatic — 11,663+ verified trajectories at ~$0.04 each, no LLM judge required. CUARewardBench benchmarks the judge itself on computer-using-agent trajectories, and EnterpriseOps-Gym offers a containerized, stateful enterprise sandbox where the best agent manages only 34.1% — and notably fails to decline infeasible tasks. DSGym extends the "gym" pattern to data-science agents. Two safety-flavored eval harnesses stand out: Jailbreak Foundry (Oral) auto-translates attack papers into runnable modules with a unified judge, reproducing 30 attacks with half the code, and SandboxEscapeBench (Oral) measures whether an agent can break out of the very container meant to isolate it — a nested-sandbox CTF on Inspect-AI.

Facet four

Orchestration: the control layer is where systems break

The evaluation tells us where to look; the orchestration papers tell us what they find. Recognize Your Orchestrator introduces a Mean-Field Entropy Dynamics view and a measurable "scheduling entropy," showing the orchestrator is the primary failure locus and identifying a "Reasoning Trap" where reasoning-heavy models degrade under context squeeze. AOrchestra answers with automatic, on-the-fly sub-agent creation via a clean 4-tuple abstraction (instruction, context, tools, model), +16.28% over the strongest baseline on GAIA/SWE-Bench/Terminal-Bench. AgentConductor treats the communication topology as a first-class learnable object. Two position-shaping results anchor the facet — AdaptOrch and Base Models Know How to Reason — and InfoPO adds a concrete harness gate: a turn-level information-gain reward teaching agents when to ask versus act.

Facet five

Safety & control harnesses: gating on effect, not intent

The fastest-moving facet is the control boundary — and it has converged on a sharp idea: gate on what an action does, not on whether the input looked harmful. The motivating result is When Benign Inputs Lead to Severe Harms, which shows innocuous-looking inputs can trigger severe harmful actions in computer-use agents. If harm is a property of the effect, a content filter on the input is the wrong instrument; you need an effect boundary at the harness.

Gate on the action's effect, not the input's apparent harmfulness.

Several papers build that boundary. SafeHarbor turns an abstract "adaptive boundary" into a concrete real-time defense pipeline over agent actions, backed by hierarchical memory. The Oversight Game gives it a formal spine: a two-player Markov game where the agent chooses act-vs-defer and the human chooses trust-vs-oversee, with a proof that — under a Markov Potential Game — an agent gaining autonomy cannot decrease the human's value. ANCHOR audits CLI agents, the highest-stakes effect surface, for real-world harmful outcomes, and TRACER provides the early-warning signal such a boundary needs, flagging unreliable trajectories within the first ~20% of a conversation.

The prior art is essential and worth citing precisely: Progent (2025) is the canonical programmable-privilege DSL enforced at execution without touching the model; AgentSpec (ICSE 2026) is a customizable runtime-enforcement layer intercepting actions at execution time; and SafeMCP filters risky tools from the action space before the agent even chooses — a pre-selection boundary rather than pre-execution. The open design question these three frame is where the boundary sits and whether it is certified or heuristic. Rounding out: CausalArmor gates indirect prompt injection at the tool boundary via causal attribution; Constitutional Black-Box Monitoring detects scheming from behavior alone; AIR frames safety operationally as detect→contain→respond; and Architecture Matters for Multi-Agent Security shows the attack surface is itself a function of the orchestration topology.

The synthesis

What it adds up to

Five threads run through these forty-five papers:

• The harness, not the model, is the locus of value and failure. (Recognize Your Orchestrator, AdaptOrch, Base Models Know How to Reason, HarnessFix, Interplay of Harness Design.)
• Bad tool output — not bad user input — is the dominant failure driver, which relocates the defensive budget to the tool-return path. (AgentNoiseBench, CausalArmor.)
• Mock-and-refine and simulation layers for tool calls are now first-class technique. (Gecko, ComplexMCP.)
• Evaluation is moving from final-answer to trajectory, and toward cheap deterministic verifiers. (TRACE, AutoWebWorld, SWE-Bench Pro, CUARewardBench.)
• Safety is converging on effect-boundary / action-authorization at the harness layer. (When Benign Inputs, SafeHarbor, The Oversight Game, Progent, AgentSpec, SafeMCP.)

The engineering lesson is that the interesting design surface has moved. The model is increasingly a component; the harness — the runtime, the tool contract, the orchestrator, the eval scaffold, the effect boundary — is the system. ICML 2026 is the year the field started building it like one.

References

The 45 papers

1. 1AutoTool: Dynamic Tool Selection and Integration for Agentic Reasoning. Jiaru Zou, Ling Yang, … Mengdi Wang. ICML 2026. link
2. 2Gecko: A Simulation Environment with Stateful Feedback for Refining Agent Tool Calls. Zeyu Zhang, Guohao Li, Zhenchang Xing, … Liang Zheng. ICML 2026. link
3. 3MAS-Orchestra: Understanding and Improving Multi-Agent Reasoning Through Holistic Orchestration and Controlled Benchmarks. Zixuan Ke, Yifei Ming, Austin Xu, … Shafiq Joty (Salesforce AI). ICML 2026. link
4. 4OpenSage: Self-programming Agent Generation Engine. Hongwei Li, Zhun Wang, … Dawn Song. ICML 2026. link
5. 5SAGE: A Dataflow-Native Framework for Modular, Controllable, and Transparent LLM-Augmented Reasoning. Jun Liu, Peilin Liu, Ruicheng Zhang, … Hai Jin. ICML 2026. link
6. 6DeepHA: Scaling Action Chains Elicits Deep Hierarchical Agents. Zihao Wang, Muyao Li, … Yitao Liang. ICML 2026. link
7. 7Agentic Proposing: Enhancing LLM Reasoning via Compositional Skill Synthesis. Zhengbo Jiao, Shaobo Wang, … Linfeng Zhang. ICML 2026. link
8. 8From Failed Trajectories to Reliable LLM Agents: Diagnosing and Repairing Harness Flaws (HarnessFix). Mengzhuo Chen, Junjie Wang, … Qing Wang. arXiv 2026 (preprint). link
9. 9The Interplay of Harness Design and Post-Training in LLM Agents. Kyungmin Kim, Youngbin Choi, … Sangdon Park. arXiv 2026 (preprint). link
10. 10AFlow: Automating Agentic Workflow Generation. ICLR 2025 (Oral). link
11. 11AgentNoiseBench: Benchmarking Robustness of Tool-Using LLM Agents Under Noisy Conditions. Ruipeng Wang, Yuxin Chen, … Tat-Seng Chua. ICML 2026. link
12. 12Beyond the Final Answer: Evaluating the Reasoning Trajectories of Tool-Augmented Agents (TRACE). Wonjoong Kim, Sangwu Park, … Chanyoung Park. ICML 2026. link
13. 13ComplexMCP: Evaluation of LLM Agents in Dynamic, Interdependent, and Large-Scale Tool Sandbox. Yuanyang Li, Xue Yang, … Hongyang Chen. ICML 2026. link
14. 14RealtimeTool: Parallel Decoding for Real-Time LLM Function Calling. Xiaoxin Shi, Jiaxin Wan, … Zengfeng Huang. ICML 2026. link
15. 15AppWorld-UL: Benchmarking Diverse Agent-User Interactions for Tool-Use. Junzhi Chen, Harsh Trivedi, … Ashish Sabharwal. ICML 2026. link
16. 16D-CORE: Incentivizing Task Decomposition in Large Reasoning Models for Complex Tool Use. ICML 2026. link
17. 17SciAgentGym: Benchmarking Multi-Step Scientific Tool-Use in LLM Agents. ICML 2026. link
18. 18VeRO: An Evaluation Harness for Agents to Optimize Agents. Varun Ursekar, Apaar Shanker, Veronica Chatrath, Yuan Xue, Samuel Denton. ICML 2026. link
19. 19AutoWebWorld: Synthesizing Infinite Verifiable Web Environments via Finite State Machines. Yifan Wu, Yiran Peng, … Yuyu Luo. ICML 2026. link
20. 20CUARewardBench: Benchmarking Reward Models on Computer-using Agent Trajectories. Haojia Lin, Xiaoyu Tan, … Xing Sun (Tencent). ICML 2026. link
21. 21EnterpriseOps-Gym: Environments and Evaluations for Stateful Agentic Planning and Tool Use in Enterprise Settings. Shiva Malay, Perampalli Shravan Nayak, … Sai Rajeswar Mudumba. ICML 2026. link
22. 22Base Models Know How to Reason, Thinking Models Learn When. ICML 2026. link
23. 23Recognize Your Orchestrator: An Entropy Dynamics Perspective for LLM Multi-Agent Systems. NJU-NLP. ICML 2026. link
24. 24AdaptOrch: Task-Adaptive Multi-Agent Orchestration in the Era of LLM Performance Convergence. arXiv 2026 (preprint). link
25. 25DSGym: A Standardized and Holistic Framework for Advancing Data Science Agents. ICML 2026. link
26. 26SWE-Compass: Towards Unified Evaluation of Agentic Coding Abilities for LLMs. ICML 2026. link
27. 27daVinci-Dev: Agent-native Mid-training for Software Engineering. Ji Zeng, Dayuan Fu, … Pengfei Liu. ICML 2026 (Oral). link
28. 28Jailbreak Foundry: From Papers to Runnable Attacks for Reproducible Benchmarking. Zhicheng Fang, Jingjie Zheng, Chenxu Fu, Wei Xu. ICML 2026 (Oral). link
29. 29Quantifying Frontier LLM Capabilities for Container Sandbox Escape (SandboxEscapeBench). Rahul Marchand, Art Cathain, … Harry Coppock. ICML 2026 (Oral). link
30. 30AOrchestra: Automating Sub-Agent Creation for Agentic Orchestration. FoundationAgents. ICML 2026. link
31. 31InfoPO: Information-Driven Policy Optimization for User-Centric Agents. ICML 2026. link
32. 32AgentConductor: Topology Evolution for Multi-Agent Competition-Level Code Generation. ICML 2026. link
33. 33SafeHarbor: Hierarchical Memory-Augmented Guardrail for LLM Agent Safety. ICML 2026. link
34. 34The Oversight Game: Learning to Cooperatively Balance an AI Agent's Safety and Autonomy. ICML 2026. link
35. 35When Benign Inputs Lead to Severe Harms: Unintended Harmful Behaviors in Computer-Use Agents. ICML 2026. link
36. 36ANCHOR: Automated Alignment Auditing for CLI Agents Detecting Real-World Harmful Outcomes. ICML 2026. link
37. 37CausalArmor: Efficient Indirect Prompt-Injection Guardrails via Causal Attribution. ICML 2026. link
38. 38Constitutional Black-Box Monitoring for Scheming in LLM Agents. ICML 2026. link
39. 39TRACER: Trajectory Risk Aggregation for Critical Episodes in Agentic Reasoning. Ranganath Krishnan et al. (Capital One / UIC). ICML 2026. link
40. 40AIR: Improving Agent Safety through Incident Response. ICML 2026. link
41. 41Architecture Matters for Multi-Agent Security. ICML 2026. link
42. 42AgentSpec: Customizable Runtime Enforcement for Safe and Reliable LLM Agents. ICSE 2026. link
43. 43Progent: Programmable Privilege Control for LLM Agents. arXiv 2025. link
44. 44SafeMCP: Proactive Power Regulation via Environment-Grounded Look-Ahead Reasoning. arXiv 2026. link
45. 45SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks? Xiang Deng, Jeff Da, … Brad Kenstler (Scale AI). ICML 2026. link

Cite

BibTeX

BibTeX for all 45 papers (and this survey): harness-engineering-icml2026.bib. To cite the survey itself:

@misc{rallabandi2026harness,
  author       = {Rallabandi, Sai Krishna},
  title        = {The Harness Is the Point: Harness Engineering at {ICML} 2026},
  year         = {2026},
  howpublished = {\url{https://saikrishnarallabandi.github.io/judith/posts/harness-engineering-icml2026.html}},
  note         = {Field notes, On Rallabandi Radar}
}

† On sourcing. Most papers here are confirmed ICML 2026 acceptances (linked to icml.cc); a handful are 2026 arXiv preprints or adjacent-venue prior art (ICLR 2025, ICSE 2026), labeled at each reference. Forty-five papers, forty-two with verified abstracts. Every citation link was checked against the paper's own title, and twelve URLs that resolved to unrelated papers — an off-by-100-months arXiv-ID error — were corrected. Shortened author lists should be verified on the linked page before formal citation.