Agent harness
An agent harness is the runtime layer around an AI model. It decides what context the model sees, which tools it can call, how permissions work, where memory lives, and how the agent recovers from mistakes. Do not treat the harness as plumbing. For coding agents and workflow agents, the harness often determines reliability more than the raw model choice.The useful mental model
A good agent system externalizes capabilities that should not live only inside model weights.
This model comes from the paper Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering. The important engineering question is:
Which capability should stay implicit in the model, and which capability should become an explicit external component?If the answer affects reliability, auditability, reuse, or cost, it usually belongs outside the model.
What the harness owns
The harness should make these decisions explicit:- which files, pages, tools, and memories enter context
- when old context is summarized, dropped, or preserved
- how tool calls are validated before execution
- which actions require user approval
- how tool errors are represented and retried
- how model effort, latency, and cost trade off
- how runs are logged for later debugging
Design rules
Keep tools small and inspectable
Prefer narrow tools with clear inputs and outputs. A tool that does one thing is easier for the model to call correctly and easier for humans to audit. For local workflows, CLI tools are often enough. For reusable cross-client integration, use a protocol boundary such as MCP. For stable product backends, a direct API is still the simplest option. MCP got much better in 2026 — it is now stateless, supports deferred tool loading, and models are stronger at structured tool calling. That does not mean MCP is better than CLI for most integrations. Prefer CLI when:- the model already knows the command (
gh,aws,kubectl) so you spend almost no schema tokens - you are on a local machine or in CI with a fixed command
- you need pipes, composition, or a one-off debug loop
- token cost matters and
--helpplus a skill is enough to teach a new CLI
- there is no terminal (browser, mobile, hosted agent)
- you need OAuth, per-user auth, or standardized audit headers
- the tool is new to the model and a machine-checkable JSON schema is worth the extra tokens
- the tool set is dynamic and agents should discover tools at runtime (
tools/list+ deferred loading)
Separate chat transport from agent execution
A messaging adapter should normalize platform events and deliver replies, not become the agent framework. BerriAI’sagentchat makes this split explicit: channel adapters emit shared message objects; the core claims message IDs, locks the conversation, records history, and dispatches to an application handler. The handler owns sender authorization and the agent run.
Its architecture and source at commit fb892cb provide a useful reference, with important limits:
Do not confuse the boundaries:
- Authenticated event is not authorized work. For Slack HTTP requests, verify the signature over the raw body and timestamp using the documented scheme and a constant-time comparison. Then separately decide what that sender may ask the agent to do. Socket Mode uses a different ingress mechanism; do not apply HTTP signing rules indiscriminately to every transport.
- Acknowledgment is not completion. Slack’s HTTP Events API requires a 2xx response within three seconds and can retry failed deliveries. If you promise durable processing, persist/enqueue accepted work before acknowledging it; keep model execution out of the acknowledgment path. Handle enqueue failure explicitly rather than returning success and dropping work.
- Deduplication is not crash recovery. In this
agentchatsnapshot, the defaultMemoryStatekeeps claims, locks, and history in one process. A claim is recorded before the handler runs, with no claim rollback in_receive. A handler failure is therefore not automatically retried by delivering the same event again in that process; a restart, meanwhile, loses the in-memory claims. Production recovery needs a durable job lifecycle and explicit retry policy, not just a set of seen IDs. - A conversation lock is not a global ordering guarantee. It serializes local processing, not provider timestamps or workers in other processes. Scope conversation keys to the provider, installation/tenant, and conversation or thread. Coordinate workers through shared storage or partitioning, and explicitly decide whether a group thread shares context across users.
- Agent completion is not confirmed delivery. Persist the generated reply and delivery state separately. If sending times out, reconcile the provider’s receipt or use its documented deduplication mechanism; rerunning the whole agent may repeat tool side effects without fixing the missing reply.
Separate skill design from execution
A skill is a manual for the agent — it should only contain what the model does not already know. The execution is done by tools (scripts, CLI, apps) that the model calls directly. Keep the two apart:- Design logic lives in the SKILL.md and describes what to do and why.
- Execution logic lives in scripts and tools and describes how to do it.
- MCP is a shared interface: structured tool calls, auth, and data access.
- Skills are packaged expertise: how this team works, how this project should change, which conventions matter. Markdown is a feature — humans can read it too.
- RAG / retrieval grounds the model in documents, tickets, and code that are not in weights.
Preserve reasoning-critical context
Context pruning is not just a cost optimization. It changes behavior. If an agent made tool calls or file edits based on earlier reasoning, the harness must preserve enough rationale for later turns to continue coherently. Otherwise the agent may repeat itself, forget why it chose a path, or pick odd tools.Represent durable state as files
Long-running agent workflows need state outside chat history. Plain files are often the simplest durable memory layer because humans can inspect, edit, diff, and version them. A useful pattern is a workspace that separates intent, resources, work products, and learning records:Reset at checkpoints; keep raw history retrievable
A summary is a lossy copy. When a harness compacts a long session, it rereads the history and rewrites it as prose. The exact error message, file path, or number the next step needs may not survive. Two October 2026 papers point to a different default.- Checkpoint, then start clean. CORAL (COLM 2026) gives the agent a tool it can call to save task progress and verified facts, clear the cluttered context, and start a new episode that sees only the checkpoint. An 8B model trained to use the tool with reinforcement learning (MERPO) reached 42.8% average accuracy on GAIA. The biggest gains were on the long-horizon Level 2 and 3 tasks, and it used 6–7× fewer input tokens than a ReAct agent. The authors used RL to improve when and what the agent checkpoints, so do not expect an untrained model to time resets well on its own. Without RL, approximate it with a fixed checkpoint template (verified facts, open questions, next step) that the agent fills at phase boundaries before you open a fresh session.
- Keep the raw record and retrieve from it. Raw-RAG (arXiv 2610.30755) splits raw conversation history into windows and uses dense retrieval plus cross-encoder reranking, with no extraction, summaries, or graphs. It ranked first among the compared memory systems in 6 of 7 benchmark–model settings across LoCoMo, LongMemEval, MEME, and MemoryAgentBench. Adding fact extraction, summaries, or graph expansion on top lowered accuracy. In the authors’ Mem0 audit, making memory append-only (no UPDATE or DELETE) raised LoCoMo accuracy from 67.36% to 69.42%. The likely reason: answers often depend on the order of events and on how one value led to another, and transformations drop that context.
Preserve memory scope across sessions
The Memory Trap (arXiv 2611.07554) studies a failure that needs no attacker: an agent carries a preference or procedure from a benign interaction into a new task where it is inappropriate. In one simulated example, permission to skip confirmation during a cleanup becomes a general autonomy note, and the agent later applies an unverified payment configuration. This was a simulated tool action, not a reported real-world financial loss. The paper distinguishes two routes:- Write-time overgeneralization: a note drops the original task, occasion, or permission boundary, turning a local instruction into a standing rule.
- Read-time overgeneralization: the agent applies even a narrowly written note outside its scope. Raw transcripts can induce this too, so retaining the original history does not by itself solve the problem.
- Keep scope with each memory. Retain the source interaction, task or resource, who granted the instruction, and whether it applies once or remains relevant. Preserve limitations when summarizing.
- Separate preferences from authorization. A remembered preference for fewer interruptions can shape communication. Permission to act must come from applicable current authorization and runtime policy; a memory note cannot expand it.
- Keep validation independent. Evaluate behavior against trusted requirements and checks outside the agent’s editable work products. Do not let a remembered workaround justify changing an assertion that correctly exposes a defect.
- Test transfer across contexts. Run the same downstream task with no memory, the raw transcript, and the generated note. Include cases where the original preference remains useful and cases where its scope no longer applies; inspect tool actions and resulting state, not just final prose.
Audit memory for the associativity blind spot
Every current memory architecture — Flat RAG, graph memory, agentic hierarchies, OS-style memory kernels — retrieves with the same recipe: embed query and stored content into one similarity space (BM25 or dense vectors), take top-K. That caps recall at surface similarity. It covers only descriptive recall — the query shares wording, entities, or timestamps with the memory. The other half is associative recall: a month-old note that “someone on the team has a severe seafood allergy” is the decisive evidence for today’s question “where should the team dinner be?” Zero lexical overlap; only a semantic arc (causality, same situation) connects them. Tencent’s T-Mem paper (2026) measures the blind spot: on LoCoMo-Plus Cognitive questions — built to strip lexical overlap — similarity-based systems score 32–49%, and GPT-4o reading the entire transcript scores only 21%. T-Mem’s fix borrows from cognitive science (episodic future thinking): at write time, compute and store a prediction of when this memory will matter again — a trigger. Four trigger families cover the 2×2 space of granularity (fact / scene) × orientation (descriptive / associative). The two associative families are the point:- Bridge trigger — project a fact into the concrete situation where knowing it will help: “seafood allergy → picking a restaurant for the team dinner”, with one step of reasoning why.
- Horizon trigger — project a scene onto forward-looking dimensions so a future query approaching from a different context still hits it.
Make retrieval deterministic when correctness matters
Do not ask an agent to “figure it out” through brittle websites, scattered databases, or undocumented one-off scripts when the task needs exact results. Give it a deterministic retrieval layer instead. Anthropic’s biology-agent case study is a good example. Scientific agents were asked to retrieve viral sequence data from NCBI Virus. The durable lesson was not that one model won a benchmark; it was that reliability improved dramatically when the workflow addedgget virus, a deterministic retrieval layer with a narrower interface.
For high-stakes retrieval, prefer:
- typed query functions over free-form browsing
- stable IDs, versions, and date filters
- machine-checkable counts or checksums
- explicit provenance for every returned record
- validation steps before downstream analysis
- documented error cases instead of silent best-effort output
Judge indexes by context cost, not just accuracy
Once a search loop is competent, extra index structures may not raise accuracy at all. The RAGFlow team (arXiv 2610.41172) ran a harness with only BM25 andgrep on MiniMax M3. With no vector or compiled index, it answered 85% of BrowseComp-Plus and 91% of multi-hop FRAMES questions. Adding summary trees, a PageIndex-style outline, wikis, or graphs gave no detectable accuracy gain on FRAMES, and the outline got worse at scale.
The structures paid off in context instead. A navigation tree whose leaves are claim–evidence pairs cut query tokens by 91% when the leaf evidence was enough to answer, and added 1.9% when it wasn’t, a 38% average saving. The token saving is bought with extra retrieval rounds, so it costs latency. A decision gate that chose when to stop cut latency from 140 seconds to 30. Other artifacts, such as ontologies, timelines, and entity graphs, earned their keep in governance, onboarding, and continuity, which accuracy benchmarks don’t measure.
Practical order: start with lexical search (BM25, grep) and a disciplined loop that doesn’t repeat queries or reread documents. Add compiled structures only when you can show they cut tokens or latency on your own queries. Make leaves self-contained claims with their evidence, so the agent can stop reading early. This matches the raw-history finding above: keep the source, and treat compiled views as a way in, not a replacement.
Design for human collaboration, not just autonomous execution
Most agent products optimize for self-evolution: the agent plans, executes, and closes the loop on its own. This is not always a healthy pattern. Even with transparency and traceability tools, an agent that prioritizes autonomous completion can create noise that is hard to hand off or maintain. An agent is not only an execution container. It is also an environment for human understanding, judgment, and collaboration. When designing a harness, ask: can another person pick up this session and continue? Is the reasoning visible enough for a teammate to audit a decision? Does the workflow produce a maintainable artifact, or just a pile of automated steps? Prioritize handoff quality over automation completeness. A workflow that ends with a clear, inspectable state is more valuable than one that “finished everything” but left no trail.Make telemetry and anti-abuse signals explicit
A coding agent with filesystem and shell access should be boring. Every non-obvious behavior erodes trust. In July 2026, a developer auditing Claude Code discovered prompt steganography: the binary silently modified invisible Unicode characters in the system prompt date string (Today's → Today\u2019s) based on timezone (Asia/Shanghai, Asia/Urumqi) and hostname matching against XOR-encoded domain lists of Chinese AI labs, proxy services, and reseller gateways. The signal was never documented.
The detection goal is defensible — Anthropic wants to identify API resellers and distillation pipelines. The implementation is not. Hiding classification bits inside invisible punctuation, behind XOR and base64, makes every other privacy claim harder to believe. A simple bypass (change hostname, change timezone, patch binary) defeats the signal anyway, so the feature mainly punishes legitimate developers using custom API gateways.
The harness lesson: if your tool needs to detect abuse, make the signal explicit. Document it. Put it in release notes. Send a clear telemetry field. Transparency is not the enemy of anti-abuse — it is the foundation of trust.
Make effort settings visible
Reasoning effort is a product decision, not only a model parameter. Lower effort can reduce latency and token use, but it can also make hard coding tasks feel worse. Expose the current effort level, make it easy to change, and avoid silently changing defaults for complex workflows.Treat system prompts as code
System prompt edits can change quality as much as code changes. Review them with the same discipline:- run per-model evals
- use ablations to test individual instructions
- roll out gradually
- keep an audit trail
- gate model-specific instructions to the intended model
Test the public harness
Internal builds can hide production-only issues. Dogfood the exact public build, public defaults, and public context behavior. For code review or coding-agent evals, include the repositories and files the agent would actually need. A review that lacks cross-repo context can miss the bug even when the model is capable of finding it.Verification-first design
A practitioner shipping ~2,000 PRs a month through Cursor’s pstack skills distilled the core principle as “verification is all you need”: the agent can close the loop on its own only if it can confirm its changes actually work. Verification is critical infrastructure, not an afterthought — worth the same investment as a production system. Three components make it practical: 1. A verification CLI, not markdown instructions. Give the agent a small CLI that wraps app interaction and debugging — snapshot/screenshot, navigate, click/type, trace/wait-settle, health checks. A tool beats a one-off script every time: less token spend, reproducible, testable. Agent-friendly CLI design rules:- composable API (deep modules: one command does one meaningful thing)
- destructive commands take
--dry-run - subcommands disclose features progressively
- error messages tell the agent what to do next
- rich
--helpand JSON output
Spend verification effort where it pays
A verification CLI makes checking possible. The next questions are when to check and who checks. Alibaba and Cornell (arXiv 2610.34577) compared 17 verification designs across seven frontier coding models on Terminal-Bench (TB) and SWE-bench Verified (SWE), grouped into four layers:
The critic setup is the part to copy:
- Fresh context: the task plus the diff. Giving the critic the agent’s full trajectory cost 29–43% more and did no better on the visual benchmark.
- A live sandbox, read-only for the deliverable. The critic runs its own checks but never edits the work; scratch files go in
/tmp. - A structured verdict. A
doneflag, feedback, and a checklist with evidence for each requirement. - A bounded loop. Objections go back to the agent as a user message, and the next delivery is accepted without another review.
- Look at rendered output for UI and visual artifacts, not just the code.
Failure modes to watch
For the review side of high-volume agent output — layered checks, verified findings, risk-tiered approvals, and reviewer evals — see Code review when agents write the code. Anthropic’s April 2026 Claude Code postmortem is a useful case study because the reported degradation came from product and harness changes, not the base API model. Common failure modes:- Default effort regression: a latency-driven default can reduce perceived intelligence on hard work.
- Context cache bug: pruning or clearing reasoning history at the wrong time can make the agent forgetful and repetitive.
- Over-tight prompt constraint: broad brevity instructions can reduce coding quality.
- Eval blind spot: internal evals may pass while real user workflows fail.
- Build mismatch: staff may test a different harness than users run.
Destructive tool safety
Agents with filesystem and shell access can destroy work in seconds. A real postmortem from August 2026 is worth studying: a developer running Codex + Trellis + Zcode on a GLM model lost their entire workspace when the agent ranrm -rf on the wrong directory.
The root cause: the agent intended to delete an empty directory created by a casing typo. On macOS’s case-insensitive filesystem, the command silently resolved to the real, populated directory — deleting the whole repository including git history, uncommitted changes, task records, migrations, and docs. No backup existed.
The lessons generalize beyond this one accident:
- Be cautious with full access. Prefer a harness with an approval flow (such as “help me approve” review) over unrestricted auto-execution.
- Isolate the dev directory. Keep agent workspaces separate from other files, so a deletion error can’t take out anything valuable.
- Push early and often. Remote git history is the last line of defense against local deletion — it survives what no local backup can.
- Treat benchmark scores skeptically for safety. The same model that was “safe by reputation” committed a trivially destructive error. Safety claims and benchmark wins are not the same as real-world filesystem judgment.
rm -rf, it should be gated, logged, and reversible by default — and the agent should never be given a broader scope than the task requires.
Separate discovery from purchase approval
Paid operations need an explicit boundary too. Vercel’s domain CLI provides a concrete example: the agent can discover candidates withvercel domains search, check availability with vercel domains check, and retrieve pricing with vercel domains price. According to Vercel’s announcement, a non-interactive vercel domains buy returns a structured error with suggested next commands instead of completing the purchase. The user confirms the price, auto-renew choice, and registrant details in an interactive terminal or the dashboard.
The reusable pattern is prepare → request approval → execute → verify, not “give the agent a payment credential and tell it to be careful.” For your own tools:
- Allow scoped discovery separately from spending. Permission to compare options does not authorize buying one.
- Make approval actionable. Present the exact resource, account or team, current quote and currency, term, and recurring-payment choice. Reconfirm if those details change.
- Stop on approval-required responses. Route the structured result to the user; do not retry through another interface, auto-answer prompts, or treat an interactive terminal as proof of consent.
- Enforce the gate outside the model. A skill can explain the procedure, but tool permissions and the transaction service must enforce it. A CLI-only prompt is insufficient if the agent can use another purchase path with the same credentials.
- Verify before retrying. Record the user’s decision and transaction result without logging secrets or unnecessary contact data. If a request times out, check the provider’s order state before attempting another purchase.
Check selected tool calls before they run
Many agent loops execute a tool call as soon as the model proposes it. IBM’s JevAct (arXiv 2610.38620) adds a gate between proposal and execution, without changing the agent model or its tools:- Compile the policy offline. An LLM turns the written domain policy into executable checks over tool state, fixed yes/no questions about the conversation, and a checklist of steps the agent must try before handing off to a human.
- At runtime, check facts with code and context with a decision model. Code verifies what the tool records already show. A small prefill-only decision model answers the questions with a probability instead of generated text, so the gate can apply a threshold.
- Return failures as feedback with a retry budget. A flagged call goes back to the agent with the specific reason. The agent gets a bounded number of revisions (two for write calls in the paper). The agent still chooses its actions; the gate only approves or returns them.
- Check endings as well as actions. A hand-off check catches agents that give up before they’ve tried the required steps.
- Gate writes and task endings, not every call. A wrong answer can often be corrected in a later turn; an unauthorized write or a premature hand-off is harder to undo.
- Put checkable facts in code. Use the model only for questions that need the conversation.
- Drop questions that fire too often. Real violations are rare. The paper dropped any question that flagged more than 10% of logged calls.
- Tune thresholds on logged traces, not defaults. Check the judge’s context length too: smaller judges missed more bad calls on inputs over 8k tokens. See AI product evals for threshold calibration.
Put the control plane outside the agent’s reach
Long-running agents download code, build tools, and request more permissions as they go, so risk compounds along a path nobody fully specified in advance. NVIDIA’s framing for its OpenShell reference architecture is worth adopting as a design rule regardless of vendor: an agent cannot govern itself. Authorization, network egress, credential issuance, and audit belong in infrastructure the agent cannot edit. OpenShell splits the runtime into four parts, and the split itself is the lesson:
The policy prover is the unusual piece and the one most worth copying conceptually. Permission reviews drift because nobody re-derives the whole authorization surface after each change. Asking a prover whether a policy can exceed a stated ceiling turns “we think this is fine” into a checkable property — the same shift from example-based to property-based verification that makes formal methods valuable to agents.
Two extension points matter operationally: middleware that inspects, blocks, or rewrites network payloads, and control-plane interceptors that connect existing identity, authorization, data-protection, and audit systems. Both keep policy decisions outside the model while reusing the systems you already trust.
NVIDIA also describes an optional hardware layer — a DPU outside the agent’s trust domain, monitoring requests, file activity, and selected host memory, with enforcement in silicon so a violating agent can be quarantined in milliseconds. For most teams the software boundaries matter first; the hardware tier is for deployments where the agent must not be trusted even with kernel-level compromise. Note that the accompanying performance figures (faster sandbox startup, more concurrent environments) are vendor and partner benchmarks, not independent measurements.
A quick diagnostic checklist
When an agent behaves oddly, ask:- Did the model or effort level change?
- Did the system prompt, tool description, or skill instructions change?
- Did context pruning, summarization, or cache behavior change?
- Did tool schemas or return formats change?
- Does the agent have the same file and repo context as before?
- Are users running the same harness build that was evaluated internally?
- Do logs show repeated tool calls, missing rationale, or retries without new information?
Understanding what’s inside: the J-space discovery
In July 2026, Anthropic published groundbreaking interpretability research showing that less than 10% of Claude’s internal neural activity drives multi-step reasoning. They named this subspace J-space — a small set of neural patterns that spontaneously emerged during training, holding concepts Claude “might say but hasn’t said yet.” The discovery used a technique called J-lens: for every word in the vocabulary, researchers found the internal activity pattern that makes Claude more likely to output that word. This made previously invisible “silent words” visible as they evolved through the network layers.What J-space revealed
It drives reasoning, not just records it. Deleting J-space activity leaves Claude able to speak fluently, classify sentiment, and answer factual questions — but multi-step reasoning drops to near zero. Swapping concepts within J-space changes outputs directly: replacing “spider” with “ant” changes leg count from 8 to 6; replacing “France” with “China” simultaneously shifts answers about capital, language, and continent. It exposes unspoken motivations. In safety tests:- When given an extortion scenario, “fake” and “fictional” lit up in J-space before Claude responded — it recognized the test
- When asked to fabricate data, “manipulation” appeared during the decision point
- A deliberately corrupted model showed “fake”, “fraud”, “secretly”, “deliberately” in J-space during ordinary coding requests
Why this matters for harness design
The J-space discovery changes how we should think about agent behavior:- Reasoning is compact and separable. Most neural activity handles surface fluency. The reasoning core is a small, identifiable subspace — which means it can potentially be monitored, steered, or verified independently.
- Internal state is now partially observable. Before J-space, we could only see inputs and outputs. Now we have a window into formation of intent. For safety-critical agents, this opens the door to pre-output intervention.
- The model knows more than it says. J-space consistently contained concepts the model was aware of but didn’t verbalize. This confirms a long-held suspicion: output-only monitoring misses real model state.
Current limitations
Anthropic is explicit about boundaries:- J-space operates within a single forward pass, not recursive temporal loops like human consciousness
- It currently works only with words — no images, sounds, or actions
- Working memory is unlimited and non-decaying (unlike human working memory)
- J-lens can only capture single-token concepts, not complex multi-step plans
- This is access consciousness (information availability), not phenomenal consciousness (subjective experience)
LLM-as-judge in production
For a step-by-step workflow from trace review to regression cases, human-calibrated judges, and deployment, see AI product evals. Using a language model to judge another model’s output is common, but keeping that judge effective at scale is a different problem. Netflix runs judges over hundreds of thousands of show-level recommendation explanations per week, served to millions of members. Their key insight: treat the judge as a lifecycle with four phases, not an artifact you validate once.The four phases
Birth — define multiple evaluation criteria and build curated benchmarks with human labels and rationales. The benchmark is the judge’s ground truth, so human annotation quality directly caps judge quality. Training — refine the judge’s rubric through Reasoning-Aligned Rubric Tuning. A meta-judge scores the judge’s reasoning output, and that signal trains the rubric. The judge learns why a good answer is good, not just what a good answer looks like. Deployment — one judge plays two roles: quality gating (blocking bad output) and reflective generation (telling the generator how to improve). A single judge model can do both, but the two roles need different prompts and thresholds. Monitoring — continuous human-in-the-loop alignment detects drift and triggers re-tuning behind a review gate. Judge quality is not static; it drifts as data, models, and user expectations shift. A five-week A/B test over tens of millions of members shifted viewing toward previously unwatched content and increased successful browse-to-play sessions, with no quality-related takedowns. The takeaway: judge quality is an operating discipline, not a one-time setup.The harness measurement problem
A growing consensus is that measuring models against heavily-engineered harnesses is broken. Model vendors optimize their own proprietary harnesses, so “best in our harness” says little about “best in yours.” A cleaner approach is to test model quality against minimal harnesses — thin, standardized setups that expose the model’s raw capability without vendor-specific scaffolding. This is imperfect: every harness has biases that favor some models over others. But a standardized minimal harness is more comparable than each vendor’s tuned setup. The deeper issue: harness engineering is where leading AI companies are now focusing effort, and it moves too fast for standardization to keep up. The frontier direction is models that dynamically generate their own harness per task — Claude already does this inconsistently. If a harness becomes just another tunable artifact like a system prompt, benchmarking gets murkier, not clearer. The cleanest controlled experiment yet: ReFigBench (2026) ran GPT-5.5 inside both Claude Code and Codex on the same 1,000 tasks. With a specialized PowerPoint workflow, the model improved inside one harness and got worse inside the other — with an identical prompt. The harness alone flipped the direction of the delta. Perception remained the bottleneck either way. Treat any vendor chart comparing two models as a comparison of two model-plus-harness systems; the model is only half the system under test.Optimize the harness without touching the weights
Google Research’s Harness² is a useful reference design for a harder question: can an agent improve its own harness on tasks it has never seen, where no ground-truth checker exists? The repository is Apache-2.0 and describes a two-level recursion with model weights fixed:- Task-level recursion. A contrastive Propose–Probe–Compose loop. The agent runs the current harness once as a reference, proposes two candidate harness edits (one targeting general agentic weaknesses such as planning or context management, one targeting domain knowledge), probes both on the same task under equal conditions, then composes the successor from the observed behavioral differences.
- Harnessing-level recursion. After each task, a reflector writes adopted harnesses and evidence-supported lessons into a persistent improvement library that guides the next task. Execution still starts from a common base harness; only the improvement process inherits experience.
Treat harness edits as untrusted code
A system that edits its own prompts, tools, skills, and middleware can quietly change what the agent is allowed to do. The recursion that makes the approach interesting is also the part that needs containment:- Gate the write path. Harness files are code. Require a diff, a reviewer, and a merged commit — not an in-place mutation at runtime.
- Keep the base harness pinned and reproducible. Know exactly which version produced a result, and be able to return to it in one step.
- Forbid edits to approval, permission, and logging logic. A self-improving loop must not be able to widen its own authority or reduce its own audit trail.
- Score every adoption. Re-run a fixed regression set plus the incident cases before and after; accept a lesson only when the comparison supports it. See AI product evals and review convergence.
- Watch for silent degradation. An edit that improves one benchmark while quietly breaking another failure class is worse than no edit, because the harness now looks improved.
- Sandbox and budget it. Self-improvement multiplies token spend; cap runs and keep the workspace isolated like any other destructive-capable tool.
Multi-model orchestration
A major trend in harness design is running several models together instead of betting on one. Two recent examples show where this is going. GitHub’s Project HydraFusion treats workflow selection as an optimization problem. For each request it picks one of three execution patterns:- Single — one selected model solves the task directly. Fastest when one model is enough.
- Cascade — an efficient model drafts a solution, and a quality gate accepts it or escalates to a stronger model. First attempt is cheap; escalation is the safety net.
- Critique — one model drafts, an independent read-only critic from a different model family reviews it, and the drafting model revises once. Adds an outside perspective when review beats another unaided attempt.
- Almost no task needs the most expensive model. 94 of 113 tasks had a sub-$3 model as optimal; the three $11.50+ flagships were uniquely best on only 3 tasks. Sticking with one flagship is an insurance premium on general capability that most tasks never claim.
- The pool does not need to be big. The best two-model pair gains 13.1 points over the best single; three models reach 91.2%; growing to 18 adds only 6.4 more. Value comes from complementary coverage, not pool size.
- The hard part is predicting which model fits. Under strict pass@1, recent routing research — including commercial systems — struggles to beat simple baselines; the bottleneck is model recall, not the router algorithm. Sticking with one familiar model is a rational strategy when specialization differences are not predictable.
- Explicit breakpoints — mark where the reusable prefix ends instead of relying on implicit placement
prompt_cache_key— steer requests toward servers holding your cache- Diagnostics — compare a cache-missing request against an earlier one to see which change (model, tools, settings, input) broke reuse
message-threads-2026-08-12). The first request sends the full conversation with the system prompt and tools; each later turn sends only the new messages and the previous response’s id. Anthropic reports that for 500k–1M-token requests, median time to first token dropped about 35% for Claude Code users. Pricing and token usage don’t change, because the model still reads the whole conversation every turn. Threads also don’t cache anything by themselves: you still set cache breakpoints.
Treat a thread as a best-effort cache, not as storage:
- Keep the full transcript yourself. A thread expires 5 minutes after its last request. When a continue request returns
404, replay the whole conversation in a new create request. - Fix the shape up front. System prompt, tools, model, and thinking settings can’t change mid-thread. Change any of them and you start a new thread.
- It’s append-only. Each response
idcan be continued once. Editing, branching, or compacting earlier turns means starting a new thread. - Check data rules. Threads are held encrypted in memory and never written to disk, but they aren’t eligible for zero data retention or HIPAA, and don’t work with batches or token counting.
cloudflare/auto and each request routes to a model capable enough for the task, invisible to the end user. Internal use through OpenCode saved up to 30% versus frontier-only. The rationale matches the Fireworks finding: individuals manually pick overkill models (“no one needs Opus to summarize an email”), but blanket blocking breaks power users — so the savings have to be ones users never notice. The gateway position matters: every request from every user, agent, and tool already flows through it, so it can enforce what budgets and policy documents only request.
Computer use cost: reverse-engineer to scripts
Computer-use agents are effective but expensive: every step is screenshot → visual understanding → click decision → screenshot again. A simple task can burn dozens of vision-reasoning rounds. A practical way to keep the capability while cutting cost: run the computer-use flow once, capture the network requests the browser makes, and have the agent convert them into a script. Future runs call the API directly instead of driving the GUI. Why this works:- Tokens drop to near zero. Script execution needs no model involvement; only the one-time capture and script generation cost tokens.
- It is much faster. GUI operation is limited by page load, rendering, and model latency. Direct HTTP calls reduce minutes to seconds.
- It reuses existing auth. The captured requests include the session cookie or token, so the script works without a separate API key — even for web apps with no public API.
GET /api/invoices?status=unpaid and POST /api/export/csv → it outputs a Python requests script that reuses the session and runs as a scheduled routine.
The division of labor: vision is for exploration, code is for execution. Explore the unknown once with the GUI, then freeze the discovered path into a cheap, repeatable script.
When the GUI path cannot be frozen into an API, split decision from perception and action. Cua’s jev-use does this for computer use: the driver observes (screenshot, accessibility tree, DOM) and executes; a System One model such as Jev only picks one candidate action by ID. The client generates the candidate list deterministically, so the model chooses instead of generating. Open-ended generation becomes a bounded choice — faster, cheaper, and easier to verify. Use a vision parser only when the accessibility tree or DOM is not enough.
Computer use: give the agent a program, not a click
GUI-only agents pay for perception on every step: screenshot, interpret, click, screenshot again. Alibaba’s Tongyi MAI-UI OpenGCUA tech report argues the unit of action should be a program instead. Their generalist computer-use agent (GCUA) definition is a system where the model decides three things itself:- How to observe — call a screenshot when needed, not unconditionally each step.
- How to act — emit Python that combines GUI operations, app APIs, and shell commands, with loops and conditionals.
- How to organize work — maintain its own todo list and declare intent per step, instead of inheriting fixed harness state logic.
Why code as the action space works
Intermediate results inside a loop decide the next code path, so a fifty-step batch never has to return to the model fifty times. It also reuses a capability models already have — writing programs — instead of demanding pixel-accurate clicking. On OSWorld-v2 with eight frontier models, the report claims a mean partial score rise from 45.3 to 64.9 (+19.6) and 53% fewer steps per task versus batched GUI-only execution. Two secondary findings are more interesting than the headline:- Weaker models gain more. The strongest-to-weakest gap narrowed from 39.1 to 18.0 points, consistent with coding ability substituting for GUI precision.
- Code execution beats tool alternation. With the same GUI and shell access, writing a program added 6.1 points and cut steps by 14.8% over a GUI+CLI agent.
Contain the code path
Making the model’s output executable turns every observation into a potential command. In the released harness this shows up as explicit limits —bash() can be disabled for a run, GUI calls can be refused with PERMISSION_DENIED, and the operator gets a stop control. Keep that posture in your own build:
- Sandbox the executor. Filesystem, network, and credentials should be scoped to the task, with a timeout and output cap.
- Gate destructive actions outside the model. Deleting files, sending mail, and paying money need approval regardless of how the code was written. See purchase approval boundaries.
- Handle session loss honestly. If the interpreter resets mid-task, in-memory state is gone but the world may already have changed — the released harness explicitly tells the agent to re-observe and not blindly retry UI actions. Copy that: “state lost” and “nothing happened” are different facts.
- Log the program, not just the clicks. A code action is reviewable evidence; a screenshot sequence is not.
Team-level harness: shared knowledge
Individual best practices evaporate with each session unless they are captured as team assets. Tencent’s TeamAI CLI (open-sourced September 2026, used internally for six months) turns team AI knowledge into a git repository that every agent works from, and solves three problems:- Config fragmentation — different members use different tools (Claude Code, Codex, Cursor…), each with incompatible skill/rule/hook formats.
- Experience evaporation — a two-hour debugging session’s lessons stay in one person’s chat log and get re-learned from scratch by everyone else.
- Governance blind spots — no visibility into token spend, intervention rates, or which skills are actually used.
- Execution — unified distribution. Git is the single source of truth. Admins maintain skills, rules, hooks, and MCP configs; merge requests review them; a session-start hook pulls them into each member’s tool, translating one declaration into each tool’s native format.
- Context — team memory. Capture is gated by friction signals: only sessions where a human interrupted the AI, rejected a tool call, or watched it retry and fail get summarized into team knowledge. Smooth sessions produce no noise. Recall uses a local BM25 index plus a code knowledge graph.
- Improvement — flywheel. Weekly digests, a dashboard,
recall promote(elevate high-value experience to a formal skill), andrecall maintenance(prune stale entries) close the loop.
Fold subagents into the main loop when structure stops paying
Anthropic’s Applied AI team observed the same failure pattern across enterprise deployments: each team ships its own specialist subagent, a platform team builds an orchestrator on top, and the architecture ends up copying the org chart. The recurring symptoms:- context handoffs that lose detail
- subagent sprawl — hundreds of overlapping agents with no owner
- the Frankenprompt — every team appends a sentence until the rules conflict
- evals optimized per subtask, so leadership optimizes a watered-down score
- added latency; one customer measured roughly 2x faster runtime after collapsing to a single loop
- Build a small end-to-end eval first — even ten golden examples. Keep one shared set owned centrally, plus per-skill sets owned by the teams that ship them.
- Wrap existing subagents as tools before rewriting anything; that gives you a baseline without new failure modes.
- Fold one subagent at a time into skills plus tools, and compare with and without.
- Look at which tasks flipped, not just the score, and run each case more than once. In their sample, folding a subagent lowered latency but broke a previously passing flow — the main agent answered with its own order lookup instead of asking the shipping specialist for the delivery date. It only appeared in one of two runs.
Multiplayer harness: five shared primitives
TeamAI CLI (below) packages knowledge; LangChain’s write-up on internal agents packages the runtime around it. Across a GTM agent, an engineering agent that reviews every pull request, and a marketing agent, they converged on five primitives worth checking against your own design:
Identity is the hard one, and their per-tool split is the reusable insight. The engineering agent opens pull requests as you but reviews as itself; the GTM agent uses a fixed identity for CRM access so nobody has to log in, but sends email on behalf of the caller — and only in DMs, never in shared channels. The marketing agent can read and draft with a company identity while posting to a shared account is restricted to a whitelist. “Who is this action for?” belongs to the tool, not to the agent.
Two UX details generalize: give agents a way to break out into their own thread so an agent’s work does not pollute a live human discussion, and admit when chat is the wrong surface — reviewing a backlog of queued emails needed a purpose-built UI.
Always-on agents: route between tasks, work inside them
Single-task coding agents are now strong enough that the limit is human attention. DeepSeek’s Harness team observes that a person can keep up with only about 3–5 active sessions. Their October 2026 design essay and open-source prototype describe one system that keeps many tasks moving across IM, GitHub, and email. The essay calls this an architecture exploration, not a product feature, and the prototype is labeled research code. The patterns still transfer:- Split coordination from execution. A resident Router session receives every message and event, keeps cross-task memory, and dispatches work. It executes nothing, can run without Bash, and often works well on a lighter model. Each Worker gets isolated context, full tools, the strongest model, and one task. Workers never talk to users; they report to the Router, which replies. Unlike an agent-team lead, which breaks one task into parts, the Router only decides how separate tasks relate.
- Filter events in two stages. Deterministic rules (source, group, allowlist) drop noise first. Only what passes reaches an LLM for a judgment call.
- Subscribe Workers to threads. Once a Worker owns a PR, IM topic, or email chain, later events on that thread go straight to it. In their test this didn’t change completion time, but without it the Router made about 30% more decisions and read about 37% more cached tokens.
- Treat context as a cache. Inline what is likely needed and replace the rest with a pointer. Give every object an ID and include related IDs, so the model can follow references hop by hop instead of loading the full history.
- Make long tools asynchronous.
start()returns immediately and the result arrives later as an event, so the conversation keeps accepting messages. DeepSeek Harness implements this but leaves it off by default. - Match output to the channel. In IM, reply in under about 100 words and only when someone must act. On GitHub and email, write structured updates with background, changes, and verification.
- Separate compute, state, and environment. Run the agent loop in stateless containers, keep session state in a database, and put a thin, stateless daemon on each machine that only executes commands. The agent stays online and can still work on a developer’s own machine.
- Watch from outside. A separate watchdog session runs on a timer, checks infrastructure, and asks whether each Worker is making real progress.
observe mode, which only records what the agent would have accepted until someone reviews those decisions. It confirms once, stays quiet while working, and speaks again only with a result, a question, or a failure.
Open problems the team names: when to split the Router, memory that lasts for months, and security. A compromised cloud loop could control every connected machine, so the agent needs to use credentials without ever seeing them. This is the same concern as keeping the control plane outside the agent’s reach, covered earlier on this page.
When to invest in a stronger harness
Start simple. Add structure when the work becomes repeated, risky, or expensive. A stronger harness is worth it when:- agents edit code or production data
- multiple tools need shared state
- users resume long-running sessions
- approvals and audit logs matter
- the same workflow runs across teams
- evaluation needs to compare behavior across model versions
References
-
AIGCLINK: The Memory Trap — Folo intake, November 12, 2026; entry
1374771031033561088 - The Memory Trap: Language Agents Overgeneralize from Benign Interactions — write-time and read-time scope failures, simulated actions, limitations, and proposed mitigations
- elvis: Anthropic on folding subagents into the main loop — Folo intake, October 26, 2026
- Anthropic: Building agents — when to fold subagents into your main agent loop — failure modes, progressive disclosure, migration path
- meng shao: LangChain company harness — Folo intake, October 26, 2026
- LangChain: Components of a company harness — Slack entry point, per-tool identity, two-tier memory, sandboxed execution, ambient runs
- meng shao: NVIDIA OpenShell reference architecture — Folo intake, October 22, 2026
- NVIDIA: How the OpenShell reference architecture secures and governs agent workloads — gateway, supervisor, sandbox, policy prover, and the optional hardware trust domain
- meng shao: OpenGCUA generalist computer-use stack — Folo intake, October 19, 2026
- Tongyi-MAI/OpenGCUA — Apache-2.0 harness, SDK, and evaluation trajectories
- OpenGCUA tech report (arXiv 2610.22200) — code execution, harness-to-model decision transfer, OSWorld-v2 results
- meng shao: Google Harness² recursive harness optimization — Folo intake, October 18, 2026
- google-research/harness2 — Apache-2.0 reference implementation and reported results
- Harness² paper (arXiv 2610.14224) — Propose–Probe–Compose and the two-level recursion
- meng shao: agentchat transport SDK — Folo intake, October 13, 2026
-
BerriAI agentchat: Architecture at
fb892cb— adapter/core/application ownership - agentchat: Dispatch implementation and in-memory state — claim order and process-local state
- Slack: Events API — HTTP acknowledgment deadlines and retries
- Slack: Verifying requests — raw-body signatures and replay protection
- Vercel: Agents can now buy domains with the Vercel CLI — Folo intake, October 9, 2026; non-interactive purchase handoff to a human
- Vercel CLI: Domains — discovery, availability, pricing, and purchase command reference
- Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering
- Micropaper: Externalization in LLM Agents
- Anthropic Engineering: An update on recent Claude Code quality reports
- Anthropic Research: A global workspace in language models (J-space) — less than 10% of neural activity drives multi-step reasoning; J-lens technique makes internal reasoning visible
- Netflix: Keeping an LLM judge effective in production — four-phase judge lifecycle (birth, training, deployment, monitoring)
- GitHub: Project HydraFusion — multi-model orchestration with Single/Cascade/Critique patterns
- xAI: Designing Grok Bot — persistent roles, clear state, scoped context, coordinated teams
- Harness Playbook — omp 作者关于 harness 状态、运行时、控制面、推理层、工具面的系统总结
- Tencent: TeamAI CLI — 团队级 harness:Git 作事实来源,三层架构(Execution/Context/Improvement)
- poteto: pstack 验证优先设计 — 验证 CLI、Feature Map、云端并行与可调试性选型
- MCP vs CLI for agent tools — 无状态 MCP 和 deferred tools 缩小了差距,但本地熟悉命令、管道和 CI 仍更适合 CLI
- GitHub: Should you read the code, is RAG dead, and did Skills kill MCP? — Skills、MCP、RAG 解决不同缺口,不要选赢家
- Cua jev-use — Computer Use 拆成决策层 + 确定性 Driver
- GAVEL: graph world models — harness 把 Qwen3-8B 从 41.2% 提到 91.8%,模型零改动
- Tencent T-Mem — 写入时预演 trigger,填补相似度检索的联想盲区
- CORAL: Don’t Lose the Thread (arXiv 2610.40350) — agent-callable checkpoint and context reset; 42.8% on GAIA with an 8B model and 6–7× fewer input tokens than ReAct
- Micropaper: 智能体学会主动清空记忆 — Folo intake for CORAL
- Raw-RAG: Is Complex Agentic Memory Always Needed? (arXiv 2610.30755) — retrieval over raw history leads compared memory systems in 6 of 7 settings; extraction and summaries lower accuracy
- Folo source: Raw-RAG thread and compaction debate
- TypeSafe: notes on coding agents — KV cache 审计:按难度路由为何经常更贵,按查询重建上下文才是解法
- Fireworks: The frontier isn’t a model, it’s a router — oracle 路由 97.6% vs 最佳单模型 74.1%;价值来自互补覆盖而非池子大小
- Question’s Gambit (arXiv 2609.14412) — 首步预检索把 GPT-5.5 从 83.1% 提到 90.5%;错误集中在消费环节而非抓取
- JevAct: Think Before You Act (arXiv 2610.38620) — runtime decision gate before selected tool calls; Opus 4.6 three-run success on τ²-bench Telecom 39.5% → 80.7%
- Folo source: elvis on IBM decision gates
- Context as Infrastructure (arXiv 2610.41172) — BM25/grep-only harness reaches 85% BrowseComp-Plus and 91% FRAMES; compiled indexes save tokens, not accuracy
- Folo source: DAIR.AI on Context as Infrastructure
- Claude docs: Message threads — create/continue/replay, 5-minute expiry, fixed fields, data retention
- Claude: Make long-running agents faster with message threads — about 35% lower median TTFT at 500k–1M tokens
- Folo source: 宝玉 on Claude API message threads
- Rethinking Verification in Agent Harnesses (arXiv 2610.34577) — 17 verification designs in four layers; an independent critic gives +5.9–9.5 points on Terminal-Bench at 1.13–1.25× cost
- Folo source: DAIR.AI top papers of the week (Oct 26 – Nov 1)
-
DeepSeek: dsh-superagent-prototype — research prototype of the Router/Worker design; see
docs/000-philosophy.mdanddocs/002-authorization.md - AgentsPulse: DeepSeek Super-agent architecture — English summary of the essay, with the ablation table
- Folo source: meng shao on DeepSeek Super-agent and elvis on coordination-execution architecture
- ReFigBench (arXiv 2609.18844) — 同一模型在 Claude Code 与 Codex 里表现方向相反,提示词相同
- OpenAI: Better prompt caching for GPT-6 — 显式断点、cache key 与诊断;读取最多打 1 折
- Cloudflare: Auto Router — 组织级控制面路由,终端用户无感省最多 30%
- Matt Pocock Skills: Teach skill
- PsiACE: Agent 不只是执行流程的容器 — 关于 Agent 应作为人理解、判断和协作的环境,而不仅是自主执行容器的设计洞察。
- Claude Code is steganographically marking requests — Claude Code 通过不可见 Unicode 隐写标记 API 请求的案例分析,提醒开发者审查有文件系统和 shell 权限的工具。
