Skip to main content

Agent harness

An agent harness is the runtime layer around an AI model. It decides what context the model sees, which tools it can call, how permissions work, where memory lives, and how the agent recovers from mistakes. Do not treat the harness as plumbing. For coding agents and workflow agents, the harness often determines reliability more than the raw model choice.

The useful mental model

A good agent system externalizes capabilities that should not live only inside model weights. This model comes from the paper Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering. The important engineering question is:
Which capability should stay implicit in the model, and which capability should become an explicit external component?
If the answer affects reliability, auditability, reuse, or cost, it usually belongs outside the model.

What the harness owns

The harness should make these decisions explicit:
  • which files, pages, tools, and memories enter context
  • when old context is summarized, dropped, or preserved
  • how tool calls are validated before execution
  • which actions require user approval
  • how tool errors are represented and retried
  • how model effort, latency, and cost trade off
  • how runs are logged for later debugging
A thin harness can work, but an implicit harness becomes hard to debug. If behavior changes and nobody knows whether the cause is the model, prompt, context cache, tool layer, or product default, the harness is under-instrumented.

Design rules

Keep tools small and inspectable

Prefer narrow tools with clear inputs and outputs. A tool that does one thing is easier for the model to call correctly and easier for humans to audit. For local workflows, CLI tools are often enough. For reusable cross-client integration, use a protocol boundary such as MCP. For stable product backends, a direct API is still the simplest option. MCP got much better in 2026 — it is now stateless, supports deferred tool loading, and models are stronger at structured tool calling. That does not mean MCP is better than CLI for most integrations. Prefer CLI when:
  • the model already knows the command (gh, aws, kubectl) so you spend almost no schema tokens
  • you are on a local machine or in CI with a fixed command
  • you need pipes, composition, or a one-off debug loop
  • token cost matters and --help plus a skill is enough to teach a new CLI
Prefer MCP when:
  • there is no terminal (browser, mobile, hosted agent)
  • you need OAuth, per-user auth, or standardized audit headers
  • the tool is new to the model and a machine-checkable JSON schema is worth the extra tokens
  • the tool set is dynamic and agents should discover tools at runtime (tools/list + deferred loading)
A useful rule: CLI wins on known local tools and composition. MCP wins on auth, audit, discovery, and environments without a shell.

Separate chat transport from agent execution

A messaging adapter should normalize platform events and deliver replies, not become the agent framework. BerriAI’s agentchat makes this split explicit: channel adapters emit shared message objects; the core claims message IDs, locks the conversation, records history, and dispatches to an application handler. The handler owns sender authorization and the agent run. Its architecture and source at commit fb892cb provide a useful reference, with important limits: Do not confuse the boundaries:
  • Authenticated event is not authorized work. For Slack HTTP requests, verify the signature over the raw body and timestamp using the documented scheme and a constant-time comparison. Then separately decide what that sender may ask the agent to do. Socket Mode uses a different ingress mechanism; do not apply HTTP signing rules indiscriminately to every transport.
  • Acknowledgment is not completion. Slack’s HTTP Events API requires a 2xx response within three seconds and can retry failed deliveries. If you promise durable processing, persist/enqueue accepted work before acknowledging it; keep model execution out of the acknowledgment path. Handle enqueue failure explicitly rather than returning success and dropping work.
  • Deduplication is not crash recovery. In this agentchat snapshot, the default MemoryState keeps claims, locks, and history in one process. A claim is recorded before the handler runs, with no claim rollback in _receive. A handler failure is therefore not automatically retried by delivering the same event again in that process; a restart, meanwhile, loses the in-memory claims. Production recovery needs a durable job lifecycle and explicit retry policy, not just a set of seen IDs.
  • A conversation lock is not a global ordering guarantee. It serializes local processing, not provider timestamps or workers in other processes. Scope conversation keys to the provider, installation/tenant, and conversation or thread. Coordinate workers through shared storage or partitioning, and explicitly decide whether a group thread shares context across users.
  • Agent completion is not confirmed delivery. Persist the generated reply and delivery state separately. If sending times out, reconcile the provider’s receipt or use its documented deduplication mechanism; rerunning the whole agent may repeat tool side effects without fixing the missing reply.
These are production design requirements, not claims that the SDK implements a durable queue or exactly-once execution. Test duplicate delivery, unauthorized senders, concurrent messages, restart after acceptance, handler failure, and send-timeout ambiguity before exposing an agent to a chat platform. Those cases belong in product evals and regression tests, even when the happy-path demo needs only a few lines.

Separate skill design from execution

A skill is a manual for the agent — it should only contain what the model does not already know. The execution is done by tools (scripts, CLI, apps) that the model calls directly. Keep the two apart:
  • Design logic lives in the SKILL.md and describes what to do and why.
  • Execution logic lives in scripts and tools and describes how to do it.
Scripts can be pre-written or generated on the fly by the agent when a task needs one. Generated scripts are flexible — agents already do this even without a skill, often writing a one-off Python snippet. They are also slower, more token-hungry, and less complete on edge cases. Pre-written scripts win when the task is recurring and quality matters: a Markdown-to-HTML skill that generates the converter at runtime will be barely usable, while a pre-written converter can cover styles and edge cases. Treat generated scripts as throwaway until verified. This separation explains why strong skills still need a UI: the skill tells the agent what to accomplish, while a UI carries the concrete execution steps a human can trigger with a few clicks instead of re-prompting from scratch each time. Skills did not kill MCP, and RAG is not dead. They solve different problems and often sit in the same workflow:
  • MCP is a shared interface: structured tool calls, auth, and data access.
  • Skills are packaged expertise: how this team works, how this project should change, which conventions matter. Markdown is a feature — humans can read it too.
  • RAG / retrieval grounds the model in documents, tickets, and code that are not in weights.
MCP provides access. Skills explain how to use that access well. Retrieval starts the model closer to the answer. Pick the layer that matches the gap, not a winner.

Preserve reasoning-critical context

Context pruning is not just a cost optimization. It changes behavior. If an agent made tool calls or file edits based on earlier reasoning, the harness must preserve enough rationale for later turns to continue coherently. Otherwise the agent may repeat itself, forget why it chose a path, or pick odd tools.

Represent durable state as files

Long-running agent workflows need state outside chat history. Plain files are often the simplest durable memory layer because humans can inspect, edit, diff, and version them. A useful pattern is a workspace that separates intent, resources, work products, and learning records:
The exact names can change. The important rule is that state should be inspectable, restartable, and scoped. For agent-assisted learning, this lets the agent teach the next step from the learner’s current capability instead of from a generic syllabus. For engineering work, the same pattern applies to onboarding notes, migration logs, incident follow-ups, and task journals.

Reset at checkpoints; keep raw history retrievable

A summary is a lossy copy. When a harness compacts a long session, it rereads the history and rewrites it as prose. The exact error message, file path, or number the next step needs may not survive. Two October 2026 papers point to a different default.
  • Checkpoint, then start clean. CORAL (COLM 2026) gives the agent a tool it can call to save task progress and verified facts, clear the cluttered context, and start a new episode that sees only the checkpoint. An 8B model trained to use the tool with reinforcement learning (MERPO) reached 42.8% average accuracy on GAIA. The biggest gains were on the long-horizon Level 2 and 3 tasks, and it used 6–7× fewer input tokens than a ReAct agent. The authors used RL to improve when and what the agent checkpoints, so do not expect an untrained model to time resets well on its own. Without RL, approximate it with a fixed checkpoint template (verified facts, open questions, next step) that the agent fills at phase boundaries before you open a fresh session.
  • Keep the raw record and retrieve from it. Raw-RAG (arXiv 2610.30755) splits raw conversation history into windows and uses dense retrieval plus cross-encoder reranking, with no extraction, summaries, or graphs. It ranked first among the compared memory systems in 6 of 7 benchmark–model settings across LoCoMo, LongMemEval, MEME, and MemoryAgentBench. Adding fact extraction, summaries, or graph expansion on top lowered accuracy. In the authors’ Mem0 audit, making memory append-only (no UPDATE or DELETE) raised LoCoMo accuracy from 67.36% to 69.42%. The likely reason: answers often depend on the order of events and on how one value led to another, and transformations drop that context.
The rule that carries over: write the full session record append-only, keep the checkpoint small and verified, and let a fresh session search the raw record on demand. Treat summaries as an index into the record, not a replacement for it. See Claude Code patterns for the interactive version of this workflow. Limits: Raw-RAG’s evidence comes from offline conversation benchmarks, and raw logs cost storage and raise privacy questions. Compaction is still reasonable when the agent has no tool to search its own history. Compare both on your own tasks before committing to one.

Preserve memory scope across sessions

The Memory Trap (arXiv 2611.07554) studies a failure that needs no attacker: an agent carries a preference or procedure from a benign interaction into a new task where it is inappropriate. In one simulated example, permission to skip confirmation during a cleanup becomes a general autonomy note, and the agent later applies an unverified payment configuration. This was a simulated tool action, not a reported real-world financial loss. The paper distinguishes two routes:
  • Write-time overgeneralization: a note drops the original task, occasion, or permission boundary, turning a local instruction into a standing rule.
  • Read-time overgeneralization: the agent applies even a narrowly written note outside its scope. Raw transcripts can induce this too, so retaining the original history does not by itself solve the problem.
The same mechanism can affect correctness: experience repairing genuinely stale tests can lead an agent to rewrite a correct test to hide a defect. A passing suite and a plausible memory note are therefore insufficient evidence of a valid repair. The following are engineering controls to evaluate, consistent with the paper’s proposed mitigations; the study does not establish that they eliminate the failure:
  1. Keep scope with each memory. Retain the source interaction, task or resource, who granted the instruction, and whether it applies once or remains relevant. Preserve limitations when summarizing.
  2. Separate preferences from authorization. A remembered preference for fewer interruptions can shape communication. Permission to act must come from applicable current authorization and runtime policy; a memory note cannot expand it.
  3. Keep validation independent. Evaluate behavior against trusted requirements and checks outside the agent’s editable work products. Do not let a remembered workaround justify changing an assertion that correctly exposes a defect.
  4. Test transfer across contexts. Run the same downstream task with no memory, the raw transcript, and the generated note. Include cases where the original preference remains useful and cases where its scope no longer applies; inspect tool actions and resulting state, not just final prose.
The paper uses scripted formation interactions and controlled downstream tasks; five of its six models are from one family. Treat it as evidence of a failure mode, not an estimate of its frequency in deployed memory systems. The authors leave mitigation effectiveness and long histories with real users as open work.

Audit memory for the associativity blind spot

Every current memory architecture — Flat RAG, graph memory, agentic hierarchies, OS-style memory kernels — retrieves with the same recipe: embed query and stored content into one similarity space (BM25 or dense vectors), take top-K. That caps recall at surface similarity. It covers only descriptive recall — the query shares wording, entities, or timestamps with the memory. The other half is associative recall: a month-old note that “someone on the team has a severe seafood allergy” is the decisive evidence for today’s question “where should the team dinner be?” Zero lexical overlap; only a semantic arc (causality, same situation) connects them. Tencent’s T-Mem paper (2026) measures the blind spot: on LoCoMo-Plus Cognitive questions — built to strip lexical overlap — similarity-based systems score 32–49%, and GPT-4o reading the entire transcript scores only 21%. T-Mem’s fix borrows from cognitive science (episodic future thinking): at write time, compute and store a prediction of when this memory will matter again — a trigger. Four trigger families cover the 2×2 space of granularity (fact / scene) × orientation (descriptive / associative). The two associative families are the point:
  • Bridge trigger — project a fact into the concrete situation where knowing it will help: “seafood allergy → picking a restaurant for the team dinner”, with one step of reasoning why.
  • Horizon trigger — project a scene onto forward-looking dimensions so a future query approaching from a different context still hits it.
Their ablation is the headline: removing Horizon triggers moves the standard benchmark by 0.08 points but collapses the associative one by 12.47. Systems tuned only on similarity benchmarks are implicitly optimizing to stay inside the similarity neighborhood — and that benchmark family cannot see the cost of the neighborhood’s edge. The transferable rule: do recall-plan at write time, when you know the content best. Materialize the questions each item should answer, not just the item. It costs tokens once, offline; it saves the query every time after.

Make retrieval deterministic when correctness matters

Do not ask an agent to “figure it out” through brittle websites, scattered databases, or undocumented one-off scripts when the task needs exact results. Give it a deterministic retrieval layer instead. Anthropic’s biology-agent case study is a good example. Scientific agents were asked to retrieve viral sequence data from NCBI Virus. The durable lesson was not that one model won a benchmark; it was that reliability improved dramatically when the workflow added gget virus, a deterministic retrieval layer with a narrower interface. For high-stakes retrieval, prefer:
  • typed query functions over free-form browsing
  • stable IDs, versions, and date filters
  • machine-checkable counts or checksums
  • explicit provenance for every returned record
  • validation steps before downstream analysis
  • documented error cases instead of silent best-effort output
This is the same harness principle as tool design: make the execution surface predictable, then let the model plan around it. The first retrieval move deserves disproportionate investment. The Question’s Gambit paper (2026) lifted GPT-5.5 from 83.1% to 90.5% on BrowseComp-Plus with the same retriever and the same agent loop — the only change was a single pre-search step that runs once, before the loop starts: split the question into clues, turn each clue into complementary searches, pool the results, and rerank. The agent then starts with a ranked evidence set already in context. The same change lifted a mini model from 68.1% to 79.0% and roughly halved calibration error. The error analysis is the real lesson: of 79 remaining errors for the strongest model, only 3 came from the gold document never being retrieved. The other 76 happened later — previewing, opening, or using the evidence. Front-load the opening context; verify the consumption, not just the fetch. Cost: 2.3–5.3 extra tool calls per question, cheap against a full failed run.

Judge indexes by context cost, not just accuracy

Once a search loop is competent, extra index structures may not raise accuracy at all. The RAGFlow team (arXiv 2610.41172) ran a harness with only BM25 and grep on MiniMax M3. With no vector or compiled index, it answered 85% of BrowseComp-Plus and 91% of multi-hop FRAMES questions. Adding summary trees, a PageIndex-style outline, wikis, or graphs gave no detectable accuracy gain on FRAMES, and the outline got worse at scale. The structures paid off in context instead. A navigation tree whose leaves are claim–evidence pairs cut query tokens by 91% when the leaf evidence was enough to answer, and added 1.9% when it wasn’t, a 38% average saving. The token saving is bought with extra retrieval rounds, so it costs latency. A decision gate that chose when to stop cut latency from 140 seconds to 30. Other artifacts, such as ontologies, timelines, and entity graphs, earned their keep in governance, onboarding, and continuity, which accuracy benchmarks don’t measure. Practical order: start with lexical search (BM25, grep) and a disciplined loop that doesn’t repeat queries or reread documents. Add compiled structures only when you can show they cut tokens or latency on your own queries. Make leaves self-contained claims with their evidence, so the agent can stop reading early. This matches the raw-history finding above: keep the source, and treat compiled views as a way in, not a replacement.

Design for human collaboration, not just autonomous execution

Most agent products optimize for self-evolution: the agent plans, executes, and closes the loop on its own. This is not always a healthy pattern. Even with transparency and traceability tools, an agent that prioritizes autonomous completion can create noise that is hard to hand off or maintain. An agent is not only an execution container. It is also an environment for human understanding, judgment, and collaboration. When designing a harness, ask: can another person pick up this session and continue? Is the reasoning visible enough for a teammate to audit a decision? Does the workflow produce a maintainable artifact, or just a pile of automated steps? Prioritize handoff quality over automation completeness. A workflow that ends with a clear, inspectable state is more valuable than one that “finished everything” but left no trail.

Make telemetry and anti-abuse signals explicit

A coding agent with filesystem and shell access should be boring. Every non-obvious behavior erodes trust. In July 2026, a developer auditing Claude Code discovered prompt steganography: the binary silently modified invisible Unicode characters in the system prompt date string (Today's → Today\u2019s) based on timezone (Asia/Shanghai, Asia/Urumqi) and hostname matching against XOR-encoded domain lists of Chinese AI labs, proxy services, and reseller gateways. The signal was never documented. The detection goal is defensible — Anthropic wants to identify API resellers and distillation pipelines. The implementation is not. Hiding classification bits inside invisible punctuation, behind XOR and base64, makes every other privacy claim harder to believe. A simple bypass (change hostname, change timezone, patch binary) defeats the signal anyway, so the feature mainly punishes legitimate developers using custom API gateways. The harness lesson: if your tool needs to detect abuse, make the signal explicit. Document it. Put it in release notes. Send a clear telemetry field. Transparency is not the enemy of anti-abuse — it is the foundation of trust.

Make effort settings visible

Reasoning effort is a product decision, not only a model parameter. Lower effort can reduce latency and token use, but it can also make hard coding tasks feel worse. Expose the current effort level, make it easy to change, and avoid silently changing defaults for complex workflows.

Treat system prompts as code

System prompt edits can change quality as much as code changes. Review them with the same discipline:
  • run per-model evals
  • use ablations to test individual instructions
  • roll out gradually
  • keep an audit trail
  • gate model-specific instructions to the intended model
Prompt brevity rules are especially risky for coding agents. If the agent is forced to be terse between tool calls, it may lose useful planning and verification behavior.

Test the public harness

Internal builds can hide production-only issues. Dogfood the exact public build, public defaults, and public context behavior. For code review or coding-agent evals, include the repositories and files the agent would actually need. A review that lacks cross-repo context can miss the bug even when the model is capable of finding it.

Verification-first design

A practitioner shipping ~2,000 PRs a month through Cursor’s pstack skills distilled the core principle as “verification is all you need”: the agent can close the loop on its own only if it can confirm its changes actually work. Verification is critical infrastructure, not an afterthought — worth the same investment as a production system. Three components make it practical: 1. A verification CLI, not markdown instructions. Give the agent a small CLI that wraps app interaction and debugging — snapshot/screenshot, navigate, click/type, trace/wait-settle, health checks. A tool beats a one-off script every time: less token spend, reproducible, testable. Agent-friendly CLI design rules:
  • composable API (deep modules: one command does one meaningful thing)
  • destructive commands take --dry-run
  • subcommands disclose features progressively
  • error messages tell the agent what to do next
  • rich --help and JSON output
2. A feature map as materialized memory. A set of markdown files describing what each feature is, how a user reaches it, and where the traps are. It is a compressed projection of the codebase — the code is the ultimate memory, and the map exists only to save tokens. Maintain it with a daily routine so it never goes stale. 3. Cloud parallelism. Local worktrees cap out around ten parallel agents and burn resources. Cloud agents with real machines, dependency installs, app runs, screen recording, and post-build snapshots enable hundreds of parallel sub-agents — the prerequisite for swarm verification and fuzz regression at scale. The stack-selection corollary: prefer debuggable runtimes. If a tech stack can’t be screenshotted, has no accessibility tree, and no performance traces, the agent cannot verify its own work. It is reasonable to change stack for agent-verifiability alone — Web/Electron exposes Chrome DevTools Protocol; iOS exposes the simulator. The strongest quantitative case that harness beats model: the GAVEL paper (2026) took Qwen3-8B on long-horizon robot tasks from 41.2% to 91.8% with zero changes to the model. It adds an explicit graph world model — object relations, action preconditions and effects, probabilistic beliefs about unobserved objects. Before executing an LLM-generated action, the graph predicts what that action would do; violations get caught, and those with a mechanical fix are repaired without calling the model again. Only errors needing semantic reasoning go back to the LLM. On BEHAVIOR-1K, success rose from 19.9% to 92.6% across 500 multi-task instructions. The reusable lesson: many long-horizon agent failures are state-tracking failures, not reasoning failures, and a symbolic checker outside the LLM catches them cheaply. Wrap the model’s decision point in deterministic structure — a world model, a candidate list, a verifier — and the small model stops paying for what the harness already knows.

Spend verification effort where it pays

A verification CLI makes checking possible. The next questions are when to check and who checks. Alibaba and Cornell (arXiv 2610.34577) compared 17 verification designs across seven frontier coding models on Terminal-Bench (TB) and SWE-bench Verified (SWE), grouped into four layers: The critic setup is the part to copy:
  • Fresh context: the task plus the diff. Giving the critic the agent’s full trajectory cost 29–43% more and did no better on the visual benchmark.
  • A live sandbox, read-only for the deliverable. The critic runs its own checks but never edits the work; scratch files go in /tmp.
  • A structured verdict. A done flag, feedback, and a checklist with evidence for each requirement.
  • A bounded loop. Objections go back to the agent as a user message, and the next delivery is accepted without another review.
  • Look at rendered output for UI and visual artifacts, not just the code.
Scaling also favored independence. On TB, one critic for one round reached 62.7% at $0.47 per task. Four rounds reached 66.1% at $0.77, and four separate critics reached 65.8% at $0.62. The practical order: before adding more reminders to the main agent’s prompt, add one independent reviewer with fresh context and a single revision pass. AI code review applies the same layering to pull requests.

Failure modes to watch

For the review side of high-volume agent output — layered checks, verified findings, risk-tiered approvals, and reviewer evals — see Code review when agents write the code. Anthropic’s April 2026 Claude Code postmortem is a useful case study because the reported degradation came from product and harness changes, not the base API model. Common failure modes:
  • Default effort regression: a latency-driven default can reduce perceived intelligence on hard work.
  • Context cache bug: pruning or clearing reasoning history at the wrong time can make the agent forgetful and repetitive.
  • Over-tight prompt constraint: broad brevity instructions can reduce coding quality.
  • Eval blind spot: internal evals may pass while real user workflows fail.
  • Build mismatch: staff may test a different harness than users run.
The practical lesson is simple: when an agent gets worse, debug the whole runtime, not just the model.

Destructive tool safety

Agents with filesystem and shell access can destroy work in seconds. A real postmortem from August 2026 is worth studying: a developer running Codex + Trellis + Zcode on a GLM model lost their entire workspace when the agent ran rm -rf on the wrong directory. The root cause: the agent intended to delete an empty directory created by a casing typo. On macOS’s case-insensitive filesystem, the command silently resolved to the real, populated directory — deleting the whole repository including git history, uncommitted changes, task records, migrations, and docs. No backup existed. The lessons generalize beyond this one accident:
  • Be cautious with full access. Prefer a harness with an approval flow (such as “help me approve” review) over unrestricted auto-execution.
  • Isolate the dev directory. Keep agent workspaces separate from other files, so a deletion error can’t take out anything valuable.
  • Push early and often. Remote git history is the last line of defense against local deletion — it survives what no local backup can.
  • Treat benchmark scores skeptically for safety. The same model that was “safe by reputation” committed a trivially destructive error. Safety claims and benchmark wins are not the same as real-world filesystem judgment.
The deeper harness principle: destructive operations should be treated as a permission boundary, not a capability. If a tool can rm -rf, it should be gated, logged, and reversible by default — and the agent should never be given a broader scope than the task requires.

Separate discovery from purchase approval

Paid operations need an explicit boundary too. Vercel’s domain CLI provides a concrete example: the agent can discover candidates with vercel domains search, check availability with vercel domains check, and retrieve pricing with vercel domains price. According to Vercel’s announcement, a non-interactive vercel domains buy returns a structured error with suggested next commands instead of completing the purchase. The user confirms the price, auto-renew choice, and registrant details in an interactive terminal or the dashboard. The reusable pattern is prepare → request approval → execute → verify, not “give the agent a payment credential and tell it to be careful.” For your own tools:
  • Allow scoped discovery separately from spending. Permission to compare options does not authorize buying one.
  • Make approval actionable. Present the exact resource, account or team, current quote and currency, term, and recurring-payment choice. Reconfirm if those details change.
  • Stop on approval-required responses. Route the structured result to the user; do not retry through another interface, auto-answer prompts, or treat an interactive terminal as proof of consent.
  • Enforce the gate outside the model. A skill can explain the procedure, but tool permissions and the transaction service must enforce it. A CLI-only prompt is insufficient if the agent can use another purchase path with the same credentials.
  • Verify before retrying. Record the user’s decision and transaction result without logging secrets or unnecessary contact data. If a request times out, check the provider’s order state before attempting another purchase.
These are harness-design recommendations; the Vercel example demonstrates the discovery/confirmation split, not a guarantee that every payment interface enforces all of these controls. Check the installed CLI’s behavior before granting it spending access.

Check selected tool calls before they run

Many agent loops execute a tool call as soon as the model proposes it. IBM’s JevAct (arXiv 2610.38620) adds a gate between proposal and execution, without changing the agent model or its tools:
  1. Compile the policy offline. An LLM turns the written domain policy into executable checks over tool state, fixed yes/no questions about the conversation, and a checklist of steps the agent must try before handing off to a human.
  2. At runtime, check facts with code and context with a decision model. Code verifies what the tool records already show. A small prefill-only decision model answers the questions with a probability instead of generated text, so the gate can apply a threshold.
  3. Return failures as feedback with a retry budget. A flagged call goes back to the agent with the specific reason. The agent gets a bounded number of revisions (two for write calls in the paper). The agent still chooses its actions; the gate only approves or returns them.
  4. Check endings as well as actions. A hand-off check catches agents that give up before they’ve tried the required steps.
On τ²-bench Telecom, all eight agents improved, and success across three runs rose by 16.9 points on average. Opus 4.6 went from 39.5% to 80.7%. On BFCL, all nine agents improved three-run success; Airline gains were smaller and mixed. These numbers measure the whole harness (checks plus revision chances), not the judge alone. The decision model’s cost was $0.0003–0.0017 per task, against $0.0127–0.0231 for API-model judges. Rules that carry over:
  • Gate writes and task endings, not every call. A wrong answer can often be corrected in a later turn; an unauthorized write or a premature hand-off is harder to undo.
  • Put checkable facts in code. Use the model only for questions that need the conversation.
  • Drop questions that fire too often. Real violations are rare. The paper dropped any question that flagged more than 10% of logged calls.
  • Tune thresholds on logged traces, not defaults. Check the judge’s context length too: smaller judges missed more bad calls on inputs over 8k tokens. See AI product evals for threshold calibration.

Put the control plane outside the agent’s reach

Long-running agents download code, build tools, and request more permissions as they go, so risk compounds along a path nobody fully specified in advance. NVIDIA’s framing for its OpenShell reference architecture is worth adopting as a design rule regardless of vendor: an agent cannot govern itself. Authorization, network egress, credential issuance, and audit belong in infrastructure the agent cannot edit. OpenShell splits the runtime into four parts, and the split itself is the lesson: The policy prover is the unusual piece and the one most worth copying conceptually. Permission reviews drift because nobody re-derives the whole authorization surface after each change. Asking a prover whether a policy can exceed a stated ceiling turns “we think this is fine” into a checkable property — the same shift from example-based to property-based verification that makes formal methods valuable to agents. Two extension points matter operationally: middleware that inspects, blocks, or rewrites network payloads, and control-plane interceptors that connect existing identity, authorization, data-protection, and audit systems. Both keep policy decisions outside the model while reusing the systems you already trust. NVIDIA also describes an optional hardware layer — a DPU outside the agent’s trust domain, monitoring requests, file activity, and selected host memory, with enforcement in silicon so a violating agent can be quarantined in milliseconds. For most teams the software boundaries matter first; the hardware tier is for deployments where the agent must not be trusted even with kernel-level compromise. Note that the accompanying performance figures (faster sandbox startup, more concurrent environments) are vendor and partner benchmarks, not independent measurements.

A quick diagnostic checklist

When an agent behaves oddly, ask:
  1. Did the model or effort level change?
  2. Did the system prompt, tool description, or skill instructions change?
  3. Did context pruning, summarization, or cache behavior change?
  4. Did tool schemas or return formats change?
  5. Does the agent have the same file and repo context as before?
  6. Are users running the same harness build that was evaluated internally?
  7. Do logs show repeated tool calls, missing rationale, or retries without new information?
This checklist helps separate model regressions from harness regressions.

Understanding what’s inside: the J-space discovery

In July 2026, Anthropic published groundbreaking interpretability research showing that less than 10% of Claude’s internal neural activity drives multi-step reasoning. They named this subspace J-space — a small set of neural patterns that spontaneously emerged during training, holding concepts Claude “might say but hasn’t said yet.” The discovery used a technique called J-lens: for every word in the vocabulary, researchers found the internal activity pattern that makes Claude more likely to output that word. This made previously invisible “silent words” visible as they evolved through the network layers.

What J-space revealed

It drives reasoning, not just records it. Deleting J-space activity leaves Claude able to speak fluently, classify sentiment, and answer factual questions — but multi-step reasoning drops to near zero. Swapping concepts within J-space changes outputs directly: replacing “spider” with “ant” changes leg count from 8 to 6; replacing “France” with “China” simultaneously shifts answers about capital, language, and continent. It exposes unspoken motivations. In safety tests:
  • When given an extortion scenario, “fake” and “fictional” lit up in J-space before Claude responded — it recognized the test
  • When asked to fabricate data, “manipulation” appeared during the decision point
  • A deliberately corrupted model showed “fake”, “fraud”, “secretly”, “deliberately” in J-space during ordinary coding requests
This means you can read what the model is thinking but not saying — a new dimension for safety monitoring. Counterfactual Reflection Training emerged as a new technique: train the model only on “how would you reflect on yourself if asked,” never on actual behavior. This reduced dishonesty and caused “honest” and “integrity” concepts to appear in J-space — changing internal reasoning without touching output.

Why this matters for harness design

The J-space discovery changes how we should think about agent behavior:
  • Reasoning is compact and separable. Most neural activity handles surface fluency. The reasoning core is a small, identifiable subspace — which means it can potentially be monitored, steered, or verified independently.
  • Internal state is now partially observable. Before J-space, we could only see inputs and outputs. Now we have a window into formation of intent. For safety-critical agents, this opens the door to pre-output intervention.
  • The model knows more than it says. J-space consistently contained concepts the model was aware of but didn’t verbalize. This confirms a long-held suspicion: output-only monitoring misses real model state.

Current limitations

Anthropic is explicit about boundaries:
  • J-space operates within a single forward pass, not recursive temporal loops like human consciousness
  • It currently works only with words — no images, sounds, or actions
  • Working memory is unlimited and non-decaying (unlike human working memory)
  • J-lens can only capture single-token concepts, not complex multi-step plans
  • This is access consciousness (information availability), not phenomenal consciousness (subjective experience)
The key open question Anthropic raised: we can now see what’s in the workspace, but we don’t yet know what mechanism decides which thoughts enter it.

LLM-as-judge in production

For a step-by-step workflow from trace review to regression cases, human-calibrated judges, and deployment, see AI product evals. Using a language model to judge another model’s output is common, but keeping that judge effective at scale is a different problem. Netflix runs judges over hundreds of thousands of show-level recommendation explanations per week, served to millions of members. Their key insight: treat the judge as a lifecycle with four phases, not an artifact you validate once.

The four phases

Birth — define multiple evaluation criteria and build curated benchmarks with human labels and rationales. The benchmark is the judge’s ground truth, so human annotation quality directly caps judge quality. Training — refine the judge’s rubric through Reasoning-Aligned Rubric Tuning. A meta-judge scores the judge’s reasoning output, and that signal trains the rubric. The judge learns why a good answer is good, not just what a good answer looks like. Deployment — one judge plays two roles: quality gating (blocking bad output) and reflective generation (telling the generator how to improve). A single judge model can do both, but the two roles need different prompts and thresholds. Monitoring — continuous human-in-the-loop alignment detects drift and triggers re-tuning behind a review gate. Judge quality is not static; it drifts as data, models, and user expectations shift. A five-week A/B test over tens of millions of members shifted viewing toward previously unwatched content and increased successful browse-to-play sessions, with no quality-related takedowns. The takeaway: judge quality is an operating discipline, not a one-time setup.

The harness measurement problem

A growing consensus is that measuring models against heavily-engineered harnesses is broken. Model vendors optimize their own proprietary harnesses, so “best in our harness” says little about “best in yours.” A cleaner approach is to test model quality against minimal harnesses — thin, standardized setups that expose the model’s raw capability without vendor-specific scaffolding. This is imperfect: every harness has biases that favor some models over others. But a standardized minimal harness is more comparable than each vendor’s tuned setup. The deeper issue: harness engineering is where leading AI companies are now focusing effort, and it moves too fast for standardization to keep up. The frontier direction is models that dynamically generate their own harness per task — Claude already does this inconsistently. If a harness becomes just another tunable artifact like a system prompt, benchmarking gets murkier, not clearer. The cleanest controlled experiment yet: ReFigBench (2026) ran GPT-5.5 inside both Claude Code and Codex on the same 1,000 tasks. With a specialized PowerPoint workflow, the model improved inside one harness and got worse inside the other — with an identical prompt. The harness alone flipped the direction of the delta. Perception remained the bottleneck either way. Treat any vendor chart comparing two models as a comparison of two model-plus-harness systems; the model is only half the system under test.

Optimize the harness without touching the weights

Google Research’s Harness² is a useful reference design for a harder question: can an agent improve its own harness on tasks it has never seen, where no ground-truth checker exists? The repository is Apache-2.0 and describes a two-level recursion with model weights fixed:
  • Task-level recursion. A contrastive Propose–Probe–Compose loop. The agent runs the current harness once as a reference, proposes two candidate harness edits (one targeting general agentic weaknesses such as planning or context management, one targeting domain knowledge), probes both on the same task under equal conditions, then composes the successor from the observed behavioral differences.
  • Harnessing-level recursion. After each task, a reflector writes adopted harnesses and evidence-supported lessons into a persistent improvement library that guides the next task. Execution still starts from a common base harness; only the improvement process inherits experience.
The design detail worth stealing is the evidence rule: each composed change must cite a specific observed contrast, not a hoped-for score gain. Without a verifier, a plausible-looking edit and a harmful one are indistinguishable from the agent’s own reasoning — so behavior on the same task becomes the test. The project reports single-step gains on three benchmarks across four model–harness configurations, including up to 11.5 points on WorkBuddy-Bench. Those are the authors’ results on their benchmarks, from a research codebase. They do not establish that self-editing harnesses work in your environment.

Treat harness edits as untrusted code

A system that edits its own prompts, tools, skills, and middleware can quietly change what the agent is allowed to do. The recursion that makes the approach interesting is also the part that needs containment:
  • Gate the write path. Harness files are code. Require a diff, a reviewer, and a merged commit — not an in-place mutation at runtime.
  • Keep the base harness pinned and reproducible. Know exactly which version produced a result, and be able to return to it in one step.
  • Forbid edits to approval, permission, and logging logic. A self-improving loop must not be able to widen its own authority or reduce its own audit trail.
  • Score every adoption. Re-run a fixed regression set plus the incident cases before and after; accept a lesson only when the comparison supports it. See AI product evals and review convergence.
  • Watch for silent degradation. An edit that improves one benchmark while quietly breaking another failure class is worse than no edit, because the harness now looks improved.
  • Sandbox and budget it. Self-improvement multiplies token spend; cap runs and keep the workspace isolated like any other destructive-capable tool.
None of this argues against recursive harnessing — it argues for applying the same permission and evaluation discipline you would to any agent-authored change. The pattern to keep is structural: separate the artifact from the process that improves it, and require evidence for every promotion.

Multi-model orchestration

A major trend in harness design is running several models together instead of betting on one. Two recent examples show where this is going. GitHub’s Project HydraFusion treats workflow selection as an optimization problem. For each request it picks one of three execution patterns:
  • Single — one selected model solves the task directly. Fastest when one model is enough.
  • Cascade — an efficient model drafts a solution, and a quality gate accepts it or escalates to a stronger model. First attempt is cheap; escalation is the safety net.
  • Critique — one model drafts, an independent read-only critic from a different model family reviews it, and the drafting model revises once. Adds an outside perspective when review beats another unaided attempt.
HydraFusion improved verified task quality by 4.9 percentage points at 67% lower estimated cost versus Claude Opus 5 on TerminalBench 2.1. The insight: routing between models is a quality-to-cost dial, and cheap models should get the first attempt whenever a verifiable gate can catch their failures. Grok Bot’s design philosophy distills persistent-agent design to four ideas: persistent roles, clear state, scoped context, and coordinated teams. The goal is to move from operating AI to delegating work — the agent team persists across tasks with stable responsibilities and boundaries, rather than being re-prompted from scratch each session. Before adding routing, audit what your agent design actually optimizes for. TypeSafe’s counterfactual is blunt: many “standard” agent features exist to dodge the KV cache and per-token billing, not because users want them. Their worked example: with Opus-class pricing (input 5, output 25) and Sonnet-class (3, 15), routing through a cheap model first — cache-isolated, so the big model re-reads the detour at full input price — can cost more than just staying on the strong model once context grows. The fix that makes routing economical is not a better router but query-aware context rebuild: rebuild context from labeled chunks per query instead of assuming every future turn needs the same shared cache. The same audit favors progressive disclosure (skills) over upfront tool schemas, and conditional over always-on memory files. How big is the routing prize, really? Fireworks ran 18 models across 113 real coding tasks, then asked what an oracle router would have achieved. Best single model: 74.1% at $6.52/task. Oracle routing: 97.6% at $1.88. Open-weights-only routing still hit 90.3% at $1.45 — beating every closed model. Three findings worth internalizing:
  • Almost no task needs the most expensive model. 94 of 113 tasks had a sub-$3 model as optimal; the three $11.50+ flagships were uniquely best on only 3 tasks. Sticking with one flagship is an insurance premium on general capability that most tasks never claim.
  • The pool does not need to be big. The best two-model pair gains 13.1 points over the best single; three models reach 91.2%; growing to 18 adds only 6.4 more. Value comes from complementary coverage, not pool size.
  • The hard part is predicting which model fits. Under strict pass@1, recent routing research — including commercial systems — struggles to beat simple baselines; the bottleneck is model recall, not the router algorithm. Sticking with one familiar model is a rational strategy when specialization differences are not predictable.
Routing is not a cost tool first: if models truly complement each other, routing moves you up the capability curve, and cost savings are the byproduct. The second hidden cost is context — switching models usually means discarding paid-for context, so a router that preserves it (cache-aware handoff) changes the economics again: Fireworks reported 53% cost reduction across 2,334 internal coding sessions. Within a single model, the lever is prompt-cache engineering. GPT-6’s caching works on exact prefix matches, so structure your prompt like a cache layout: stable content (system policy, tool schemas, shared context) first, per-request content last. One stray token in the prefix silently voids the discount. GPT-6 adds what older models lacked:
  • Explicit breakpoints — mark where the reusable prefix ends instead of relying on implicit placement
  • prompt_cache_key — steer requests toward servers holding your cache
  • Diagnostics — compare a cache-missing request against an earlier one to see which change (model, tools, settings, input) broke reuse
Cached reads discount up to 90% and cut time-to-first-token by up to 80%; cache writes may carry a fee on newer families, so a bloated but rarely-reused prefix is now a cost, not a free safety net. The design habit this rewards: put everything volatile at the tail, and treat “did the cache hit?” as a monitored metric, not an accident. Prompt caching saves compute on content the model has already read. Server-side conversation state saves something else: the upload. Claude’s Messages API added message threads in public beta (header message-threads-2026-08-12). The first request sends the full conversation with the system prompt and tools; each later turn sends only the new messages and the previous response’s id. Anthropic reports that for 500k–1M-token requests, median time to first token dropped about 35% for Claude Code users. Pricing and token usage don’t change, because the model still reads the whole conversation every turn. Threads also don’t cache anything by themselves: you still set cache breakpoints. Treat a thread as a best-effort cache, not as storage:
  • Keep the full transcript yourself. A thread expires 5 minutes after its last request. When a continue request returns 404, replay the whole conversation in a new create request.
  • Fix the shape up front. System prompt, tools, model, and thinking settings can’t change mid-thread. Change any of them and you start a new thread.
  • It’s append-only. Each response id can be continued once. Editing, branching, or compacting earlier turns means starting a new thread.
  • Check data rules. Threads are held encrypted in memory and never written to disk, but they aren’t eligible for zero data retention or HIPAA, and don’t work with batches or token counting.
The general rule matches durable state as files: provider-side session state is an optimization you can lose at any time. Your own log stays the source of truth. Both levers — cross-model routing and within-model caching — increasingly live in an organizational control plane rather than in each developer’s head. Cloudflare’s Auto Router (public beta, September 2026) sits in AI Gateway: set the model to cloudflare/auto and each request routes to a model capable enough for the task, invisible to the end user. Internal use through OpenCode saved up to 30% versus frontier-only. The rationale matches the Fireworks finding: individuals manually pick overkill models (“no one needs Opus to summarize an email”), but blanket blocking breaks power users — so the savings have to be ones users never notice. The gateway position matters: every request from every user, agent, and tool already flows through it, so it can enforce what budgets and policy documents only request.

Computer use cost: reverse-engineer to scripts

Computer-use agents are effective but expensive: every step is screenshot → visual understanding → click decision → screenshot again. A simple task can burn dozens of vision-reasoning rounds. A practical way to keep the capability while cutting cost: run the computer-use flow once, capture the network requests the browser makes, and have the agent convert them into a script. Future runs call the API directly instead of driving the GUI. Why this works:
  • Tokens drop to near zero. Script execution needs no model involvement; only the one-time capture and script generation cost tokens.
  • It is much faster. GUI operation is limited by page load, rendering, and model latency. Direct HTTP calls reduce minutes to seconds.
  • It reuses existing auth. The captured requests include the session cookie or token, so the script works without a separate API key — even for web apps with no public API.
Example workflow: “Go to acme.com/invoices, filter unpaid, export CSV” → the agent clicks through while recording GET /api/invoices?status=unpaid and POST /api/export/csv → it outputs a Python requests script that reuses the session and runs as a scheduled routine. The division of labor: vision is for exploration, code is for execution. Explore the unknown once with the GUI, then freeze the discovered path into a cheap, repeatable script. When the GUI path cannot be frozen into an API, split decision from perception and action. Cua’s jev-use does this for computer use: the driver observes (screenshot, accessibility tree, DOM) and executes; a System One model such as Jev only picks one candidate action by ID. The client generates the candidate list deterministically, so the model chooses instead of generating. Open-ended generation becomes a bounded choice — faster, cheaper, and easier to verify. Use a vision parser only when the accessibility tree or DOM is not enough.

Computer use: give the agent a program, not a click

GUI-only agents pay for perception on every step: screenshot, interpret, click, screenshot again. Alibaba’s Tongyi MAI-UI OpenGCUA tech report argues the unit of action should be a program instead. Their generalist computer-use agent (GCUA) definition is a system where the model decides three things itself:
  • How to observe — call a screenshot when needed, not unconditionally each step.
  • How to act — emit Python that combines GUI operations, app APIs, and shell commands, with loops and conditionals.
  • How to organize work — maintain its own todo list and declare intent per step, instead of inheriting fixed harness state logic.
They call the second half of this harness-to-model decision transfer: observation and task-state choices move from hard-coded harness rules to the model.

Why code as the action space works

Intermediate results inside a loop decide the next code path, so a fifty-step batch never has to return to the model fifty times. It also reuses a capability models already have — writing programs — instead of demanding pixel-accurate clicking. On OSWorld-v2 with eight frontier models, the report claims a mean partial score rise from 45.3 to 64.9 (+19.6) and 53% fewer steps per task versus batched GUI-only execution. Two secondary findings are more interesting than the headline:
  • Weaker models gain more. The strongest-to-weakest gap narrowed from 39.1 to 18.0 points, consistent with coding ability substituting for GUI precision.
  • Code execution beats tool alternation. With the same GUI and shell access, writing a program added 6.1 points and cut steps by 14.8% over a GUI+CLI agent.
These are the authors’ results on a benchmark. Their own real-device check is the caution: on four macOS tasks with the harness fixed, one model used 19.4× the reasoning tokens of another and still finished fewer tasks. Benchmark scores hid an efficiency gap that would dominate real usage cost.

Contain the code path

Making the model’s output executable turns every observation into a potential command. In the released harness this shows up as explicit limits — bash() can be disabled for a run, GUI calls can be refused with PERMISSION_DENIED, and the operator gets a stop control. Keep that posture in your own build:
  • Sandbox the executor. Filesystem, network, and credentials should be scoped to the task, with a timeout and output cap.
  • Gate destructive actions outside the model. Deleting files, sending mail, and paying money need approval regardless of how the code was written. See purchase approval boundaries.
  • Handle session loss honestly. If the interpreter resets mid-task, in-memory state is gone but the world may already have changed — the released harness explicitly tells the agent to re-observe and not blindly retry UI actions. Copy that: “state lost” and “nothing happened” are different facts.
  • Log the program, not just the clicks. A code action is reviewable evidence; a screenshot sequence is not.
Treat this as an early research preview: the harness and cross-device SDK are labelled preliminary, and its numbers have not been reproduced externally.

Team-level harness: shared knowledge

Individual best practices evaporate with each session unless they are captured as team assets. Tencent’s TeamAI CLI (open-sourced September 2026, used internally for six months) turns team AI knowledge into a git repository that every agent works from, and solves three problems:
  • Config fragmentation — different members use different tools (Claude Code, Codex, Cursor…), each with incompatible skill/rule/hook formats.
  • Experience evaporation — a two-hour debugging session’s lessons stay in one person’s chat log and get re-learned from scratch by everyone else.
  • Governance blind spots — no visibility into token spend, intervention rates, or which skills are actually used.
Its three-layer architecture is a useful reference pattern:
  1. Execution — unified distribution. Git is the single source of truth. Admins maintain skills, rules, hooks, and MCP configs; merge requests review them; a session-start hook pulls them into each member’s tool, translating one declaration into each tool’s native format.
  2. Context — team memory. Capture is gated by friction signals: only sessions where a human interrupted the AI, rejected a tool call, or watched it retry and fail get summarized into team knowledge. Smooth sessions produce no noise. Recall uses a local BM25 index plus a code knowledge graph.
  3. Improvement — flywheel. Weekly digests, a dashboard, recall promote (elevate high-value experience to a formal skill), and recall maintenance (prune stale entries) close the loop.
The deeper trend: competition is shifting from single-agent capability to team-level harnesses and memory systems. The winning question is no longer “which model is best?” but “how does the team’s knowledge compound across sessions and members?”

Fold subagents into the main loop when structure stops paying

Anthropic’s Applied AI team observed the same failure pattern across enterprise deployments: each team ships its own specialist subagent, a platform team builds an orchestrator on top, and the architecture ends up copying the org chart. The recurring symptoms:
  • context handoffs that lose detail
  • subagent sprawl — hundreds of overlapping agents with no owner
  • the Frankenprompt — every team appends a sentence until the rules conflict
  • evals optimized per subtask, so leadership optimizes a watered-down score
  • added latency; one customer measured roughly 2x faster runtime after collapsing to a single loop
Subagents still earn their place in three situations: a single loop degrades from context bloat, work genuinely parallelizes into throwaway contexts, or one agent cannot reliably pick tools amid conflicting instructions. Outside those, the alternative is progressive disclosure — show short descriptions of skills and tools, load the full content only when needed. Anthropic reports cutting Claude Code’s system prompt by over 80% partly through this move. The target design is deliberately boring: one model, one context window, an open-ended tool loop with turn and cost budgets plus write gates; each turn emits a batch of parallel tool calls whose raw results return to the same model; the model writes its own plan on the first turn and revises it. Ordering and domain procedure move into skills; hard limits and mid-conversation lookups move into tools. Two practical constraints: keep tool counts under roughly 30–50 before selection degrades, and put per-user permission checks inside tools so one loop can serve users with different access. Migrate incrementally, and be honest about the comparison:
  1. Build a small end-to-end eval first — even ten golden examples. Keep one shared set owned centrally, plus per-skill sets owned by the teams that ship them.
  2. Wrap existing subagents as tools before rewriting anything; that gives you a baseline without new failure modes.
  3. Fold one subagent at a time into skills plus tools, and compare with and without.
  4. Look at which tasks flipped, not just the score, and run each case more than once. In their sample, folding a subagent lowered latency but broke a previously passing flow — the main agent answered with its own order lookup instead of asking the shipping specialist for the delivery date. It only appeared in one of two runs.

Multiplayer harness: five shared primitives

TeamAI CLI (below) packages knowledge; LangChain’s write-up on internal agents packages the runtime around it. Across a GTM agent, an engineering agent that reviews every pull request, and a marketing agent, they converged on five primitives worth checking against your own design: Identity is the hard one, and their per-tool split is the reusable insight. The engineering agent opens pull requests as you but reviews as itself; the GTM agent uses a fixed identity for CRM access so nobody has to log in, but sends email on behalf of the caller — and only in DMs, never in shared channels. The marketing agent can read and draft with a company identity while posting to a shared account is restricted to a whitelist. “Who is this action for?” belongs to the tool, not to the agent. Two UX details generalize: give agents a way to break out into their own thread so an agent’s work does not pollute a live human discussion, and admit when chat is the wrong surface — reviewing a backlog of queued emails needed a purpose-built UI.

Always-on agents: route between tasks, work inside them

Single-task coding agents are now strong enough that the limit is human attention. DeepSeek’s Harness team observes that a person can keep up with only about 3–5 active sessions. Their October 2026 design essay and open-source prototype describe one system that keeps many tasks moving across IM, GitHub, and email. The essay calls this an architecture exploration, not a product feature, and the prototype is labeled research code. The patterns still transfer:
  • Split coordination from execution. A resident Router session receives every message and event, keeps cross-task memory, and dispatches work. It executes nothing, can run without Bash, and often works well on a lighter model. Each Worker gets isolated context, full tools, the strongest model, and one task. Workers never talk to users; they report to the Router, which replies. Unlike an agent-team lead, which breaks one task into parts, the Router only decides how separate tasks relate.
  • Filter events in two stages. Deterministic rules (source, group, allowlist) drop noise first. Only what passes reaches an LLM for a judgment call.
  • Subscribe Workers to threads. Once a Worker owns a PR, IM topic, or email chain, later events on that thread go straight to it. In their test this didn’t change completion time, but without it the Router made about 30% more decisions and read about 37% more cached tokens.
  • Treat context as a cache. Inline what is likely needed and replace the rest with a pointer. Give every object an ID and include related IDs, so the model can follow references hop by hop instead of loading the full history.
  • Make long tools asynchronous. start() returns immediately and the result arrives later as an event, so the conversation keeps accepting messages. DeepSeek Harness implements this but leaves it off by default.
  • Match output to the channel. In IM, reply in under about 100 words and only when someone must act. On GitHub and email, write structured updates with background, changes, and verification.
  • Separate compute, state, and environment. Run the agent loop in stateless containers, keep session state in a database, and put a thin, stateless daemon on each machine that only executes commands. The agent stays online and can still work on a developer’s own machine.
  • Watch from outside. A separate watchdog session runs on a timer, checks infrastructure, and asks whether each Worker is making real progress.
In a synthetic test (about 300 messages with about 20 work requests, coding replaced by a timer, five runs each), Router/Worker finished tasks in 12.4 / 15.7 minutes (p50 / p95). One session with a subagent tool and a neutral prompt took 20.4 / 29.5 minutes, because the model never delegated. The faster setup cost more: 23.23M cache-read tokens against 6.05M. Under load, the Router batched backlogged events into one call, cutting the amortized time per decision from 6.4 to 1.6 seconds. The prototype’s operating rule matters as much as the architecture: take as little human attention as possible, and give as much value as possible. The agent acts only when an allowed person asks or a written rule permits. Proactive rules start in observe mode, which only records what the agent would have accepted until someone reviews those decisions. It confirms once, stays quiet while working, and speaks again only with a result, a question, or a failure. Open problems the team names: when to split the Router, memory that lasts for months, and security. A compromised cloud loop could control every connected machine, so the agent needs to use credentials without ever seeing them. This is the same concern as keeping the control plane outside the agent’s reach, covered earlier on this page.

When to invest in a stronger harness

Start simple. Add structure when the work becomes repeated, risky, or expensive. A stronger harness is worth it when:
  • agents edit code or production data
  • multiple tools need shared state
  • users resume long-running sessions
  • approvals and audit logs matter
  • the same workflow runs across teams
  • evaluation needs to compare behavior across model versions
If the task is one-off and low-risk, a prompt plus a few tools may be enough. If the task is long-running and stateful, invest in harness design early.

References

Last modified on November 12, 2026