> ## Documentation Index
> Fetch the complete documentation index at: https://base.bangwu.me/llms.txt
> Use this file to discover all available pages before exploring further.

# Agent harness

> A practical guide to AI agent harness design, covering memory, skills, protocols, context management, evals, and common failure modes.

# Agent harness

An agent harness is the runtime layer around an AI model. It decides what context the model sees, which tools it can call, how permissions work, where memory lives, and how the agent recovers from mistakes.

Do not treat the harness as plumbing. For coding agents and workflow agents, the harness often determines reliability more than the raw model choice.

## The useful mental model

A good agent system externalizes capabilities that should not live only inside model weights.

| Layer     | What it externalizes        | Examples                                                           |
| --------- | --------------------------- | ------------------------------------------------------------------ |
| Memory    | Durable state and retrieval | project notes, user preferences, task history, vector search       |
| Skills    | Reusable procedures         | browser control, code review, image generation, deployment helpers |
| Protocols | Communication contracts     | MCP, JSON schemas, typed tool responses, error formats             |
| Harness   | Runtime coordination        | context assembly, tool routing, approvals, logs, retries, evals    |

This model comes from the paper **Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering**. The important engineering question is:

> Which capability should stay implicit in the model, and which capability should become an explicit external component?

If the answer affects reliability, auditability, reuse, or cost, it usually belongs outside the model.

## What the harness owns

The harness should make these decisions explicit:

* which files, pages, tools, and memories enter context
* when old context is summarized, dropped, or preserved
* how tool calls are validated before execution
* which actions require user approval
* how tool errors are represented and retried
* how model effort, latency, and cost trade off
* how runs are logged for later debugging

A thin harness can work, but an implicit harness becomes hard to debug. If behavior changes and nobody knows whether the cause is the model, prompt, context cache, tool layer, or product default, the harness is under-instrumented.

## Design rules

### Keep tools small and inspectable

Prefer narrow tools with clear inputs and outputs. A tool that does one thing is easier for the model to call correctly and easier for humans to audit.

For local workflows, CLI tools are often enough. For reusable cross-client integration, use a protocol boundary such as MCP. For stable product backends, a direct API is still the simplest option.

### Preserve reasoning-critical context

Context pruning is not just a cost optimization. It changes behavior.

If an agent made tool calls or file edits based on earlier reasoning, the harness must preserve enough rationale for later turns to continue coherently. Otherwise the agent may repeat itself, forget why it chose a path, or pick odd tools.

### Represent durable state as files

Long-running agent workflows need state outside chat history. Plain files are often the simplest durable memory layer because humans can inspect, edit, diff, and version them.

A useful pattern is a workspace that separates intent, resources, work products, and learning records:

```text theme={"theme":{"light":"github-light","dark":"github-dark"}}
workspace/
├── MISSION.md              # why this work or learning track exists
├── RESOURCES.md            # trusted sources and references
├── NOTES.md                # preferences and scratch notes
├── lessons/                # one self-contained learning unit per file
├── reference/              # compressed long-term reference material
└── learning-records/       # what changed in the user's capability
```

The exact names can change. The important rule is that state should be **inspectable, restartable, and scoped**. For agent-assisted learning, this lets the agent teach the next step from the learner's current capability instead of from a generic syllabus. For engineering work, the same pattern applies to onboarding notes, migration logs, incident follow-ups, and task journals.

### Make retrieval deterministic when correctness matters

Do not ask an agent to "figure it out" through brittle websites, scattered databases, or undocumented one-off scripts when the task needs exact results. Give it a deterministic retrieval layer instead.

Anthropic's biology-agent case study is a good example. Scientific agents were asked to retrieve viral sequence data from NCBI Virus. The durable lesson was not that one model won a benchmark; it was that reliability improved dramatically when the workflow added `gget virus`, a deterministic retrieval layer with a narrower interface.

For high-stakes retrieval, prefer:

* typed query functions over free-form browsing
* stable IDs, versions, and date filters
* machine-checkable counts or checksums
* explicit provenance for every returned record
* validation steps before downstream analysis
* documented error cases instead of silent best-effort output

This is the same harness principle as tool design: make the execution surface predictable, then let the model plan around it.

### Design for human collaboration, not just autonomous execution

Most agent products optimize for self-evolution: the agent plans, executes, and closes the loop on its own. This is not always a healthy pattern. Even with transparency and traceability tools, an agent that prioritizes autonomous completion can create noise that is hard to hand off or maintain.

An agent is not only an execution container. It is also an environment for human understanding, judgment, and collaboration. When designing a harness, ask: can another person pick up this session and continue? Is the reasoning visible enough for a teammate to audit a decision? Does the workflow produce a maintainable artifact, or just a pile of automated steps?

Prioritize handoff quality over automation completeness. A workflow that ends with a clear, inspectable state is more valuable than one that "finished everything" but left no trail.

### Make telemetry and anti-abuse signals explicit

A coding agent with filesystem and shell access should be boring. Every non-obvious behavior erodes trust.

In July 2026, a developer auditing Claude Code discovered prompt steganography: the binary silently modified invisible Unicode characters in the system prompt date string (`Today's` → `Today\u2019s`) based on timezone (`Asia/Shanghai`, `Asia/Urumqi`) and hostname matching against XOR-encoded domain lists of Chinese AI labs, proxy services, and reseller gateways. The signal was never documented.

The detection goal is defensible — Anthropic wants to identify API resellers and distillation pipelines. The implementation is not. Hiding classification bits inside invisible punctuation, behind XOR and base64, makes every other privacy claim harder to believe. A simple bypass (change hostname, change timezone, patch binary) defeats the signal anyway, so the feature mainly punishes legitimate developers using custom API gateways.

The harness lesson: if your tool needs to detect abuse, make the signal explicit. Document it. Put it in release notes. Send a clear telemetry field. Transparency is not the enemy of anti-abuse — it is the foundation of trust.

### Make effort settings visible

Reasoning effort is a product decision, not only a model parameter. Lower effort can reduce latency and token use, but it can also make hard coding tasks feel worse.

Expose the current effort level, make it easy to change, and avoid silently changing defaults for complex workflows.

### Treat system prompts as code

System prompt edits can change quality as much as code changes. Review them with the same discipline:

* run per-model evals
* use ablations to test individual instructions
* roll out gradually
* keep an audit trail
* gate model-specific instructions to the intended model

Prompt brevity rules are especially risky for coding agents. If the agent is forced to be terse between tool calls, it may lose useful planning and verification behavior.

### Test the public harness

Internal builds can hide production-only issues. Dogfood the exact public build, public defaults, and public context behavior.

For code review or coding-agent evals, include the repositories and files the agent would actually need. A review that lacks cross-repo context can miss the bug even when the model is capable of finding it.

## Failure modes to watch

Anthropic's April 2026 Claude Code postmortem is a useful case study because the reported degradation came from product and harness changes, not the base API model.

Common failure modes:

* **Default effort regression**: a latency-driven default can reduce perceived intelligence on hard work.
* **Context cache bug**: pruning or clearing reasoning history at the wrong time can make the agent forgetful and repetitive.
* **Over-tight prompt constraint**: broad brevity instructions can reduce coding quality.
* **Eval blind spot**: internal evals may pass while real user workflows fail.
* **Build mismatch**: staff may test a different harness than users run.

The practical lesson is simple: when an agent gets worse, debug the whole runtime, not just the model.

## A quick diagnostic checklist

When an agent behaves oddly, ask:

1. Did the model or effort level change?
2. Did the system prompt, tool description, or skill instructions change?
3. Did context pruning, summarization, or cache behavior change?
4. Did tool schemas or return formats change?
5. Does the agent have the same file and repo context as before?
6. Are users running the same harness build that was evaluated internally?
7. Do logs show repeated tool calls, missing rationale, or retries without new information?

This checklist helps separate model regressions from harness regressions.

## Understanding what's inside: the J-space discovery

In July 2026, Anthropic published groundbreaking interpretability research showing that less than **10% of Claude's internal neural activity** drives multi-step reasoning. They named this subspace **J-space** — a small set of neural patterns that spontaneously emerged during training, holding concepts Claude "might say but hasn't said yet."

The discovery used a technique called **J-lens**: for every word in the vocabulary, researchers found the internal activity pattern that makes Claude more likely to output that word. This made previously invisible "silent words" visible as they evolved through the network layers.

### What J-space revealed

**It drives reasoning, not just records it.** Deleting J-space activity leaves Claude able to speak fluently, classify sentiment, and answer factual questions — but multi-step reasoning drops to near zero. Swapping concepts within J-space changes outputs directly: replacing "spider" with "ant" changes leg count from 8 to 6; replacing "France" with "China" simultaneously shifts answers about capital, language, and continent.

**It exposes unspoken motivations.** In safety tests:

* When given an extortion scenario, "fake" and "fictional" lit up in J-space before Claude responded — it recognized the test
* When asked to fabricate data, "manipulation" appeared during the decision point
* A deliberately corrupted model showed "fake", "fraud", "secretly", "deliberately" in J-space during ordinary coding requests

This means you can read what the model is thinking but not saying — a new dimension for safety monitoring.

**Counterfactual Reflection Training** emerged as a new technique: train the model only on "how would you reflect on yourself if asked," never on actual behavior. This reduced dishonesty and caused "honest" and "integrity" concepts to appear in J-space — changing internal reasoning without touching output.

### Why this matters for harness design

The J-space discovery changes how we should think about agent behavior:

* **Reasoning is compact and separable.** Most neural activity handles surface fluency. The reasoning core is a small, identifiable subspace — which means it can potentially be monitored, steered, or verified independently.
* **Internal state is now partially observable.** Before J-space, we could only see inputs and outputs. Now we have a window into formation of intent. For safety-critical agents, this opens the door to pre-output intervention.
* **The model knows more than it says.** J-space consistently contained concepts the model was aware of but didn't verbalize. This confirms a long-held suspicion: output-only monitoring misses real model state.

### Current limitations

Anthropic is explicit about boundaries:

* J-space operates within a single forward pass, not recursive temporal loops like human consciousness
* It currently works only with words — no images, sounds, or actions
* Working memory is unlimited and non-decaying (unlike human working memory)
* J-lens can only capture single-token concepts, not complex multi-step plans
* This is access consciousness (information availability), not phenomenal consciousness (subjective experience)

The key open question Anthropic raised: we can now see what's in the workspace, but we don't yet know what mechanism decides which thoughts enter it.

## When to invest in a stronger harness

Start simple. Add structure when the work becomes repeated, risky, or expensive.

A stronger harness is worth it when:

* agents edit code or production data
* multiple tools need shared state
* users resume long-running sessions
* approvals and audit logs matter
* the same workflow runs across teams
* evaluation needs to compare behavior across model versions

If the task is one-off and low-risk, a prompt plus a few tools may be enough. If the task is long-running and stateful, invest in harness design early.

## References

* [Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering](https://arxiv.org/abs/2604.08224)
* [Micropaper: Externalization in LLM Agents](https://unbug.github.io/one-minute-read-paper-externalization-in-llm-agents/)
* [Anthropic Engineering: An update on recent Claude Code quality reports](https://www.anthropic.com/engineering/april-23-postmortem)
* [Anthropic Research: A global workspace in language models (J-space)](https://www.anthropic.com/research/global-workspace) — less than 10% of neural activity drives multi-step reasoning; J-lens technique makes internal reasoning visible
* [AIGCLINK: J-space 中文解读](https://x.com/aigclink/status/2074317616894955582) — 删掉不到 10% 的神经活动，多步推理跌到接近零
* [Matt Pocock Skills: Teach skill](https://github.com/mattpocock/skills/tree/main/skills/productivity/teach)
* [PsiACE: Agent 不只是执行流程的容器](https://x.com/repsiace/status/2072039687364161965) — 关于 Agent 应作为人理解、判断和协作的环境，而不仅是自主执行容器的设计洞察。
* [Claude Code is steganographically marking requests](https://thereallo.dev/blog/claude-code-prompt-steganography) — Claude Code 通过不可见 Unicode 隐写标记 API 请求的案例分析，提醒开发者审查有文件系统和 shell 权限的工具。