what a harness actually does

note: i've taken the help of claude for the images in this post.

i've been building a coding agent harness since september (loopcraft, written in go), and the question i get most is what a harness actually is. it usually comes right after "isn't that just a while loop?"

it is a while loop. the short definition, which theo put well in what is a "harness"?, is a handful of parts: tools that can actually do something, a syntax the model uses to ask for them, code that spots that syntax and runs the right tool, a way to hand the result back, and a list of the tools so the model knows what exists. written out, that comes to about ten lines of code, and i'll show them below.

what the short definition leaves out is everything that goes wrong. the ten lines took me an afternoon. the weeks since went into the edges: the model asking for something malformed, you changing your mind halfway through a run, three tool calls arriving at once, the context window filling up. so instead of another definition, i want to follow one prompt through a real harness, stop at every place where the simple version breaks, and then look at how you extend one past what ships in the box.

two layers in one box

a model on its own can only produce text. it can describe a perfect refactor in great detail and still never touch a file. the harness is what turns "edit line 40" into an edited line 40.

the line most people blur is the one between the harness and the app around it. claude code is two things shipped in one box: an interface you look at, and a harness underneath that does the work. because they ship together, it's easy to think of them as one thing. you only see the seam when you try to pull them apart.

loopcraft is a terminal app, but run loopcraft acp and it speaks the agent client protocol instead. now zed, a jetbrains ide or neovim is the interface, and the same harness runs underneath. the loop doesn't change at all. the editor receives a stream of events (some text arrived, a tool call started, here's a diff to review) and draws them however it wants. the model has no way to tell which one it's running in.

the ten lines

here's the whole idea, with the model api and the tools waved away:

messages = [user("rename get_user to fetch_user")]

while true:
    reply = model(system_prompt, tool_list, messages)
    messages.append(reply)
    if not reply.tool_calls:
        break                                  # nothing asked for: done
    for call in reply.tool_calls:
        output = tools[call.name](call.args)   # the harness does the work
        messages.append(tool_result(call.id, output))

print(reply.text)

every part of the short definition is in there. tool_list is the list with descriptions. reply.tool_calls is the syntax, already parsed. the for loop runs the right tool, and tool_result is the way back. the one thing that has changed since the early days is that you no longer invent the syntax yourself. models are now trained to emit tool calls in a structured form, and the api hands them to you as data. a claude reply that wants to search looks roughly like this:

{
  "stop_reason": "tool_use",
  "content": [
    { "type": "text", "text": "let me find every caller first." },
    { "type": "tool_use", "id": "toolu_01", "name": "grep",
      "input": { "pattern": "get_user" } }
  ]
}

"stop_reason": "tool_use" is the model putting its pen down. it has said what it wants, and now it waits for the harness. if the reply had ended with "end_turn" instead, the loop would stop and you'd get the text.

the part that took me longest to really absorb is what the model remembers between those requests, which is nothing. there's no session on the other end. every trip around the loop sends the entire conversation again, from the system prompt to the latest tool result, and the model reads all of it fresh. the transcript is the memory, and the harness is the only one holding it. here's our rename, request by request:

three requests, and the last one carries seven messages to answer one sentence you typed. a real session runs for dozens of turns, and every one of them pays to resend everything before it. most of what follows is the harness coping with that, or with the model being imperfect, or both.

make being wrong cheap

the ten lines assume the model asks for things correctly. it doesn't, always. models drop required fields, send "5" where the schema says a number, invent parameters that don't exist, and nest an object one level too deep. weaker models do it more often, but every model does it sometimes.

the naive loop hands those arguments straight to the tool, the tool crashes, and now every tool needs its own error handling, each one complaining in its own words. loopcraft checks every call against the json schema the model was shown, before the tool runs. a call that doesn't fit never reaches the tool. it gets an answer instead, naming what was wrong and echoing back what was sent, and the model reads that on its next turn and almost always fixes the call itself. pi, the minimal harness loopcraft started from, makes the same check but quietly converts "5" into 5 for you. i went the other way and reject it. if the harness rewrites the arguments behind the model's back, the transcript says one thing, the tool did another, and nothing in the conversation tells the model it got the type wrong.

tools should fail the same way, out loud and in words the model can act on. the edit tool is a good example. it replaces an exact snippet of text, and the snippet has to appear exactly once in the file. that rule sounds fussy until you see what happens without it:

if the tool had guessed and replaced the first match, you'd get a file that's a third renamed and a model that believes it's finished. the error costs one extra turn. the guess costs you an afternoon of wondering why the tests fail on line 40.

if i had to keep one design decision from the whole harness, it would be this one: when something goes wrong during a run, it becomes a message the model reads, never an exception that kills the loop. loopcraft's run only returns a go error when it can't start at all, because the config is broken or the transcript can't be continued. a provider timing out, a tool failing, you pressing escape: each of those gets written into the transcript as something that happened, and the model gets to react to it.

every question gets an answer

there's a rule the apis enforce that the ten lines quietly depend on. every tool call the model makes has to be answered by a tool result with the same id before the next request goes out. send a transcript where one call has no result, and anthropic and openai both reject the whole request. so a harness can never leave a question hanging, and the interesting part is all the ways a call can end up unanswered.

you press escape while a test suite is running. a reply gets cut off by the output limit halfway through writing a tool call, so its arguments are half a json object. you type something while the agent is busy. in each case the naive loop either runs a call it shouldn't, or drops one and corrupts the transcript.

loopcraft's rule is that every call gets a result, including calls that never ran. the calls in a cut-off reply are answered with an error that says they weren't executed, since their arguments might be incomplete, and the model gets another turn to issue them properly. an abort gives running tools five seconds to wrap up, answers everything that didn't start, and ends with a transcript you can pick up again tomorrow.

my favorite case is steering, which is what happens when you type while the agent works. before each tool call, the loop checks whether you've said anything. if you have, the calls that haven't started are skipped, and each one gets this as its result:

a call that's already running is never interrupted by steering, only by an abort. the harness can't know whether stopping a half-finished edit leaves your file worse off than letting it finish, so it lets it finish, and the model decides what to redo once it has read what you said.

two reads can run at once

models often ask for several things in one reply: read three files, grep for two names. run one after another, those calls cost the sum of their latencies. run side by side, they cost only the slowest. the obvious move is to run everything in parallel, and it's wrong. an edit followed by a read of the same file has to see the edit, and two shell commands can depend on each other in ways the harness has no way to detect.

so loopcraft walks the calls in the order the model wrote them, and only overlaps the ones that change nothing. a tool can declare itself read-only, and read-only calls run together, up to eight at a time. anything else waits for everything before it and runs alone, and that includes every tool that didn't declare anything, like an mcp server's. the result is exactly what you'd get by running them in order, only faster wherever that's safe.

the results go back in the order the calls were made, even when the second one finishes first. that keeps the transcript readable for the model, which sees its answers in the order it asked, and it means a saved session comes out as the same bytes every time it's replayed.

same bytes matter more than they sound, because of prompt caching. providers cache the beginning of each request, and if the start of what you send matches what you sent a few minutes ago, that part is billed at a discount. anthropic charges a cache read at a tenth of the normal input price. every trip around the loop resends the whole conversation, so in a long session almost all of each request is a prefix the provider has already seen, and the cache is the difference between an affordable session and a painful one. the catch is that the bytes have to be identical. for that reason, loopcraft reads each tool's definition once at the start of a run and sends exactly that text with every request in it. a tool whose description included something that changes, like the current time, would quietly break the cache on every single turn.

the window fills up

back to the fact that every request carries the whole conversation. a coding session fills a context window quickly, because every file read and every command's output stays in the transcript for the rest of the session. the harness has two defenses, a small one and a big one.

the small one is never letting a single tool result get huge. loopcraft caps output at 2,000 lines or 50 kib, whichever comes first, both numbers taken from pi. you need both caps. a line cap alone lets a minified javascript bundle through whole, since it's one enormous line. a byte cap alone lets through ten thousand one-character lines, each costing a token or more. when output gets cut, the model is told exactly where and how to continue:

[showing lines 1-2000 of 9000. use offset=2001 to continue.]

the big one is compaction. before every request, loopcraft checks whether the conversation would leave less than 16,384 tokens of the window free, which is the room it keeps for the reply and the next tool output. if it would, the older part of the conversation goes to the model one more time with instructions to write a structured summary: the goal, your constraints, what's done, what's next, and the exact file names and error messages needed to carry on. from then on, each request carries that summary followed by roughly the most recent 20,000 tokens of conversation, word for word.

two details make this work. the cut never lands on a tool result, because a result separated from the call it answers is exactly the broken transcript from earlier. and nothing gets deleted. the summary is appended to the session file as a new entry, so the full history stays on disk, you can scroll back through it, and you can branch from a point before the compaction and get the original messages back. the model sees the short version, the session keeps the long one, and only the harness knows about both.

the check runs before every request, including the ones in the middle of a run. that matters more than it sounds. one prompt like "figure out why the build is slow" can read enough files on its own to cross the line before you've typed a second message.

what the model never sees

mcp servers are the usual way to give a model more tools without building them into the harness. a built-in tool is code that runs where the harness runs, while an mcp server keeps its code somewhere else and the model sends it requests.

the catch is that every tool the model might call has to be described to it, and descriptions cost tokens. connect five mcp servers the naive way and every tool from every server rides along in every request, whether this task needs a github issue or not. loopcraft gives the model a single mcp tool instead. its description names each server with one line about what it's for, and the model looks up a server's tools only when it needs one. skills work the same way: the model sees each skill's name and a one-line description, and reads the full file only when a task matches it.

people often describe the harness as the invisible part, the plumbing nobody using the app ever sees. that's true of its code, and i think it's a problem when it's true of its choices. the harness decides what goes into the model's context: which instruction files, which tools, what got summarized, what got cut off at line 2,000. if those choices are invisible, then when the agent does something strange you can't tell whether the model reasoned badly or simply never saw the file that would have stopped it. loopcraft keeps pi's rule here. every file loaded into context is listed at startup with an estimate of what it costs, and /context breaks the window down by system prompt, tools, files and messages at any point. you don't need to see the plumbing, but you should be able to see what the model was given.

where the harness says no

the last stop before a tool runs is a gate. loopcraft has two hooks around every call. the first runs before the tool and can allow the call, block it, change its arguments or answer in its place. the second sees the result before the model does and can replace it. permission prompts, secret filters and "never run rm -rf outside the project" all live here. both hooks fail closed: if a hook errors, the call doesn't run, or its result is withheld. a permission check that waves things through whenever it crashes has its default backwards.

runs also have budgets, with caps on responses, tokens, dollars and wall-clock time. when a reply pushes a run past one, the harness still runs that reply's tool calls and records their results before stopping, for the same reason as everywhere else in this post. the transcript has to stay valid, so tomorrow you can type "keep going" and it will.

six places to plug in

everything so far ships in the box. the more interesting question, once you build on a harness instead of just using one, is what you can change without forking it. in loopcraft, and in pi before it, the answer is that the loop exposes a small number of seams, and every advanced feature i've seen is some combination of them.

a new tool (1) is the obvious one. the two hooks around each call (2 and 3) are the gate from the last section. the context hook (4) is the most powerful and the least appreciated. it gets a copy of the transcript right before every request and returns what the model should see instead, while the session keeps the original. compaction is built on it, and so is anything that wants to attach the current git diff, redact a secret from old output, or drop tool results that have gone stale. message injection (5) lets something other than you steer a run or queue a follow-up. events (6) are the read-only stream of every token, call and result, which is how an interface, a logger or a cost tracker watches a run without being able to change it.

extensions are processes

the harder design question is how outside code reaches those seams. i looked at four options for loopcraft: go's plugin package, an embedded go interpreter (yaegi), webassembly, and plain child processes talking json-rpc over stdin and stdout. i picked processes, and the reasoning was mostly about failure. a plugin that crashes takes the harness down with it, and one that's stuck in a loop can't be stopped. a separate process can be timed out and killed, its memory is its own, and it can be written in any language that can print a line of json.

the cost i worried about was latency, since every hook becomes a round trip between two processes. measured on an m1 pro, a round trip carrying a 200-byte tool call took 15 µs at the median and 290 µs at p99, and starting a go extension and getting its first reply took 3.6 ms. a tool call already waits seconds for the model, so a few microseconds per hook is noise. here's a permission gate and a secret filter on the wire, trimmed from the protocol's own test transcript:

the failure rules are where most of the design went. a hook that doesn't answer in time (ten seconds by default, and an extension can ask for up to five minutes) counts as a failure, and every failure closes the gate. if a permission hook crashes, the call it was guarding doesn't run. if a redaction hook crashes, the model gets an error in place of the result, so a filter that breaks can't leak the thing it was meant to hide. an extension that stops reading its input is treated as hung and restarted, so a slow extension never holds up a token. and extensions that live inside a repository only start after you've said you trust that project, because cloning someone's repo and typing loopcraft shouldn't be enough to run their code with your permissions.

where i'm taking it next

the seams are built. most of what i want to put on them isn't yet, so read this part as a plan with reasons rather than a feature list.

first, subagents: a task tool that starts a second copy of the loop with its own context, tools and budget, and hands back only a summary. the appeal is arithmetic. a search that reads forty files costs the parent one tool result instead of forty. the risk is that the child turns into a black box, so the plan is to save each child's transcript as a linked branch of the session, where you can open it and see exactly what it read. once children exist, they can also run on different models, a cheap fast one for searching and a strong one for the edit.

second, a real sandbox, which changes how i think about the permission hooks above. a hook that reads the command string is guessing. rm -rf build can also be spelled as a python one-liner, a find with -delete, or a script the model wrote two turns ago, and no pattern catches all of them. the operating system doesn't have to guess. landlock and seccomp on linux, and seatbelt on macos, can stop a process from writing outside the project or opening a network connection, however the command was spelled. the hooks then go back to being about policy and asking you things (confirm before a push, say), and the kernel does the enforcing. running extensions as separate processes was partly so the same sandbox can wrap them later.

third, feedback the model didn't ask for. after each edit, run the language server and attach its diagnostics to the edit's result, so a type error shows up in the same turn that caused it instead of three turns later when the tests run. the after-call hook already makes that possible. the work is in making it fast and reporting only what the edit changed. per-turn checkpoints belong in the same family: a snapshot of the files before each turn and an /undo that works without git, so letting the agent try something risky stops being a decision you have to think hard about.

last, running with nobody at the keyboard. the same loop already speaks a json rpc mode on stdin and stdout as well as acp, so a ci job or a github action that reviews a pr is one more interface on top. that's also how you'd measure any of this. every change in this post is a guess about what helps the model until it runs against a benchmark, and a headless harness is what lets you run the same tasks before and after a change and see whether the guess was right.

so what's an agent?

the usual one-line answer is that an agent is a model plus a harness. having built one, i'd add a footnote. the ten lines are the agent. they haven't changed since the first demos, and they're the same ten lines in claude code, codex, pi and loopcraft. the harness is everything else, and almost everything else turns out to be a list of the ways those ten lines break, plus the seams that let you fix the next one yourself.

models keep getting better, and parts of that ring will shrink. a model that never sends a string where a number belongs makes one check redundant, and a model that never loses the plot past 100,000 tokens makes compaction rarer. i don't think the ring ever disappears, though, because the entry that keeps growing sits on the other side of the screen: the person at the keyboard, the one who reads the first tool call, realizes they asked for the wrong thing, and starts typing while the loop is still running.