Agent Harnesses and Loop Engineering

Evaluate agent reliability by checking both the model and the runtime loop that manages its calls.

Three senses of harness

Harness is not a fixed term. Its meaning shifts with what it is wrapping. Recent literature uses the same word for at least three different things, and without separating them first, every paper becomes confusing.

A software test harness runs code and checks its results. A model evaluation harness manages benchmark runs. In this post, an agent harness is the runtime layer that manages model calls and tool execution.

This post is about the third one. Everything that happens after the first prompt belongs here. It is the code layer that manages the exchange between the model and its tools, filesystem and environment.

The four jobs of an agent harness

An agent harness handles four things: tool execution, context management, state persistence, and safety and verification. This is the design frame used in this post. Measure the relative effects of the model and the harness for each task.

flowchart LR
  M["Model<br/>decides the next action"] --> D["Tool dispatch<br/>request to actual execution"]
  D --> E["Environment execution<br/>filesystem, search, APIs"]
  E --> C["Context management<br/>compaction and selection"]
  C --> S["State persistence<br/>session and memory"]
  S --> G["Safety guards<br/>invariants and stop conditions"]
  G --> M

The diagram shows the order in which one turn runs. The model picks an action, and the harness executes the tool. The result is compacted before it returns to the context. Progress is then written outside the session, safety invariants and stop conditions are checked, and control moves to the next turn. The model is called many times inside this loop, and the harness is what connects one call to the next.

Of the four jobs, context management ties directly to the token budget. Compaction keeps only what is needed from retrieval and tool output. The two basic axes are extraction, which preserves the original text, and summarization, which reconstructs the meaning. Tasks where provenance matters default to extraction; tasks where length is the pressing problem default to summarization.

Where cost leaks when the loop gets long

The same harness costs a completely different amount on a one-turn task than on a fifty-turn one. A single-turn chatbot pays the retrieval and tool overhead exactly once, so it never becomes a problem. A long-running agent pays it at every loop step, and the cost compounds.

Sending accumulated context on every call adds input-token cost. Actual cost depends on input and output prices, caching, tool fees, and iteration count, so measure these components separately.

All three failure modes arise when the loop cannot stop itself. Costs rise when the model alone decides when to end retrieval, how often to call tools, and which output to discard. That is why the stopping machinery has to live in the harness.

Stopping a tool-calling loop

The basic design principle for tool-calling RAG is to state a maximum call count and an explicit loop termination condition. When the model decides on its own when to call retrieval or a tool, failure shows up in both directions: not calling when it should, and calling excessively on every query. Both grow when the termination condition is weak.

# pseudo code: termination control for a tool-calling loop (illustrative)
MAX_STEPS = 12
step = 0
while step < MAX_STEPS:                    # upper bound on call count
    action = model.decide(context)         # decide the next action
    if action.is_final:                    # the model declares it is done
        break
    result = dispatch(action)              # dispatch and run the tool
    context = compact(context + result)    # compact the context
    if evaluator.met(goal, context):       # a separate evaluator judges the stop condition
        break
    step += 1

MAX_STEPS here is the last safety net against an infinite loop. evaluator.met is the slot where a small, fast, fresh model, rather than the model that did the work, judges the termination condition. This is the maker-checker principle: separate the generator from the verifier. Large transfers use a similar separation between the person entering the transfer and its approver. The evaluator can also make mistakes, so combine it with execution tests and explicit completion conditions.

Tool calling introduces a new failure axis, so it gets its own observability metrics. The common failures are three: choosing the wrong tool, malformed input, and misreading the result. In production, keep watching tool call count, failure rate and average loop length so you catch the moment any of the three spikes.

Tools with side effects need a permission guard independent of termination control. Tools that write to the outside world, such as sending email or writing files, run only after user confirmation. Expose only authorized tools to the model, and record every call in an audit log. The safety filter has to treat the tool-calling decision itself as an object of inspection, not just the answer text.

Wiring retrieval in as a tool

There are two ways to put Retrieval-Augmented Generation (RAG) into a harness. Either retrieval is a fixed stage in the pipeline, or it is exposed as a tool the model calls when it decides to. The second option moves control over RAG from the system to the model.

Exposed as a tool, it looks to the model like one more function with a signature such as search(query, k) → documents. Here the tool schema drives quality. If the only argument is a query string, the model must handle all query refinement. Adding filter arguments such as date, domain and permissions lets the model use metadata filters directly. Too many arguments, though, and the model gets confused or tries to fill in unnecessary filters on every call.

This post uses harness to mean the code that manages tool execution and the storage, retrieval, and presentation of information. To evaluate a change, hold the model, retriever, task, and budget fixed, then vary one stop condition or context policy. This is an evaluation proposal, not a reported performance improvement.

Loop engineering, one layer up

In this post, loop engineering means the operational procedure that discovers work, executes it, verifies the result, and schedules the next run. The harness manages one turn. This procedure manages state and schedules across turns.

A turn consists of five moves: discovery, handoff, verification, persistence and scheduling. Verification is both the easiest one to cut corners on and the one you must not skip. A loop without a real check does not verify the agent's work.

Context and memory have to be distinguished here. Context is what the model sees in this round, and it disappears when the window is cleared. Memory is state that persists across rounds and across days, written outside the conversation in something like a markdown file or a board. Keeping a loop's progress record only in the context window means one refresh erases all of it.

Summary

An agent harness is the runtime loop around model calls. It handles four jobs: tool execution, context management, state persistence, and safety verification. As tasks grow longer, measure model errors and harness errors separately. A longer loop compounds input token cost while leaking through retrieval thrash, tool storms and context bloat. The baseline design therefore needs a maximum call count, termination conditions, maker-checker verification by a separate evaluator, and observable tool call metrics.

Exposing retrieval as a tool hands control to the model, so schema design drives quality. Compare harness changes on the same tasks and budget. Loop engineering, one layer above, puts that turn on a schedule and makes it repeat. It only works when context and memory are kept separate and the verification move is not skipped.