Speculative Tool Execution for Agents
How agents predict the next tool call, overlap it with model generation, and preserve authoritative execution and side-effect boundaries.
Speculative tool execution predicts and starts the next call early, but isolates its result until the model confirms the same invocation.
Waiting on the serialized path
A large language model (LLM) agent usually alternates between model generation and tool execution. The model emits a call, the executor returns a result, and the model uses that result to decide its next action. This semantic order must remain because a later step depends on the output of an earlier one.
The cost appears when a system maps semantic order directly to physical serialization. The executor sits idle while the model generates, and the session waits while the tool runs. Pattern-Aware Speculative Tool Execution (PASTE) predicts the next call and overlaps it with generation. It isolates the result until the model confirms the same call (PASTE, verified 2026-09).
Speculation overlaps these waiting periods. If the runtime can predict the next call from the observed execution history, it starts the tool while the model is still generating its response. The agent keeps its semantic action order while the runtime moves physical execution earlier.
Parallel width and speculation
Parallel tool calling and speculative tool execution reduce different dimensions. Both run work concurrently, but they differ in when calls become authoritative and which waiting period they reduce.
| Dimension | Scaling parallel width | Speculative tool execution |
|---|---|---|
| Start condition | The model confirms several calls in one turn | The runtime predicts a future call |
| Concurrent work | Independent calls from the same turn | Model generation and the predicted next call |
| Main effect | Fewer reasoning turns | Less exposed tool wait on the serialized path |
| Cost of a bad choice | Unnecessary authoritative calls | Resources spent on discarded predictions |
Wide and Deep (W&D) studies width by confirming several search calls in one reasoning step and collecting their results together. It explores several sources within one turn so the model can compare them and use fewer reasoning turns (W&D, verified 2026-09).
Speculation does not depend on increasing the number of authoritative calls in a turn. It moves execution earlier in recurring sequences such as reading a document after search. It can also coexist with width scaling that broadens exploration through independent queries.
An example from search to reading
The following execution example explains the mechanism rather than reproducing the paper's implementation. Assume the runtime predicts that the agent will read the first search result while the model generates its next decision.
The tool name read web page is not enough to start. The runtime also needs the web address from the current result and permission to access it. Only a prediction with complete arguments becomes an executable call.
Each service can choose its own prediction signal. It may use repeated task traces, a defined workflow, or rules for the current stage. The important constraint is to complete the call with values already observed instead of inventing future values.
A guess that the agent will probably read next should not trigger execution. Without an address, the runtime may only prepare a connection or inspect a cache without changing state. Separating executable predictions from preparation hints reduces calls with incorrect arguments.
Authoritative execution and result isolation
The call actually emitted by the model is the authoritative invocation. In this example, the runtime compares the tool name and arguments when the model requests a web page. A matching tool name with a different address is a miss.
If the predicted read has finished and matches, the runtime reuses its stored result. If it is still running, the session waits only for the remaining time. If the model selects the second search result, the runtime discards the first job and reads the authoritative page normally.
The predicted result stays in separate storage until the match decision. The runtime discards it if the same call does not arrive before expiry and stops it when resources become scarce. Only a matching call can place the result in authoritative session state.
This example also needs a validity check for the retrieved content. Content or access permissions can change at the same address, so matching arguments alone cannot justify reuse. A task requiring a fresh lookup should discard the prediction and read again when the authoritative call arrives.
Result isolation prevents the model from reading unconfirmed output. A bad search prediction cannot enter the next reasoning step, so session history follows the authoritative path. This isolation does not undo a write that has already reached an external system.
The side-effect boundary
Speculation does not grant permission to send email or modify data. An external write, payment, message, or file overwrite can change state when the call runs. Discarding the result object or hiding it from the session cannot recover an email already sent or a record already changed.
| Tool type | Recommended handling |
|---|---|
| Search, lookup, pure computation | Speculate after checking arguments and data permissions |
| Cache warm-up, dependency loading | Allow preparation from a partial prediction |
| File changes, external system writes | Transform into an isolated temporary run or a non-mutating operation |
| Sending, payment, deletion | Exclude from speculation and wait for normal confirmation and approval |
Result isolation and side-effect isolation solve different problems. Hiding a predicted result from the model does not stop a tool from changing an external system. The operating policy must therefore restrict which tools can run speculatively.
A production policy should use a narrow allowlist for each tool. Even a read can consume money, rate limits, or access to private data, so the speculation budget must cover cost and authorization. External writes should run once, after the authoritative call passes the existing permission checks, including user approval where required.
Resource scheduling and fit
A correct prediction can still hurt if it delays authoritative work. An operating scheduler should consider hit likelihood, waiting time that could be hidden, and resource demand. Predicted jobs should use bounded spare capacity and yield whenever authoritative calls need it.
Early tool completion can also send many sessions back to the model server at once. If requests arrive in a burst, time saved on tools becomes extra model waiting. The runtime should limit the return rate and admit sessions according to current model load.
The PASTE v3 abstract reports a 43.5 percent reduction in average task completion time (verified September 2026). It evaluates deep research, coding, and scientific-agent workloads. The result comes from the paper's models, workloads, and serving setup rather than a guarantee for every service.
The best fit combines recurring patterns, long tool latency, and arguments derived from observed state. Test loops and opening a document after search make successful work easy to reuse. Open-ended exploration, short tools, expensive external services, and tasks dominated by side effects can waste more than speculation saves.
Summary
Parallel width executes independent calls that the model confirmed in one turn, while speculation overlaps an unconfirmed future call with model generation. Useful speculation predicts a concrete tool name and arguments, then commits the result only when the authoritative invocation matches. Result isolation cannot reverse external writes, so sending, payment, and deletion stay outside speculation and keep their existing permission checks. The runtime should spend spare capacity only on recurring, tool-heavy patterns and pace sessions returning to the model server so the latency reduction survives.