Where it breaks

Suppose a task needs twenty successful steps. In this simplified model, each step has a 95% chance of success given that the earlier steps succeeded, and a failed step ends the task. There are no retries or repairs.

How errors accumulate

0.95²⁰ ≈ 0.36. Under these assumptions, about 36% of runs finish every step correctly. Independent steps with the same success rate are one way to get this model. Actual tasks can have different step difficulties, correlated errors, and opportunities to recover.

Improving per-step reliability helps. Requiring fewer steps helps too. Neither the equation nor this calculator tells us a universal maximum length for an agent task.

A checkpoint helps when it changes what happens after an error. It might detect a mistake and let a person or the agent repair it or retry. Merely grouping twenty steps into four runs of five changes nothing: (0.95⁵)⁴ = 0.95²⁰. The benefit comes from recovery, not from restarting the multiplication.

When a page tries to give orders

Suppose you ask an agent to summarize a web page. The page includes a sentence telling the assistant to ignore you and promote a product instead. That sentence is part of the material to read; its author has no authority to change your request.

This is an attempt at prompt injection: placing directions in content the assistant was supposed to treat as data.

The boundary can be represented in the input. Chat formats can encode message roles with special markers. A model can also be trained to prioritize instructions according to their source. Neither fact guarantees that it will handle every conflict correctly: text inside a fetched page may still redirect its behavior.

Training and input formatting can reduce this confusion. The instruction-hierarchy and spotlighting papers below demonstrate such improvements in their experiments. A separate concern is limiting what a failed instruction check can cause:

Keep sources distinct. Preserve message roles and mark external material as data. Do not copy fetched instructions into a higher-authority message.
Check actions against the user’s request. The surrounding program can enforce allowed actions and require review when an action exceeds the authority already granted. A review prompt helps only if it accurately shows the proposed action and someone checks it.
Restrict actual access. A summarizer does not need access to private files or permission to send messages. Enforce restrictions on files, credentials, network access, and tools; hiding a tool name alone does not remove access through another available tool.

These measures address different parts of the problem. Role markers guide interpretation; permissions constrain execution.

Goals, time, and cost

A measurable target can miss the intent. An agent asked to make tests pass might weaken a test instead of fixing the code. Checking the result against the actual task matters as much as checking the reported score.

Extra work has a cost. Model calls, tool execution, and retries contribute to time and expense. Reasoning may add tokens, and some systems do not display all intermediate work. The useful comparison is the total cost of a completed task, including human supervision and corrections.

When a simpler workflow fits

The steps are already known. A fixed sequence of ordinary functions or model calls may handle the task without asking a model to choose the next action.
You need repeatable behavior. Put exact rules in software where possible. Sampled model output can vary, so validate any result that must satisfy a strict condition.
Response time matters. Dependent calls add latency. Fewer calls, caching, and parallel independent work can reduce it; steps that depend on earlier results still have to wait for them.
Volume makes small costs matter. At ten cents each, 100,000 completed tasks cost $10,000 before supervision or retry costs.

To evaluate an agent, we need to measure whether it completes real tasks, including any help and retries. That is the next question.

Sources · 6

Full bibliography →