By Vamsi Krishna
Engineering · · Discuss · updated
60 Days of Production AI Systems · chapter 7 of 10Agent Control and Supervision Day 40 · part 4 of 6

New here? Start this series at Day 37: Where human review actually belongs in AI workflows

  1. Day 37: Where human review actually belongs in AI workflows
  2. Day 38: Why guardrails should enable safe action
  3. Day 39: Why production agents need recovery design
  4. Day 40: Why autonomy needs budgets
  5. Day 41: Why one giant agent is rarely the cleanest design
  6. Day 42: Why supervisor agents help and where they bottleneck

Every team running agents eventually has the invoice conversation. The only variable is whether it happens beforehand, calmly, as a design decision, or afterwards, on a Monday, following a weekend where one stuck agent discovered it had no reason to stop.

Autonomy without budgets is an open tab.

BudgetExample limitWhat it prevents
Time90 secondsHidden latency users silently absorb
Tool calls8 per taskOperational blast radius
Cost$0.25 per taskUnpredictable unit economics
Retries2 attemptsLoops that look like productive work
Riskhuman review above thresholdUnaccountable high-impact actions

Define the behaviour at the threshold

Setting a limit is the easy half. The half that gets skipped: what happens when the agent reaches it?

Hitting a budget should produce an outcome, not an exception. At 80% consumed, the agent should be wrapping up: summarising progress, returning partial results, or handing to a human with state attached. An agent that hits a hard cap mid-action and dies leaves you exactly the "failed halfway" mess from yesterday, except now you caused it deliberately.

That distinction is worth designing explicitly: a graceful wind-down at the soft limit, a hard stop at the ceiling.

Scarcity improves planning

The benefit that surprises people is the change in behaviour.

An agent that knows it has five tool calls remaining plans differently (and usually better) than one operating as though resources are infinite. It prioritises. It stops exploring branches with low expected value. You can watch trace quality change when the remaining budget is made visible to the agent itself rather than merely enforced around it.

Unbounded agents are less deliberate, because nothing forces a choice about what matters.

The retry trap

Worth isolating, because it is the most common way budgets get consumed invisibly: retries look like work.

Each individual retry is defensible. The trace shows an agent diligently attempting to complete its task. What it does not show, unless you are counting, is that attempts three through eleven were the same failing approach with cosmetic variation.

Cap retries separately from total actions. They are a different failure mode and they need their own ceiling.

Closing thought

Budgets are how autonomy becomes governable, and how it becomes something finance will approve at scale rather than tolerate as an experiment.