Why this matters
The gateway decision looks free in a diagram. In production it has a bill, and the bill decides whether the pattern survives the first budget review.
The lines
Compute
One small service, horizontally scaled. For a dozen tool servers and a few hundred internal users, this is the cheapest line by far.
Audit storage
Every tool invocation is recorded with inputs, outputs, identity and decision. At a few kilobytes per call, high-volume tools fill terabytes within a year. Retention policy is a cost decision.
Availability
The gateway is in the path of every agent action. It needs the same availability as the systems behind it, which means at least two regions and a tested failover.
People
Someone reviews tool descriptions, rotates credentials, answers "why was this call denied" and carries the pager. This is the line that is always missing from the first estimate.
What I would not cut
Audit, because it is the reason the gateway exists. Cut retention, sample inputs for high-volume read-only tools, but never cut the decision record.
Key takeaways
- Compute is the smallest line. Audit storage and on-call are the largest.
- Tool description reviews are a recurring cost, not a one-off.
- Budget for the gateway as production infrastructure from day one.
More on these topics
Checklist · · 12 checks
RAG production-readiness checklist
Twelve checks to pass before a retrieval system answers a real user. Tick them locally; progress stays in your browser.
Failure story · · 1 min read
The retrieval cache that served stale policies
A well-meaning cache in front of retrieval kept answering from last quarter's HR policy for eleven days.
Explainer · · 2 min read
Nothing broke, and the system is still getting worse
Drift is quiet by construction. SLOs and a standing review are what make it audible.
Discussion