Scaling Reliable AI Agents

Kadhir Manickam

(4 minutes)

<section data-section-id='3906425a-7937-4822-aba9-0ab0678941c9'>Running an agent once is easy. Running it thousands of times a day is a different sport. LLMs are non-deterministic, they do weird stuff a lot. That's easy to hide in a demo and brutal in production, where you're chasing 9s of reliability and every weird edge eventually shows up. The tail of failures is so wide that two things get hard fast: reproducing an issue, and hillclimbing your way past it. There are too many one-off failures to replay by hand, so we run in high volume and watch the probabilities instead. When a failure rate stops moving, we know the issue is ironed out. At ProductNow, a few principles get us there. They all point the same way: ask the model to do as little as possible, and let deterministic engineering carry the rest. </section> <section data-section-id='bb25fde0-22db-44dc-bd24-ae48b64e2813'>1) Shrink what the agents do The first principle: put determinism everywhere the model does not need to think. In practice, every part of the system that touches the outside world is a deterministic script. When we fetch information from an external system, a script does it, with a strict schema on its inputs and outputs. When we write back downstream, same thing. That leaves the agent in the middle doing the minimum: it assembles context inside ProductNow and acts on it. A one-off agent can afford to be loose. Ours run thousands of times a day at high reliability, and we've built our systems with that constraint in mind. But that raises a question: what does the agent in the middle actually do? </section> <section data-section-id='280a9ba7-6f4d-4784-ad07-e0ac49aa35a7'>2) Let the schema drive the loop The answer is a retry loop. Every script in our harness takes a particular schema on its inputs and outputs, and that contract sits at the center of the design. On each call, the agent's job stays small: assemble context and produce a payload that matches the schema. When the payload misses, validation rejects it and hands the error back. The agent adjusts and retries until the output hits the schema. Only then does the script run. <mermaid data-height="260"> flowchart LR A[Agent assembles context] --> B[Agent produces payload] B --> C{Schema validation} C -->|Miss: error returned| B C -->|Pass| D[Deterministic script runs] </mermaid>This is what makes our agent loops so effective. Every pass carries a crisp pass-or-fail signal, retries are cheap and converge fast, and the retry logic lives in the harness, so the model spends its tokens on judgment instead of plumbing.That clean signal powers hillclimbing too. We watch the first-pass hit rate across thousands of runs, and every change to a prompt or a schema shows up in that number. </section> <section data-section-id='d4db5761-666b-4016-810e-2f63a8f880a7'>3) Make search the interface Even with a tight loop, the agent still has to find the right context. That's the second place where reliability leaks out. Point a model at a repository and tell it to open the right files, and it will wander. It reads the wrong file, follows a stale reference, and burns tokens on paths that go nowhere. It'll only do it a small percentage of time, but it's enough to ruin our 9s. So we made search the interface instead. Rather than telling agents which files to open, we tell them which terms to search for and which queries to run. The queries are fixed, and the results they return are baked in, which pushes the determinism into the search layer, where we control it. That shift does two things for us. First, it asks the model to do less on every call. Reading a whole file and judging what matters is expensive and unpredictable. Running a query that returns a handful of relevant snippets is cheap and behaves the same way every time. Second, it gives us a surface we can actually test. Because every lookup is a query, we built a separate set of retrieval evals around the search layer. We know what a good result looks like for each query, so we can measure whether the right context comes back before the agent ever runs. If the retrieval is right, the agent starts on track by construction. This is the balance we're after. The model keeps the flexibility it's good at, judging and acting on what comes back. The system owns the part that has to be repeatable. Deterministic where it counts, flexible where it helps. </section> <section data-section-id='bbdcafe9-9c1c-4b42-947a-00827b8ac4e6'>What this buys Put all of this together and you get 9s of reliability on genuinely complex workflows. You also cut token spend, cut latency, and keep context windows free, because the model carries as little as possible and the engineering carries the rest. The harness itself is the hard part. Wrapping a deterministic shell this tight around a non-deterministic core is a real engineering problem, and solving it is where the reliability lives. If you are fighting the same long tail of weird failures, we'd love to hear what is working for you. </section>

Published

October 28, 2026

Keep reading

The Company Brain: What It Buys You and What It Takes

Every company runs on knowledge it can't find. What a company brain is, the four ways it pays for itself, the six failure modes that sink homegrown attempts, and how ProductNow answers each one.

The Context Engine

Every company now has thousands of agents trying to get work done, and almost none of them know what the organization actually knows. Organizational context is a big data problem with its own 4Vs. What a context engine is, the five attributes that define one, and why ProductNow built it as a plain document editor.

AI-native data integration without an ontology

Every data integration effort historically required an ontology — slow to build, locking clients out of their own system, never truly self-service. AI changes that equation: continuous ingestion into living documents, agent-managed pruning, and a deep graph — integrations in 24 hours, not months.