The Agentic Operating System —Deterministic Control on Non-Deterministic Foundations

Here is the fact that should anchor every serious conversation about agentic systems, and that most vendor decks quietly skip: a large language model is non-deterministic. Send it the same prompt twice and you can get two different answers. Not wildly different, usually — but different. Different word choices, different structure, occasionally a different decision. The output is sampled from a probability distribution, and a sample is, by definition, not a guarantee.

You cannot run a business directly on a component that behaves that way. An approval workflow that sometimes routes to legal and sometimes does not is not a workflow — it is a liability. The instinct of most engineering leaders, on first contact with this problem, is correct: this thing is not reliable enough to build on.

And yet reliable systems are being built on it. The resolution to that contradiction is the entire craft of agentic engineering, and it is the thing Facet builds. You do not make the model deterministic — you mostly cannot. You wrap the non-deterministic model inside a deterministic system, so that the system is dependable even though the model is not. That scaffolding — typed schemas, validation gates, state machines, idempotent retries, deterministic tools, scoring thresholds, human gates, and evals — is the agentic operating system. The model is the probabilistic core. Everything around it is the deterministic OS that makes outcomes auditable, repeatable, and safe to put in front of a customer.

This piece is about that OS: why the non-determinism is real and not a tuning problem, why "just prompt better" fails at the system level, what the layers of deterministic control actually are, and what it means for you as a buyer.

The Non-Determinism Is Real — and It Is Not a Bug You Can Configure Away

The common belief is that you set temperature = 0, which makes the model "greedy" — always pick the highest-probability token — and you get deterministic output. It is a reasonable belief. It is also wrong, and the reason it is wrong matters.

Temperature zero is a lie, as one widely-shared write-up put it bluntly: it makes only the token-selection rule deterministic; it does not make the whole inference system produce identical results run to run (per Vincent Schmalbach). The deeper explanation comes from Thinking Machines Lab, whose engineers traced the real culprit. It is not the usual scapegoat — GPU concurrency and floating-point race conditions. The actual source is that inference kernels are not batch-invariant: the numerical result of a computation depends on how many other requests are batched alongside yours, and that batch size shifts unpredictably with server load (per Thinking Machines Lab).

The empirical demonstration is the part to internalize. Sampling 1,000 completions from a large model at temperature 0 — the supposedly deterministic setting — they got 80 unique completions, with outputs first diverging at the 103rd token (per Thinking Machines Lab). It is possible to fix this with specially-engineered batch-invariant kernels, and they did — at which point all 1,000 completions became identical. But that is deep infrastructure work most teams will never do, on models they do not control. For practical purposes, when you call a hosted model, you are calling a non-deterministic function.

The point is not that the model is broken. The point is that non-determinism is a property of the substrate, not a misconfiguration. You should stop trying to eliminate it and start engineering around it.

Why "Just Prompt Better" Fails at the System Level

When an agentic pilot misbehaves, the reflexive fix is to rewrite the prompt. Add instructions. Add an all-caps "ALWAYS return valid JSON." Add examples. And it works — the failure rate drops, the demo passes, everyone exhales.

Prompt engineering is real and it matters. But it cannot carry the weight people put on it, for a structural reason: a prompt is a request, and the model answers requests probabilistically. Better prompting shifts the probability distribution toward the behavior you want. It does not pin it. A 98%-reliable instruction is also a 2% failure rate, and at system scale 2% is not a rounding error — it is an incident queue.

This is where the math turns against the naive approach. Agentic workflows are compositional: an agent chains many steps, and the failure probabilities compound. As the team at Inngest lays it out, five steps at 99% reliability each yields roughly 95% end-to-end; ten steps drops you to about 90%; and real workflows involve dozens of operations (per Inngest). You cannot prompt your way out of compounding. Every step you make 99% reliable through better instructions still multiplies against every other step. The arithmetic guarantees that a long enough chain of "almost always right" components is "usually wrong" end to end.

So the prompt is the wrong layer to solve reliability at. The prompt steers the model. Reliability is a property of the system — and you get it by building deterministic structure around the probabilistic core, so that the system can detect, contain, and recover from the model's inevitable misses.

The Layers of the Agentic Operating System

An operating system does not make the underlying hardware perfect. It manages imperfect hardware — flaky disks, contended memory, interrupts that arrive out of order — and presents a dependable abstraction on top. The agentic OS does the same thing for an imperfect, probabilistic model. Here are its layers, in the order we tend to build them.

1. Typed schemas and structured outputs — constrain the shape

The first move is to stop accepting free-form text and start demanding typed, schema-conformant output. Modern model APIs support exactly this: with structured outputs, the model is constrained during decoding so the response is valid JSON matching a supplied schema. OpenAI reports that its structured-outputs models achieve a perfect score on its own JSON-schema-adherence evaluation — full compliance — where prior prompt-only approaches fell short (per Protecto's summary of OpenAI's announcement).

This is the highest-leverage layer because it converts an open-ended generation problem into a typed-data problem. Once the output is guaranteed to be a well-formed object — { "decision": "approve" | "reject", "confidence": number, "reasons": string[] } — every downstream component can rely on its shape, even while remaining skeptical of its content.

2. Validators and scoring gates — check the content

Shape is necessary but not sufficient. A schema guarantees the model returned a number for confidence; it does not guarantee the number is sane. So the next layer is deterministic validation: business-rule checks, range checks, cross-field consistency, and scoring thresholds the output must clear before it proceeds. In our own content engine, every draft must clear a weighted scoring rubric above a fixed numeric threshold before it can advance a stage — a deterministic gate on top of probabilistic generation. The gate does not care how the draft was written; it cares whether the result clears the bar. The model proposes; a deterministic check disposes.

3. State machines — make the flow legible and bounded

An agent left to "figure out the steps" each time is a non-deterministic process wrapped around a non-deterministic model — variance squared. Instead, we encode the workflow as an explicit state machine: defined states, defined transitions, defined terminal conditions. The model's job is narrowed to making a single bounded decision within a state — which validated transition to take — rather than inventing the control flow. The resilient pattern in the field adds an explicit re-planning step as a node in the workflow rather than as an open-ended improvisation (per the agentic-workflow literature). The control flow is deterministic; only the leaf decisions are probabilistic, and each one is gated.

4. Idempotency and retries — make recovery safe

Because individual steps fail, the system must retry. But retrying a probabilistic, side-effecting step is dangerous: the same prompt can produce a different result on the second call, and a naive retry can fire a charge or send an email twice. The durable-execution pattern addresses both halves — wrap each side-effecting operation in a step that persists its result and is keyed for idempotency, so a retry resumes from saved state rather than re-running, and external mutations are deduplicated (per Inngest). Retries make the system resilient to transient failure; idempotency makes the retries safe.

5. Deterministic tools — push real work out of the model

The model should decide; it should rarely compute. Arithmetic, lookups, data transforms, API calls — anything with a correct answer — belongs in deterministic tools the model invokes, not in the model's own generation. A function that totals an invoice returns the same total every time; a model asked to "add these up" does not, reliably. The agentic OS confines the model to the one thing it is uniquely good at — judgment under ambiguity — and routes everything verifiable to code that is right by construction.

6. Human gates — judgment where the stakes justify it

For high-consequence actions, the right control is a human in the loop. The mature pattern is risk-tiered: low-risk steps proceed automatically, while irreversible or high-stakes actions pause for explicit approval. The human is not re-doing the agent's work — they are exercising judgment at the exact point where probabilistic confidence is not enough to bet the business on. In our content pipeline, editorial sign-off is a hard gate no score can bypass.

7. Evals — the regression suite for a probabilistic system

A probabilistic system needs a different measurement discipline than deterministic code. Evals are that discipline: a graded test set, run continuously, that catches quality regressions before users do. The field consensus is that the hard part is not the scoring technique but the operational workflow that turns production failures into reproducible test cases (per LangChain), and that a complete framework combines deterministic checks, LLM-as-judge scoring, and human annotation (per Confident AI). Evals are to an agentic system what a regression suite is to a codebase: the thing that lets you change the probabilistic core and know you did not break behavior.

A Worked Example: A Reliable Loop with a Probabilistic Heart

Make it concrete. Suppose you want an agent that triages inbound support tickets and either auto-resolves them or routes them to a human. The model in the middle is non-deterministic. The loop around it does not have to be.

1. Ingest (deterministic). A ticket arrives. The system writes it to a durable store with a unique ID. This record is the state; nothing downstream relies on the model's memory.

Classify (probabilistic, constrained). The model is asked to classify the ticket — but via a structured output bound to a schema: { "category": enum, "confidence": 0-1, "suggested_action": enum }. The shape is guaranteed even though the content is sampled.

1. Validate (deterministic). A gate checks the output against business rules. Is the category in the allowed set for this customer tier? Is confidence above the threshold for auto-resolution? Fail any check and the ticket transitions to the human-review state.
2. Act through tools (deterministic). If auto-resolving, the agent does not "perform" the resolution in prose — it calls a real, idempotent tool (close the ticket, send the templated reply) keyed to the ticket ID, so a retry cannot double-send.
3. Gate (human). If confidence is low or the action is irreversible, the loop pauses for a human. They approve or correct; their correction is captured.

Record and evaluate (deterministic). Every decision and outcome is logged. That log becomes eval data: misroutes become test cases, and the next model or prompt change is measured against them before it ships.

Run that same ticket through the loop a hundred times and the model may phrase its reasoning a hundred ways. But the ticket is either auto-resolved correctly, or it lands in front of a human — never silently mishandled, never double-charged, always logged. The model is non-deterministic. The loop is reliable. That gap — between an unreliable component and a reliable system — is the whole product.

What This Means for Buyers: You Are Buying a System, Not a Model

This reframes the most important question a technical buyer can ask. The market is loud about which model — the benchmark wars, the context windows, the leaderboard of the month. For most production use cases, the model is increasingly a commodity and an interchangeable one. The differentiator is the OS around it.

Two teams can use the identical model and ship products worlds apart in reliability, because one wrapped it in schemas, gates, state machines, idempotent tools, human checkpoints, and evals — and the other wrapped it in a clever prompt and hope. The first has an engineered system whose failure modes are bounded, observable, and recoverable. The second has a demo that works until it does not, in front of a customer, with no audit trail explaining why.

So when you evaluate an agentic partner, do not ask which model they use. Ask: Where are your validation gates? What is your state model? How do you make retries safe? Which actions require a human, and why those? What is in your eval suite, and what happens when production surfaces a failure your evals missed? A vendor who answers those questions is building you an operating system. A vendor who answers "we use the best model and great prompts" is selling you the probabilistic core with none of the control that makes it safe to run a business on.

Facet's Point of View: We Build the OS, Not Just the Prompts

We will say plainly what we believe, because it is the thesis our practice is organized around: prompts are the cheapest, least durable layer of an agentic system, and the deterministic scaffolding around the model is where the real engineering — and the real reliability — lives. That is an opinion, but it is one earned in the building. We run our own operations, including the content engine that produced this article, on exactly this architecture: probabilistic generation at the core, surrounded by typed schemas, scoring gates above fixed thresholds, explicit state machines, idempotent steps, deterministic tools, hard human gates for editorial judgment, and an eval discipline that turns every miss into a regression test.

None of the individual layers are exotic. Schemas, validators, state machines, idempotency, audit logs, staged rollouts, regression suites — a competent platform team has shipped all of these for a decade. The shift is recognizing that an agentic system needs all of them at once, because for the first time the component in the middle is non-deterministic by nature, and the old habit of leaning on a deterministic core to paper over a missing gate no longer holds. The discipline is doing all the layers, not one.

If your agentic pilot impresses in the demo and frightens you in production, the problem is almost never the model and almost never your engineers. It is that you have a probabilistic core with no operating system around it. That is a fixable problem, and it is precisely the kind we build for a living. Talk to us about an agentic readiness assessment — we will map where your schemas, gates, state model, and evals stand today, and what your system looks like once the model is wrapped in an OS you can actually run a business on.

Sources

Thinking Machines Lab — "Defeating Nondeterminism in LLM Inference": https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/

Vincent Schmalbach — "Does Temperature 0 Guarantee Deterministic LLM Outputs?": https://www.vincentschmalbach.com/does-temperature-0-guarantee-deterministic-llm-outputs/

Protecto — "OpenAI Introduces Structured Outputs for API: Enhancing Reliability with JSON Schemas" (summarizing OpenAI's announcement): https://www.protecto.ai/blog/openai-introduces-structured-outputs-for-api-enhancing-reliability-with-json-schemas/

Inngest — "Durable Execution: The Key to Harnessing AI Agents in Production": https://www.inngest.com/blog/durable-execution-key-to-harnessing-ai-agents

Prompt Engineering — "Agents At Work: The 2026 Playbook for Building Reliable Agentic Workflows": https://promptengineering.org/agents-at-work-the-2026-playbook-for-building-reliable-agentic-workflows/

LangChain — "LLM Evals: Production Monitoring to Regression Tests": https://www.langchain.com/articles/llm-evals

Confident AI — "LLM Agent Evaluation Metrics in 2026": https://www.confident-ai.com/blog/llm-agent-evaluation-complete-guide