The Coverage Problem: Why LLMs Can't Do Matrix Math

LLM limitations are not about intelligence — they are about guarantees. A large language model can produce a plausible answer for almost any single input you hand it. What it cannot do is guarantee correctness across the full combinatorial space of inputs and conditions that a real production system must handle. That gap — between "works on the inputs I tested" and "works on every input that will ever arrive" — is the coverage problem. And until you understand it precisely, you will deploy AI in the wrong places and trust it with work it is structurally incapable of doing reliably.

This article explains what the coverage problem is, why it is intrinsic to how language models work, and what disciplined engineering teams do about it. The answer involves wrapping probabilistic models in deterministic checks and evals that verify not just sampled cases, but every cell of the matrix that matters.

What LLMs Actually Do (And Don't Do)

A language model is, at its core, a probabilistic next-token predictor. Given a sequence of tokens as input, it assigns a probability distribution over what token should come next, samples from that distribution, and repeats. The output is coherent — often strikingly so — because the model has learned statistical patterns from an enormous corpus of text. It has not learned rules. It has learned correlations.

This distinction sounds academic until you think about what it means for correctness guarantees.

A rule-based system — a billing engine, a compliance checker, a data transform — is evaluated against its specification. Either the VAT calculation is correct for all applicable jurisdictions, or it is not. Either the HIPAA audit log captures every required field, or it does not. The test matrix for a deterministic system is finite: you enumerate the cases that matter, you verify each one, and you ship when the matrix is green.

A language model has no such specification. Its behavior on any given input is a sample from a probability distribution. Run the same prompt twice and you may get two different answers. The distribution is concentrated enough that most outputs look right — and that concentration is what makes LLMs useful. But "concentrated" is not "guaranteed." There is always a tail. And in production, the tail eventually arrives.

The Combinatorial Explosion

The coverage problem compounds when inputs interact. Consider a billing transform that must handle:

• Three pricing tiers (entry, professional, enterprise)
• Twelve billing periods (monthly, quarterly, annual — across multiple fiscal year starts)
• Seven jurisdictions with distinct tax rules
• Four contract types with different proration logic
• Two currencies with rounding rules

That is already 3 × 12 × 7 × 4 × 2 = 2,016 distinct combinations before you layer in edge cases like mid-cycle upgrades, trial extensions, and retroactive discounts. A developer testing manually might cover 30 or 40 of those combinations. A model asked to "write a billing transform" produces something that looks right for the cases it was shown. The other 1,970+ combinations are a tail. In a probabilistic system, that tail is unknown — not verified-correct, not verified-wrong, just uncharted.

For anything where coverage matters, "looks right" is not an acceptance criterion.

The LLM Limitations Prompting Cannot Fix

A common response to the coverage problem is to improve the prompt. Add more constraints. Be more specific. Add examples. Chain the reasoning with better structure.

These techniques raise the probability of getting the right answer on typical inputs. They do not close the coverage gap. The model is still sampling from a distribution. The distribution is now better-calibrated, but it is still a distribution. You have narrowed the tail; you have not eliminated it. AI coverage — the guarantee that a system behaves correctly across every required input condition — cannot be achieved by better prompting. It requires deterministic checks that sit outside the model and verify outputs against a complete specification.

The same logic applies to model scale. Larger models produce outputs that are more consistently correct on well-represented inputs. They still have tails on rare or adversarial inputs. The coverage problem does not disappear with scale — it shrinks on the dense regions of input space and persists on the sparse ones. Those sparse regions are exactly where production failures live: the unusual jurisdiction, the edge-case proration, the malformed upstream record that nobody tested.

Understanding these llm limitations is prerequisite knowledge for any engineering team building production AI systems. The model is a powerful generation engine. It is not a verifier. It cannot check its own work against a complete specification, because it has no complete specification to check against. You have to provide that from outside the model.

The Matrix Your Model Can't Cover

Here is a useful mental model: think of your system's input space as a matrix. Rows are input types or conditions. Columns are behavioral requirements. Each cell is a combination that your system must handle correctly. The total number of cells is the coverage requirement.

A language model, left to its own devices, fills in some cells confidently and leaves others unaddressed — often without signaling which is which. The confident-looking output is the same whether the model has good coverage or poor coverage. The model does not know what it does not know, because it has no specification against which to check its own coverage.

This matters most in three categories of work:

Billing and financial calculations. Every combination of pricing tier, jurisdiction, contract type, and time period is a cell. A wrong answer in any cell is a financial error — potentially a compliance exposure. The matrix must be fully covered and the coverage must be verified.

Compliance and audit requirements. Compliance is defined by enumerated rules, not by probability. Either every required field is captured, or the audit fails. Either every consent log is written with the correct timestamp, or you have a gap. The model cannot enumerate the rules it might have missed.

Data transforms and ETL pipelines. A transform that runs correctly on the 90% of records that look like the training data will silently corrupt the other 10%. In a batch pipeline processing tens of thousands of records, that 10% represents real records belonging to real people or real transactions. The silent corruption is worse than a loud failure, because you may not discover it until downstream systems have already acted on the bad data.

The CHECK Layer: Deterministic Verification at Scale

The engineering response to the coverage problem is not to avoid LLMs — it is to pair them with a deterministic verification layer. This is what Facet's Agent Loop architecture calls the CHECK layer: a set of evals and assertions that run after model output and before any downstream action is taken.

The CHECK layer does not replace the model. It validates the model's output against the specification the model cannot hold itself. Where the model is good at generating plausible structure, the CHECK layer — built from deterministic checks against your coverage matrix — is good at confirming that the structure satisfies every required constraint.

What goes in a CHECK layer:

• Schema validation. The output must conform to a typed schema. Missing fields, wrong types, and values outside allowed ranges are caught before they propagate.
• Business rule assertions. For each cell in your coverage matrix, write a deterministic assertion. The VAT rate for jurisdiction X is Y%. The proration formula for mid-cycle upgrade is Z. These are not learned — they are specified, and the CHECK layer enforces them.
• Coverage evals. For each category of input your system must handle, run the model output against a test suite that exercises the full matrix. Not a sample. The full matrix. An eval suite that covers 40 of 2,016 combinations is not a coverage eval — it is a spot check. Rigorous LLM evaluation at the cell level, not the aggregate level, is what separates a verified system from an optimistic one.
• Invariant checks. Some properties must hold regardless of input. A billing total must be non-negative. A transformed record must preserve the source record ID. An audit log entry must include a timestamp. These invariants are the cheapest assertions to write and the most important to have.
• Retry and escalation logic. When a CHECK layer assertion fails, the system does not silently proceed. It retries with a different generation, escalates to a human review queue, or halts — depending on the severity of the failure and the operational context.

The CHECK layer is not elegant. It requires you to enumerate your requirements explicitly, which is work. But that work is the whole point: the act of writing the matrix forces you to discover coverage gaps before deployment rather than after.

Illustrative Math: What The Tail Costs

Consider a billing pipeline processing 50,000 invoices per month (illustrative). If an LLM-based billing transform is correct on 98% of inputs — a strong performance — that still leaves 1,000 invoices per month with incorrect calculations. If the average invoice value is $2,000 (illustrative), the exposure from the unchecked tail is $2 million per month. Not from malice. Not from a bug anyone noticed. From a tail that the model never told you existed.

Now add a CHECK layer with complete coverage assertions. The 1,000 failures surface as assertion failures instead of billing errors. The system halts or escalates. You fix the generation logic. The tail shrinks. The exposure disappears.

The cost of the CHECK layer — engineering time to enumerate the matrix and write the assertions — is a one-time investment that pays for itself on the first month of avoided errors.

Where This Fits In An Agent Loop

Facet's framework for agentic systems is built around a three-layer loop: PLAN, ACT, CHECK. The PLAN layer structures the task and sequences the work. The ACT layer generates outputs — this is where the LLM operates. The CHECK layer verifies those outputs before they leave the loop.

The coverage problem lives at the ACT/CHECK boundary. The ACT layer will always have a tail. The CHECK layer is what determines whether that tail causes damage. Without a CHECK layer, the loop is open-ended: the model's output is the final word, and you discover tail failures in production. With a complete CHECK layer, the loop is closed: no output leaves the system until it has been verified against the full coverage matrix.

This is the engineering discipline that makes agentic systems production-grade. It is also what separates a well-built Agent Loop from a demo that happened to work on the inputs you showed it. The demo's tail is invisible because demos don't run the full matrix. Production does.

For technical leaders evaluating agentic AI — whether you're building internal capability or deciding where in your stack to apply LLMs — the coverage problem and the CHECK layer are the two concepts that determine whether an AI integration is production-grade or a liability.

How Facet Approaches This In Practice

Facet has been building agentic systems for production use across professional services, healthcare administration, and financial services. The coverage problem shows up consistently, and the teams that handle it well share one discipline: they enumerate the matrix before they write the model integration, not after.

That means starting with a coverage specification — every input condition and behavioral requirement the system must handle — before any model is selected or prompt is written. The specification becomes the test suite. The test suite becomes the CHECK layer. The coverage is verified before deployment and runs automatically on every change.

Teams that start with the matrix ship agentic systems that stay reliable in production. Teams that start with the model and test optimistically find the tail in production, usually at the worst possible moment. The LLM is a powerful component in either approach. The CHECK layer is what makes it a trustworthy one.

Let’s talk it through at a time that works for you.

FAQ

What Makes LLM Limitations Different From Regular Software Bugs?

A regular software bug is deterministic: given the same input, the bug produces the same wrong output every time. That makes it findable — reproduce the input, observe the failure, fix the logic. An LLM limitation is probabilistic: the model might answer correctly on 99 out of 100 calls with the same input, then fail on the 100th because the sampling landed differently. That makes it harder to catch in testing and harder to reason about in production. The correct response is not better debugging — it is external verification via a CHECK layer that does not rely on the model's consistency.

Can Fine-Tuning Solve The Coverage Problem?

Fine-tuning improves the model's behavior on the distribution it was tuned on. It does not guarantee coverage on the full input space, because the input space is always larger than the fine-tuning dataset. If you fine-tune on 10,000 billing examples, you improve performance on inputs that look like those 10,000 examples. You do not automatically improve performance on the combinations that were absent from the fine-tuning set — and those are exactly the rare cases that compose the dangerous tail. Fine-tuning and a CHECK layer are complementary, not alternatives.

How Do You Size A CHECK Layer For A Real System?

Start with the coverage matrix: enumerate the distinct input conditions your system must handle. For each condition, write at minimum one assertion that would catch a wrong answer. Prioritize assertions on the conditions with the highest consequence — financial exposure, compliance risk, irreversible downstream actions. A useful heuristic: if a wrong answer in that condition would require a manual correction, the condition needs an assertion. The goal is not 100% automation of every conceivable check — it is that every high-consequence cell in the matrix is covered by at least one deterministic assertion before output leaves the system.

Is This Only Relevant For Large Or Complex Systems?

The coverage problem scales with the number of distinct input conditions your system must handle, not with the overall size of the system. A small but high-stakes integration — processing reimbursements for 200 employees, say — can have a large coverage matrix if the reimbursement rules are complex. Conversely, a large system processing simple, homogeneous inputs might have a small coverage matrix that is straightforward to verify. The question to ask is not "how big is this system?" but "how many distinct input conditions must it handle correctly?" If the answer is more than a handful, the coverage problem is real and the CHECK layer is necessary.