AI Reliability Is a Systems Problem

Consider an illustrative case. An internal assistant answers employee questions from a company's policy documents. Before launch, the team writes 200 representative questions with reference answers, and the assistant answers 90% of them correctly. On that question set, the retrieval pipeline and the model do their jobs.

In its first month of use, three things go wrong. A travel policy is revised, but the index still holds the previous version, so the assistant gives a confident answer that was correct last quarter. A contractor asks about pay bands and receives a passage from a document restricted to HR, because permissions were enforced in the interface but not in retrieval. A ticketing tool times out, and the assistant summarises the partial result as if it were complete.

The 90% could not have revealed any of this. The test fixed the questions, the index, the user role and the tools; production varies all four.

Model performance is not system reliability

The NIST AI Risk Management Framework adopts the ISO/IEC TS 5723 definition of reliability: the ability to “perform as required, without failure” for a given time interval, under given conditions [1]. A benchmark score supplies none of the three qualifiers. It specifies no requirement, has no time dimension, and reflects test conditions rather than operating ones.

Several properties are routinely merged under the word reliable. Evidence for one is not evidence for another.

Availability and safety are different again. A service can be fully available while returning stale answers, and safety concerns the harm a failure causes rather than how often failures occur.

Sculley et al. explained why a system's behaviour cannot be read off its components. In ML systems, components are entangled through data, so “changing anything changes everything”, and outputs acquire undeclared consumers that break when behaviour shifts [2]. LLM applications add a further coupling. The model reads whatever upstream components hand it and writes for whatever consumes its output, so its behaviour inside the system is conditional on both.

Where failures emerge

The three failures in the opening share a structure. Each starts at a boundary between components, passes through components that behave correctly given their inputs, and reaches the user as fluent text.

Stale retrieval. Re-indexing runs weekly, and embedding similarity does not encode recency, so the superseded policy ranks first. The model answers faithfully from that context and cites it, so a groundedness check passes. The failure lies in an unwritten requirement that retrieved content be current. Barnett et al., drawing on three RAG case studies, report related failure points, including missing content and relevant documents ranked too low to be used, and conclude that validation of a RAG system is only feasible during operation [3].

Permission leakage. Access control applies when the interface lists documents, but retrieval runs over a shared index, and the model cannot know a passage is restricted. The evaluation ran under one administrator account, so the defect could not appear. This is a security failure, and it needs its own tests.

Partial tool results. The ticketing API times out after returning part of a result set, and the integration layer passes it on without an error flag. Asked to summarise, the model does so fluently. A broken page at least signals a fault. Here the model turns an upstream error into a plausible answer, and the signal is lost.

The retrieval channel is also an attack surface. Greshake et al. showed that instructions planted in content an application later retrieves can redirect its behaviour, including in deployed systems such as Bing Chat [4]. Once an assistant can call tools, retrieved text becomes a path to actions.

Evaluating the complete system

Four kinds of testing answer different questions. Component evaluation shows whether retrieval, generation or a tool meets its own specification, but not how each behaves on the others' real outputs. End-to-end evaluation shows whether realistic requests produce acceptable outcomes, but not why failures occur. Execution traces narrow that gap: recorded with tools such as LangSmith or Arize Phoenix, a trace shows which documents a request retrieved, in what order, and what the model and tools then did with them. It records what happened, not whether it should have: a trace shows that a policy document was retrieved, not whether that document was current, authoritative or permitted for the user who asked. Integration and fault testing shows behaviour under stale data, failed tools and restricted users, but not how often those conditions arise. Monitoring shows what real users experience, but only what is logged. Three properties of LLM applications widen the gaps between these layers.

Component scores do not compose

Let $r$ be the retrieval outcome for a request: relevant and current, stale, or empty. Treating these outcomes as exclusive, end-to-end correctness is

A generator evaluated on gold context approximates, at best, the term for relevant retrieval, on cleaner context and a different mix of queries than production. The term that often carries the risk, a confident answer when retrieval is stale or empty, is not measured at all. Errors may also correlate, since queries that retrieve poorly are plausibly harder to answer. Multiplying component accuracies can then misstate end-to-end accuracy in either direction.

Averages hide consistency

Yao et al. define $\mathrm{pass}^k$ as the probability that all $k$ independent trials of a task succeed, averaged over tasks [5]. If task $t$ succeeds with probability $p_t$ on a single run:

Two systems with an 80% mean can differ sharply. System A succeeds with probability 0.8 on every task, so $\mathrm{pass}^8 = 0.8^8 \approx 0.17$. System B always succeeds on 80% of tasks and always fails on the rest, so $\mathrm{pass}^8 = 0.80$. In the τ-bench retail domain, the 2024 version of gpt-4o scored 61.2% at $k = 1$ and below 25% at $k = 8$ [5].

System B is easier to engineer around only if its failing task classes can be identified before a request is answered, and routed somewhere acceptable. Otherwise its failures are as unpredictable to a user as A's. Either way, the evidence required is repeated trials per task, which multiplies evaluation cost by $k$.

The evaluation is part of the system

An evaluation is a measurement, and its own error bounds what can be claimed.

• Sample size. If $n$ independent cases drawn from the production distribution yield no failures, the one-sided 95% upper bound on the failure rate is about $3/n$ (the rule of three):
  $$p_{\max} = 1 - 0.05^{1/n} \approx \frac{3}{n}$$
  The opening's 90% on 200 questions has a 95% Wilson interval of roughly 85% to 93%, and that interval applies only to the population the questions were drawn from.
• Judge error. Suppose an LLM judge flags a true failure with probability $s$ (its true positive rate) and a correct output with probability $q$ (one minus its true negative rate). A true failure rate $f$ is then measured as
  $$m = f\,s + (1 - f)\,q$$
  With $f = 0.05$, $s = 0.8$ and $q = 0.05$, the measured rate is 8.75%. If $s$ and $q$ are estimated against human labels, as Husain and Shankar's practitioner guide recommends [6], the true rate can be recovered, with their estimation error carried through:
  $$f = \frac{m - q}{s - q}$$
• Moving criteria. Shankar et al. observed people refining their criteria while grading outputs, which they call criteria drift [7]. A specification written before error analysis is likely to change after it, and the test set ages with it.

Offline results also drift from online behaviour. A hosted model can change behind an unpinned version name, and prompts tuned against an evaluation set inflate its score. A held-out test set, pinned model versions and a staged rollout to a small share of users guard against each of these. Platforms such as LangSmith or MLflow can run each application version against a stored evaluation dataset, score it and record the results, so a regression suite is rerun on every change rather than remembered. They execute and record the tests. Choosing representative cases and writing the acceptance criteria remain the team's work, and a well-instrumented experiment on an unrepresentative dataset still measures the wrong population.

A practical reliability framework

The structure below is a working synthesis for organising a review, not a validated standard. It builds on Google's ML Test Score, a rubric of 28 tests and monitoring needs for production ML systems [8], and on the trustworthiness characteristics of the NIST AI RMF [1]. What it adds for LLM applications is the evidence each dimension should produce. The mitigations it lists are established practitioner practice, not empirically validated controls.

Step 0: Specification. State the intended use, the cost of a wrong output, latency and cost limits, and acceptance criteria per task type. Revisit them after the first round of error analysis.

1. Task performance. Does the application produce correct, useful outcomes for realistic requests, consistently?
2. System integrity. Do components pass data, state and errors between them correctly, and fail detectably?
3. Security and boundaries. Does the system enforce who may see and do what, including under deliberate misuse?
4. Operational reliability. Will degradation be noticed, attributed and reversed? Instrumenting retrieval, model and tool calls as spans of one distributed trace, for example with OpenTelemetry, lets a change be attributed to the component that caused it. Instrumentation is not reliability: a fully traced system can still return wrong answers, and its traces will record them faithfully.

Security is separate because an adversary chooses its inputs. Reliability evidence assumes realistic use, so a high pass rate says little about exploitability. Human oversight is a mitigation within any dimension rather than a fifth one. None of the dimensions establishes correctness or safety; together they organise the evidence a team needs before making that judgement.

Assessment matrix

R marks the retrieval assistant from the opening. F marks a hypothetical financial analysis agent that reads filings and computes ratios, to show where higher risk demands more.

Reliability is an engineering decision

How much evidence is enough depends on what a failure costs and whether anyone notices it. Failure mode and effects analysis captures this by ranking each failure on severity, likelihood of occurrence and difficulty of detection, and multiplying the three scores, an ordinal guide rather than a probability. In our view, detection is often the cheapest term to reduce: an assistant that abstains when retrieval finds nothing current can be safer to deploy than a more accurate one that never abstains.

For the policy assistant, errors are usually recoverable and often noticed, so a regression suite, sampled review, freshness checks and role-based retrieval tests may be proportionate. The financial agent's errors are numerical, plausible and can flow into decisions. It needs repeated-trial consistency, exact reference checks, fault injection on every data source, and human approval before consequential actions. The framework is the same; the evidence threshold is not.

Some of the strongest reliability decisions remove components. Zinkevich's first rule is not to be afraid to launch without machine learning; his fourth is to keep the first model simple and get the infrastructure right [9]. For LLM applications, that means arithmetic in code, fixed workflows where the steps are known, and only the tools a task needs.

Before launch, a team should be able to show:

• written acceptance criteria;
• an evaluation whose confidence interval lies within the tolerance the business has set;
• no leakage across the role matrix;
• defined behaviour for each injected fault;
• a tested rollback;
• a named owner, with monitoring in place.

A gap in any of these is a reason to delay or narrow the scope.

Conclusion

A benchmark score is evidence about a model on a fixed distribution. A deployment decision needs evidence about an application over time, under real conditions, including those in which its parts fail. A better model does not close that gap, and a more elaborate system can widen it.

For an application already in use, three checks are a practical start. Run each evaluation task several times and examine the spread. Re-test generation on the context retrieval actually returns, including stale and empty results. And report the uncertainty of the evaluation itself: its sample size, and how often its judge agrees with people. Where a check exposes a weakness, simplifying the system is often cheaper than adding tests.

Explore Pontus' Scientific AI expertise →

References

1. NIST, Artificial Intelligence Risk Management Framework (AI RMF 1.0), Section 3, 2023. Reliability definition adopted from ISO/IEC TS 5723:2022.
2. D. Sculley et al., “Hidden Technical Debt in Machine Learning Systems”, NeurIPS 2015.
3. S. Barnett et al., “Seven Failure Points When Engineering a Retrieval Augmented Generation System”, CAIN 2024.
4. K. Greshake et al., “Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection”, arXiv:2302.12173, 2023.
5. S. Yao et al., “τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains”, ICLR 2025.
6. H. Husain and S. Shankar, “LLM Evals: Everything You Need to Know”, practitioner guide.
7. S. Shankar et al., “Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences”, UIST 2024.
8. E. Breck et al., “The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction”, IEEE Big Data 2017.
9. M. Zinkevich, Rules of Machine Learning: Best Practices for ML Engineering, Google.

Further reading: C. Huyen, AI Engineering: Building Applications with Foundation Models, O'Reilly, 2024, chapters 3, 4 and 10, on evaluation methodology and system architecture. E. Yan, “Patterns for Building LLM-based Systems & Products”, 2023, on evals, guardrails and defensive UX.