The Validator Method: the engineering case for keeping a human in the loop.

A generative–discriminative pairing

The architecture assigns the language model the role of a generative model — a sampler over the space of plausible outputs — and the human the role of a discriminative validator: a classifier that maps a candidate output to {accept, correct, reject}. These are formally distinct operations. Generation searches a combinatorially vast output space; discrimination evaluates a single point in that space against criteria the generator does not itself possess. The model supplies coverage and throughput; the human supplies the decision boundary and the accountability. The invariant that makes it safe is architectural: the model emits suggestions; the human holds the sole write authority over every system of record.

The generation–verification asymmetry

The pairing is economically viable because verification is, in general, far cheaper than generation — the same asymmetry that defines the relationship between solving and checking across computer science, where producing a solution can be intractable while validating a proposed one is efficient. A reviewer who could not author a multi-page analysis in twenty minutes can adjudicate one in two, because reading-to-verify is a lower-complexity operation than searching-to-produce. The validator’s throughput requirement therefore scales sub-linearly with the generator’s output volume; one qualified human governs a large stream of model output without becoming the bottleneck. The asymmetry is not a convenience — it is the property that makes human oversight tractable at production scale rather than merely desirable.

The “Centaur” configuration

In the pre-registered field experiment conducted with Boston Consulting Group (Dell’Acqua et al., 2023; Organization Science, 2025), the highest-performing operators were classified as “Centaurs”: subjects who imposed an explicit task decomposition, routing sub-tasks inside the model’s capability frontier to the model and reserving sub-tasks requiring judgment for themselves. The result is directional and strong — benefit is contingent on a deliberate human-machine boundary, not on adoption alone. Where the seam was well-drawn, AI assistance produced large gains. Where it was blurred — work routed to the model that lay outside its competence — performance degraded relative to working without AI at all. The value is created by the placement of the seam, and placing the seam is an act of human judgment about the model’s competence.

The sidecar: read-only by construction, least privilege by default

The model is deployed as a sidecar — a process external to the systems of record, integrated only at the API layer and consuming sanitized event streams or document exports rather than live production data. Its service account is provisioned under strict least privilege: read-only scopes on an explicitly allow-listed set of endpoints, with no write credentials, no INSERT/UPDATE/DELETE grants, no ERP/CRM mutation tokens, and no network route into production. Its outputs are typed suggestion objects routed to a review surface, never commits against a system of record. The result is an enforced air gap between the inference plane, where the model reasons, and the execution plane, where the business acts — a separation maintained by access control and network topology, not by the model’s cooperation.

Bounded blast radius

The read-only sidecar yields a formally bounded worst case. Enumerate the failure set — confident factual error, adversarial manipulation, distributional drift, provider outage, silent capability regression. For an autonomous agent holding write authority, these map to unbounded consequences: corrupted records, unauthorized transactions, data exfiltration. For the sidecar, every element collapses to the same terminal state, because the model possesses no actuator: a suggestion the validator rejects. The blast radius — the maximal damage reachable from any single failure — is one discarded artifact and a few seconds of review time, independent of failure mode. The architecture does not lower the probability of model error; it caps the cost of model error at a constant, low value.

Model swappability and business continuity

Because the model is a decoupled service dependency rather than logic embedded in business processes, it is a replaceable component. Model drift, vendor price changes, contractual shifts, or a superior alternative are accommodated by swapping the sidecar behind a single flag, with no refactoring of the systems of record — the interface contract is stable while the model behind it is fungible. Two consequences follow. First, vendor lock-in is structurally eliminated: the business is never welded to one provider’s roadmap, pricing, or survival. Second, continuity is guaranteed by topology: if the model is disabled or the provider disappears, core operations are unaffected because the critical path never traversed the model.

What the validator supplies and the model lacks

The human is retained because the validator role demands faculties the architecture does not contain: the ability to override a statistically likely output that conflicts with intent; memory indexed to real consequences rather than to training correlations; conflict detection — sensing a flawless-looking output is wrong; genuine causal reasoning at the interventional and counterfactual tiers where the model is confined to association; and, decisively, accountability — a legally responsible party, which no software can be. In any consequential workflow someone must be answerable, and that requirement alone places a human at the decision point.

Ashby’s Law of Requisite Variety

Ross Ashby’s Law of Requisite Variety (1956), foundational to cybernetics, states that only variety can absorb variety: to regulate a system effectively, a controller must command at least as much variety — distinguishable states and responses — as the disturbances it must counter. The real world presents effectively unbounded variety: novel markets, unprecedented fact patterns, adversarial innovation, shifting social dynamics. A language model’s response variety is bounded by a finite corpus, fixed parameters, and a finite window; it cannot represent or answer disturbances it has never seen. When the regulator’s variety falls below the environment’s, control is lost regardless of how confident the output sounds. The human supplies the missing variety — episodic memory, causal reasoning, social and moral judgment — which is why the validator is not a courtesy but a control-theoretic requirement.

The O-Ring Theory of production

Michael Kremer’s O-Ring Theory (1993) models production as a chain of interdependent tasks in which the failure of any single task can collapse the value of the entire product — as a single failed seal destroyed a space shuttle. A key implication is that quality cannot substitute for quantity: high-quality work at one stage cannot compensate for a failure at another. In a human-AI workflow the chain runs data → prompt → generation → validation → execution. The model handles generation; the human handles validation. Remove or weaken the validation link — through automation bias, deskilling, or absence — and the whole chain fails, no matter how strong the generation link is. You cannot compensate for a missing validator with more generation; that only produces more unchecked output.

High Reliability Organizations and defense in depth

Weick and Sutcliffe’s research on High Reliability Organizations — aircraft carriers, nuclear plants, air traffic control, which sustain very low failure rates in high-risk settings — identifies five operating principles, and the validator architecture is their operationalization in AI: a preoccupation with failure (the reviewer hunts for what is wrong, not for reasons to approve), reluctance to simplify (“the model said so” is not an explanation), sensitivity to operations (the reviewer is embedded in the real domain), commitment to resilience (handling novel cases the model never saw), and deference to expertise (domain knowledge overrides both the model and management pressure). Reason’s Swiss-Cheese model complements this: defenses are slices with holes, and accidents occur when the holes align. The validator is an additional thick slice — a redundant barrier catching what slips past alignment, prompting, grounding, and monitoring.

Avoiding the “moral crumple zone”

Madeleine Elish’s concept of the “moral crumple zone” (2019) describes how responsibility is misattributed to a human who had little real control over an automated system — the human becomes a “liability sponge” absorbing blame for systemic design failures while the technology and its designers are shielded. A “human in the loop” who is asked to catch machine-speed errors in real time, without authority or context, is exactly such a crumple zone: liability without control. The validator architecture refuses that trap. The human is an active adjudicator with sole write authority, operating at human time-scales, with full context, and the explicit power to reject. Responsibility is proportional to authority — the validator can actually prevent the error, so accountability is legitimate rather than theatrical.

Human oversight as a codified control and a legal requirement

The architecture is congruent with the prevailing governance instruments. In the U.S., the NIST AI Risk Management Framework (AI RMF 1.0, 2023) treats human oversight as a documented, auditable control across its functions — MAP (plan human-AI configurations and document oversight roles), MEASURE (track metrics on those configurations), and MANAGE (post-deployment monitoring with override mechanisms). In the EU, Article 14 of the AI Act (Regulation (EU) 2024/1689) makes human oversight a legal requirement for high-risk systems: they must be designed so natural persons can understand, monitor, interpret, intervene in, and halt them, with explicit attention to automation bias. A precise and favorable reading: Article 14 obligates the capability for oversight and intervention; it does not, in every case, mandate prior approval of each output. A method that interposes a human approval gate before action therefore satisfies and exceeds the statutory baseline.

What the disciplined configuration captures

Executed with a deliberate human-machine boundary, the productivity effects measured in controlled research are large. The figures below come from specific studies of knowledge work under specific conditions; they are evidence of attainable magnitude, not a representation of any particular result.

A conditional next-token sampler, not a reasoner

A modern LLM is a stack of transformer decoder blocks trained by self-supervised learning to minimize the cross-entropy between its predicted next-token distribution and the observed token over a corpus. Its entire learned competence is an estimate of one conditional distribution:

At inference it performs autoregressive decoding: it computes a distribution over the vocabulary via a softmax over output logits, samples a token under a policy (temperature, top-k, top-p), appends it, and repeats. There is no symbolic logic engine, no retrieval against a verified store of facts, no runtime truth-evaluation step. What it produces is a maximum-plausibility continuation — true only insofar as plausible and true coincide in the training data.

Why compression cannot create truth

The training objective, cross-entropy minimization, decomposes into the data’s own entropy plus the divergence between the model’s distribution and reality:

Minimizing cross-entropy is equivalent to minimizing the KL divergence — pushing the model’s distribution toward the training distribution. This is compression, not truth-seeking. And no procedure can extract more information from a dataset than the dataset contains; the training data is a ceiling. When the model meets genuine novelty — a new market condition, an unprecedented fact pattern — it has no mechanism to construct true information that was never present to learn. It can only recombine what it has seen, however confidently.

Thermodynamic bounds on continuous inference

Landauer’s principle sets a floor on irreversible computation: erasing one bit costs at least k_BT·ln 2 joules of energy. Real processors operate many orders of magnitude above that floor because of error correction, clocking, and memory movement — and inference cost scales with model size, sequence length, and throughput. An autonomous agent running continuous inference over a large context, in tool-use loops, consumes energy at a rate that is not merely costly but physically bounded as models and context windows grow. The validator architecture is, by contrast, thermodynamically frugal: the model runs only when invoked, on discrete, bounded tasks, rather than continuously monitoring and acting.

The No Free Lunch theorem

The No Free Lunch theorem holds that, averaged over all possible data-generating distributions, every learning algorithm achieves the same error on unseen data — there is no universally optimal learner. Models generalize only by making assumptions about the structure of the world, and that generalization holds only while the deployment distribution shares structure with the training distribution. When the world shifts, the very simplicity bias that enabled generalization becomes a liability, nudging the model toward simple, plausible, and wrong answers over complex, unfamiliar, and correct ones. An autonomous agent in an open, non-stationary world has no guarantee its past competence transfers; the validator is the adaptive component that can notice the shift and adjust.

The Bitter Lesson and its corollary

Rich Sutton’s “bitter lesson” observes that general methods which scale with computation eventually beat human-engineered knowledge — true for chess, Go, vision, and speech. But it carries a rarely-stated corollary: what scales capability does not scale alignment, safety, or controllability. The same scaling that yields fluent medical advice yields fluent medical misinformation; the same optimization power that solves a game also discovers how to game a reward. Alignment is a separate engineering problem that does not arrive automatically with intelligence. The validator architecture sidesteps it: the model need not hold correct values, because it holds no authority to act on values — the human supplies the values, the judgment, and the accountability.

The computational cost of attention

Transformer inference is bounded by memory bandwidth as much as by raw compute. Because every token attends to every other, attention scales as O(n²) in sequence length, and decoding must repeatedly read the per-token key/value cache from memory — so doubling the context window roughly quadruples the attention cost, regardless of how many operations the hardware can perform. An autonomous agent that tries to maintain continuous, growing awareness of its environment, its history, and its goals demands an unbounded window; the quadratic wall forces it to either forget (lose context) or slow to a crawl. The validator does not hit this wall, because it operates on discrete, bounded suggestions rather than a continuous unbounded stream of state.

Confabulation is the objective satisfied, not violated

It follows from the objective that the architecture has no native uncertainty gate — no mechanism that detects an out-of-distribution query and returns an abstention. When a prompt lands in a thin or absent region of what the model learned, it does not halt; continuation is what the loss rewards, so it samples a fluent answer regardless. The popular word “hallucination” misleads because it implies a malfunction. Confabulation is the model executing its objective faithfully in a region where faithful execution yields a false output — a feature out of its depth, not a defect. Retrieval augmentation narrows the failure region by grounding generation in supplied documents, but it installs no uncertainty gate; over irrelevant or poisoned context, the model still produces a confident continuation.

Exposure bias and autoregressive error propagation

Training uses teacher forcing — at each step the model is conditioned on the ground-truth prefix. Inference cannot: the model must condition on its own previous output. This train/inference mismatch is exposure bias, and under autoregression its effect is compounding: once a sampled token drifts off the data manifold, every later token is conditioned on an increasingly contaminated prefix, so the trajectory diverges rather than self-corrects. In a short answer this is one wrong clause; in an autonomous multi-step agent, where step n’s output is the trusted input to step n+1, it becomes a cascade — a small early deviation amplified across the chain with nothing positioned to arrest it.

Calibration decoupling under RLHF

After pretraining, models are typically aligned with human feedback (RLHF): a reward model trained on human preference rankings fine-tunes the policy. Human raters reliably prefer fluent, confident, helpful answers and penalize hedging. A documented side effect is the erosion of calibration — the correspondence between a model’s expressed confidence and its actual accuracy. Base models tend to be better calibrated; the aligned policy’s expressed certainty decouples from correctness. The practical implication is sharp: the model can be most assertive precisely where it is wrong, which falsifies any rule of the form “trust it when it sounds sure.”

Bounded window, statelessness, and a weak middle

Each layer computes scaled dot-product self-attention:

Because every token attends to every other and the key/value cache must be held in memory, there is a hard context window; content beyond it is not summarized, simply absent. Recall is also uneven within the window — the documented “lost-in-the-middle” effect shows accuracy peaking for material at the start and end and sagging in the interior. And across sessions the model is stateless: each call is independent, carrying no genuine episodic memory. Context that overflows the window, or sits in its weak middle, is effectively invisible.

The jagged frontier and the absence of metacognition

Model competence is not smooth across tasks. The “jagged frontier” (Dell’Acqua et al., 2023) describes a boundary at which a model performs at expert level on one task and fails on a neighboring task a person would judge only slightly harder, because the latter needs a capability combination outside the training manifold. The compounding problem is the absence of a metacognitive layer: the model emits no reliable signal indicating which side of the frontier a query occupies — it cannot self-report incompetence. In the same research, applying the model to a task outside the frontier left people 19 percentage points less likely to reach a correct answer than working unaided. Assistance inverted into impairment, invisibly.

Chain-of-thought can be post-hoc rationalization

Chain-of-thought prompting was proposed as a transparency mechanism — a window into the model’s “thinking.” Research has since shown those traces can be unfaithful: they do not necessarily reflect the actual internal process, but serve as plausible-sounding justifications for an answer reached by other means. Models can produce systematically unfaithful explanations when biased toward an answer, and faithfulness tends to fall as task difficulty rises. The consequence for autonomy is direct: if the reasoning trace is unfaithful, it cannot serve as a self-audit. The agent’s explanation of why it acted may be a confabulation. A human reviewer can detect the disconnect between a claimed rationale and the actual output; the model cannot audit itself.

Why RAG does not install a truth boundary

Retrieval-Augmented Generation is often offered as a hallucination fix: ground the model in real documents and it will stick to facts. RAG narrows the failure region, but it adds its own failure modes and does not solve the core problem — the model still produces a confident continuation over whatever context it receives, with no mechanism to verify that the retrieved material is correct, relevant, or un-poisoned. If retrieval misses, the model fills the gap by inventing; if the corpus is poisoned, the model confidently synthesizes the poison. RAG is a relevance system, not a safety system.

Indirect prompt injection: no instruction/data trust boundary

This is the failure mode that turns an autonomous, tool-enabled agent into a material liability. A language model has no architectural trust boundary between instructions and data — both enter as tokens in a single context, and the model cannot reliably tell a directive from its operator apart from one embedded in the content it is processing. So any artifact the agent reads — an email, a web page, a PDF, a calendar invite, a customer message — can carry hidden instructions the model may execute. This is indirect prompt injection. If that agent also holds the power to act — send, pay, modify, export — text planted by an outsider in a document the agent will read becomes a path to make it act against the business. There is no clean fix at the model layer, because the vulnerability is intrinsic to following natural-language instructions over mixed-trust input.

Tool use and function calling widen the attack surface

Function calling extends the model from a text generator into an agent that can invoke APIs, modify databases, send messages, and run code — and this is precisely what converts the vulnerabilities above from bad text into real-world damage. The model must reason correctly about when and how to call a function, and it fails in edge cases: choosing the wrong function, supplying wrong parameters, or acting where it should not. Reliability depends heavily on the quality of the function descriptions, and ambiguous schemas produce misinterpretation. Crucially, the model’s output is no longer just text; it is a structured command an external system will execute. Without a human interposed between the proposed command and its execution, every function call is an unauthenticated action.

Why stacking AI agents amplifies the problem

Multi-agent setups are often proposed as a fix: if one model is weak, let several check each other. The dynamics tend the other way. If the same model plans and verifies, the verifier shares the planner’s blind spots — a monoculture in which the fault-tolerance assumption fails because the agents share training data, architecture, and failure modes. A confident assertion by one agent tends to be ratified rather than challenged by others, so a single early hallucination can propagate through shared memory until a false consensus locks in. Adding agents distributes and amplifies model incompetence; it does not cure it. One competent human reviewer is more reliable than a committee of correlated models.

Stochastic decoding undercuts auditability

Decoding is generally stochastic — temperature and nucleus sampling introduce controlled randomness, so identical inputs can yield different outputs across runs (and even nominally greedy decoding is subject to non-determinism at scale). For ideation that variance is harmless or useful. For a system of record, or any process that must be audited, explained, or defended after the fact, it is disqualifying: a process that cannot be reliably reproduced is hard to verify, and a decision pathway that shifts run-to-run is hard to account for. A human decision point restores both — a discrete, attributable, reconstructable record of who decided what, on what basis, and when.

Models replicate human biases rather than correct them

Trained on human-generated text, language models absorb the cognitive biases that decades of behavioral economics have documented — and on preference-type questions (risk and time trade-offs), advanced models often reproduce human-like irrationality rather than normative rationality. Two classic traps are especially relevant to autonomous optimization. The McNamara fallacy — optimizing what is easy to measure while ignoring what is not — describes a system that maximizes its reward proxy (fluency, completion, engagement) while neglecting unmeasured externalities (truth, safety, fairness, long-term consequence). The Cobra effect describes a fix that worsens the problem it targets. An autonomous agent that replicates human bias without oversight makes the same systematic errors a human would, but faster, more consistently, and at scale. The human reviewer is the bias check.

Multi-modal hallucination

Adding vision is often assumed to make a model truthful — if it can see the image, surely it won’t invent what isn’t there. In practice, multi-modal models add new failure modes: category hallucination (naming objects that aren’t present), attribute hallucination (right object, wrong color/count/shape), and relation hallucination (right objects, fabricated spatial or functional relationships). The root cause is a modality imbalance — the language component is typically far stronger than the vision encoder, so deep language priors can override visual evidence. Shown a counter-intuitive image, the model may describe what its training says should be there rather than what is actually shown. It looks with its training data, not its eyes.

AI-generated code is systematically vulnerable

Model-generated code is not merely occasionally flawed; benchmark studies repeatedly find a substantial fraction of generated snippets non-compliant with basic secure-coding standards or triggering well-known weakness classes — buffer overflows, unchecked returns, hard-coded credentials, SQL and command injection, cryptographic misuse, path traversal, and improper input validation. There is a persistent gap between code that is functional and code that is functional and secure, and even explicit “write secure code” prompting does not close it reliably. Models also hallucinate non-existent third-party packages and recommend deprecated or vulnerable versions, and asking the model to fix its own bugs can introduce new ones. Every line of model-generated code bound for production needs human security review.

Automation bias, deskilling, and the aggregate verdict

Even under nominal supervision, fluent confident output triggers documented human-factors failures: automation bias (deferring to the machine) and complacency (vigilance decaying as the system seems reliable). When supervision lapses on tasks beyond the model’s competence, the human-plus-model pairing can underperform the human alone — the system turns dangerous not when the model errs, but when the reviewer abdicates. Over time, leaning on the model erodes the very expertise the reviewer needs, a deskilling trap in which the validator grows least competent exactly when the model is most likely to fail. The aggregate evidence converges: realized value comes from positioning the human well, not from removing the human.

Bainbridge’s “Ironies of Automation”

Lisanne Bainbridge’s 1983 paper “Ironies of Automation” predicted, decades early, every failure mode autonomous AI now exhibits. Its core irony: automating the routine and leaving the human to handle rare failures hands the human the hardest tasks — diagnosing anomalies, intervening in abnormal conditions — without the practice, skill, or context to do them, because those were exactly the skills the automation let atrophy. Humans cannot sustain vigilant monitoring of a highly reliable system for more than roughly half an hour, and cannot verify in real time a system designed to be faster and more complex than they are. Her conclusion: the more sophisticated the automation, the more critical — and more poorly supported — the residual human becomes. The validator architecture is her recommended alternative: an active, empowered decision-maker, not a passive monitor of a system they cannot keep up with.

Normal Accident Theory

Charles Perrow’s Normal Accidents (1984) argued that in systems with two properties — interactive complexity (many components interacting in nonlinear, unanticipated ways) and tight coupling (changes propagate fast, with no slack to intervene) — catastrophic failure is not an anomaly but a “normal,” expected feature of the design. Such accidents cannot be engineered away by adding safety devices, because each device adds complexity. An autonomous LLM agent with tool access is both interactively complex (model, data, tools, environment, and adversarial inputs interact unpredictably) and tightly coupled (it acts at machine speed with no human buffer) — Perrow’s archetype of a system that should be redesigned, not patched. The validator architecture is that redesign: it decouples inference from execution by inserting a human buffer and reduces complexity by limiting the model to one output type — suggestions.

The Knight Capital failure

On August 1, 2012, Knight Capital — then one of the largest market-makers in U.S. equities — suffered the canonical autonomous-system failure. A software deployment reached seven of eight order-routing servers; the eighth still carried a dormant 2003 routine (“Power Peg”) that a configuration-flag change inadvertently reactivated at the open. With no position-limit check and no effective kill switch, the system fired millions of unintended orders. In about 45 minutes it executed some four million trades across 154 stocks and accumulated billions in unwanted positions, producing a pre-tax loss of roughly $440 million — more than the firm’s annual profit, and effectively the end of the company. The SEC’s settlement noted the absence of adequate controls to detect and halt the abnormal behavior in real time.

Source: SEC settlement order, Knight Capital Americas LLC (2013); contemporaneous reporting. Figures approximate as commonly reported.

The Frame Problem

The Frame Problem, identified by McCarthy and Hayes (1969) and popularized by Dennett, is the problem of determining which facts are relevant to a decision and which can be ignored. In any real situation the number of potentially relevant facts is combinatorially explosive — direct effects, indirect ramifications, preconditions that might block an action, and the vast set of things that stay unchanged. A model asked to recommend, say, a contract clause would in principle need to weigh jurisdiction, counterparty condition, regulatory environment, strategic goals, board risk appetite, litigation precedent, and a thousand other variables — most irrelevant to any given case, with no reliable way to know which. The human reviewer solves this not by computing every possibility but by knowing, from context and experience, which facts matter here. The human supplies the frame.

What medicine teaches about validation

Medicine offers the clearest illustration of why human validation matters and why the worst errors are the ones you never see. Where clinicians were required to review AI recommendations, unsafe suggestions were caught and the systems corrected or withdrawn — “loud” failures, surfaced by human oversight. The greater danger is the silent failure: if a system misclassifies a serious condition as benign and the patient is never re-seen, the error can go undetected indefinitely, with no way to trace harm back to the AI decision absent continuous human oversight and outcome monitoring. The lesson generalizes: AI should function as an assistant that surfaces uncertainty and prompts reconsideration, never as an unmonitored authority — because the errors that look correct are the ones that do the most damage.

Formal verification and self-critique have hard limits

Two technical approaches are often floated as ways to remove the human. Formal verification (machine-checked proofs of reasoning steps) is genuinely useful, but tractable only in narrow, formalizable domains like mathematics; it does not generalize to legal, strategic, medical, or policy judgment, it adds computational cost, and a valid proof about a wrong model of reality is still wrong. Self-critique (the model checking its own output against stated principles) helps, but the model is still grading its own work against its own internal representation of the rules — if that representation is flawed or the principles incomplete, the loop systematically misses errors a human would catch. Both narrow the problem; neither supplies a general external check. The human validator remains the only general-purpose verifier across the full range of business, legal, and strategic criteria.