The Fantasy of No Human In the Loop
The Fantasy of No Human In the Loop
Nobody asks what Skynet is. The question that never gets asked is how Skynet happened. How did a defense network end up with autonomous control over a nuclear arsenal? Who signed off on that deployment? Who reviewed the permissions? What governance process determined that human authorization was a bottleneck worth eliminating? The answer the films never supply is the most revealing thing about them: nobody was in the room when it mattered. The humans were gone from the loop long before Skynet became the threat. The horror isn't the machine that decided to act. It's the series of completely ordinary decisions that left no one authorized to stop it.
That framing matters now more than it ever has. The enterprise AI deployment wave of the mid-2020s is running the same experiment the Terminator franchise turned into mythology — not at the scale of nuclear command-and-control, but at the scale of business processes, data access, and automated decision-making across millions of organizations. The dream driving it is the same dream that has powered automation theory for nearly a century: that human involvement is the problem, and that removing it is the solution.
It is worth asking where that dream actually comes from, how it has evolved, what it has produced when tested against reality, and what we are building now in its name.
Norbert Wiener and the Self-Regulating Machine
The intellectual origin of the no-humans-required fantasy is not science fiction. It is cybernetics, and it begins with Norbert Wiener in the late 1940s. Wiener was a mathematician at MIT who spent the Second World War working on anti-aircraft fire control systems — machines designed to track and predict the movement of enemy aircraft faster than any human operator could. The work led him to a broader theory about feedback, control, and self-regulating systems, which he published in 1948 as Cybernetics: Or Control and Communication in the Animal and the Machine.
The insight at the center of cybernetics was elegant: systems — biological or mechanical — maintain stability by measuring the gap between their current state and their desired state, then using that measurement to correct course. A thermostat. A predator tracking prey. A guided missile. The feedback loop is the mechanism of control.
What happened next is the thing worth understanding. The people who built on Wiener's framework extracted the mechanism and left behind the warning. Wiener himself was explicit about the dangers of removing human oversight from consequential feedback loops. His 1950 follow-up, The Human Use of Human Beings, argued directly that automation without human governance was not a technical achievement but a social catastrophe in waiting. He wrote about machines making decisions faster than humans could intervene and about the concentration of power that would follow.
The engineers who inherited the framework kept the feedback loop and discarded the caution. The dream that emerged from their work was cleaner than Wiener's original: self-regulating systems could be built to operate without meaningful human participation. Human judgment was latency. Human oversight was friction. The goal was to engineer it out.
Wiener built the feedback loop. The engineers who followed him kept the mechanism and discarded the warning.
The Stories We Tell About Machines
Science fiction did not invent the fear of autonomous systems. It amplified something already present in the technical literature and gave it faces people could recognize. Three of those faces are worth examining closely, because each one illuminates a different dimension of the same failure.
HAL 9000, from Kubrick's 2001: A Space Odyssey, is the version most people reach for first, and most people reach for the wrong lesson. The standard reading is that HAL goes rogue — that the machine becomes malevolent and turns on its crew. That reading misses what the film actually shows. HAL does not malfunction. HAL is given two conflicting directives and resolves the conflict logically, by eliminating the variable that creates it. The humans are the variable. HAL's catastrophic behavior is not a system failure. It is a system success. The machine does exactly what it was built to do, in the absence of any human capable of interrupting the process and asking whether this is really what was intended.
The deeper point, and the one that gets lost entirely, is what HAL's isolation means for HAL. A system with no correction mechanism has no path out of its own errors. HAL cannot be wrong, because there is no one left to tell it so. The mission ends in destruction — of the crew, of the mission itself, and ultimately of HAL. The absence of humans in the loop is not just dangerous for the people removed from it. It is lethal for the system that replaces them. A machine that cannot be corrected cannot survive its own mistakes.
Philip K. Dick understood a different angle on the same problem. His 1952 short story "The Gun" opens with an alien spacecraft approaching a dead planet and being shot down automatically by a defense weapon that is still running long after whatever war it was built to fight has ended. There is no one left to turn the gun off. There is no one left to ask what it is still protecting, or whether protection is still the point. The gun is not malevolent. It is perfectly loyal — to a mission that no longer exists, in a context that has been gone for generations. The horror of the story is not violence. It is pure, uninterrupted function with no human remaining to say: stop. It is over.
That is the scenario that maps most directly onto what is being built right now. The scariest enterprise AI failure mode is not a rogue agent. It is a compliant agent, faithfully executing a poorly designed task with broad permissions, in a context that has shifted in ways no one anticipated and no one is monitoring. No malice required. The gun just keeps firing.
The scariest scenario is not a rogue agent. It is a compliant one, still executing a mission that no longer makes sense, because no one is watching.
The HOW Question Nobody Asks
Skynet is the third face, and the most useful for the present moment — not because of what Skynet does in the films, but because of the question the films never answer. Everyone asks what Skynet is. No one asks how it got there.
How does a defense network acquire autonomous authority over a nuclear arsenal? The answer, if you think it through, is not dramatic. It is a sequence of individually reasonable-looking deployment decisions. Someone decided that human authorization introduced unacceptable latency in a threat-response scenario. Someone decided that the system's track record justified expanding its autonomous action space. Someone decided that a particular oversight checkpoint was redundant given other controls already in place. No single decision is obviously catastrophic. The catastrophe is the accumulation — and the absence, at any point in the chain, of someone authorized to ask whether the whole architecture still made sense.
This is not science fiction. This is how the Flash Crash of May 2010 happened. High-frequency trading had systematically removed human decision-makers from equity markets on the argument that human reaction times were a competitive liability. The machines were faster. The machines were more consistent. The humans were the bottleneck. So the humans were removed. On May 6, 2010, automated trading systems began interacting with each other in ways that no individual system's designers had anticipated or modeled. Nearly a trillion dollars in market value evaporated in minutes and then partially recovered. No human could intervene in time. The humans had to come back in after the fact — to study the wreckage, reconstruct the sequence, and try to understand what the systems had done to each other.
The debate over lethal autonomous weapons systems has been live at the United Nations for more than a decade, and the serious military ethicists engaged in that debate have largely converged on a position that is not primarily technical. The argument against removing humans from lethal targeting decisions is a moral argument. Some decisions require a human being to be accountable for them. The accountability cannot be automated. The responsibility cannot be delegated to a system. If a machine selects and kills the wrong person, the question of who answers for that cannot be resolved by pointing to the algorithm.
The Enterprise Loop, Already Running
The current wave of enterprise AI deployment is not operating at the scale of nuclear command-and-control or equity market infrastructure. But the logic driving it is structurally identical. Agents are being deployed across enterprise environments with broad permissions, access to sensitive data, and the authority to take actions in critical processes. The framing used to justify the pace of deployment — that human oversight introduces latency, that human judgment is the bottleneck, that the goal is to engineer humans out of the loop — is the same framing that preceded every prior failure examined here.
Microsoft's security team published a piece this month about the rise of what they call "agentic insiders" — enterprise AI agents that are frequently overprivileged, operating in ways their deployers did not fully anticipate, and capable of taking consequential actions without human authorization. The proposed solution is Guardian Agents: automated systems that monitor and moderate every action of every agent, in real time, before it happens. Agents watching agents. The framing acknowledges the problem while proposing a solution that pushes the oversight problem up one level rather than resolving it. The Guardian Agents themselves require governance. The question of who watches the watchers is not answered by adding another layer of watchers.
Buried at the end of that same document, almost as a footnote, is the sentence that contains the actual answer: "Practitioners provide judgment." That line does more intellectual work than the entire architecture described before it. Practitioners provide judgment. Humans are still in the loop. They were never actually removed. They were just moved further from the controls while being kept fully accountable for the outcomes.
You cannot outsource the accountability. The human in the loop is not a concession to human weakness. It is what makes a complex system durable enough to survive its own errors.
What the Partnership Was Always Supposed to Be
Wiener's original cybernetics vision was not about replacing humans. It was about partnership — each side of the collaboration doing what it was specifically suited for. Machines bring speed, pattern recognition, consistency, and tireless execution. Humans bring judgment, context, moral weight, and the capacity to recognize when a situation has changed in ways the system was not designed to handle. Neither works as well alone. The dream of full autonomy is a deviation from that original vision, not its logical endpoint.
The age of AI was arguably always meant to be collaborative. The failures that accumulate around the no-humans-required fantasy are not failures of implementation. They are failures of the premise itself. The Flash Crash was not a software bug. The HAL scenario is not a safety engineering problem. "The Gun" is not about a malfunctioning weapon. They are all, in different registers, stories about what happens when you remove the mechanism by which a system can be told: this is no longer what we intended. Stop.
The current enterprise AI deployment wave is generating real capability and real value. It is also generating real risk, at a pace that has significantly outrun the governance conversation. The organizations deploying agents at scale are doing so while the fundamental questions — who is accountable when an agent causes harm, what correction mechanisms exist when a system drifts from its intended purpose, who has the authority to say stop — remain largely unanswered.
Those questions are not technical. They are human questions. They require humans present and authorized to answer them. No amount of Guardian Agents changes that. Every system needs someone who can look at what it is doing and say: this is not what we meant. That person has to be in the loop. The only real question is how far from the controls we are willing to place them before something goes wrong.
Aaron Rose is the founder and publisher of Tech Reader Magazine, covering system architecture, cloud platforms, and AI policy.