When AI Becomes a Toggle

AI rarely arrives with ceremony anymore. It appears in the release notes of software we already use: an assistant in the service desk, an agent inside the CRM, an automated reviewer in the development environment, or a recommendation engine in the finance workflow. Someone enables a feature, accepts a new licensing tier, or turns on a toggle.

The interface barely changes. The operating model does.

Two things stand out to me about this moment. First, embedded AI is expanding extraordinarily fast. Gartner predicts that up to 40 percent of enterprise applications will include task-specific AI agents by the end of 2026, up from less than 5 percent in 2025.[1] Many organizations will not encounter AI through one deliberate company-wide transformation. They will encounter it incrementally, application by application and workflow by workflow.

Second, what we are experiencing is not entirely new.

In 1983, psychologist Lisanne Bainbridge published a five-page paper called Ironies of Automation. It was written about industrial process control and flight-deck automation, not language models or enterprise software. Yet it may be one of the most useful things an operator can read today.[2]

Bainbridge’s central observation was deceptively simple: automation can expand rather than eliminate the problems left for human beings. My shorthand for it is this: automation does not necessarily remove complexity. It relocates it.

The work that remains

The conventional automation story is about subtraction. We automate routine activity, reduce manual effort and allow people to concentrate on higher-value work. This is often true. But it is incomplete because it says very little about the shape of the work that remains.

Bainbridge noticed that designers tended to automate the tasks they could describe and leave people responsible for everything they could not. The human operator was therefore left with an arbitrary collection of exceptions, abnormal conditions and situations the designer had not anticipated. That residual work was not easier. It was harder by definition.

The same pattern is emerging around embedded AI. An agent can summarize hundreds of customer interactions, classify routine support cases or prepare a standard response. The person who once worked through the entire queue may now see only the ambiguous complaint, the emotionally charged escalation or the case where several policies contradict one another.

A coding agent may handle familiar implementation work. The developer is brought in when the generated change behaves unexpectedly across several systems. A finance agent may reconcile ordinary transactions. The operator receives the handful that do not match, often because the business reality is unusual, poorly documented or genuinely disputed.

We should value the routine work that disappears. We should also recognize what is being concentrated in the human role: ambiguity, judgment, exception handling and accountability. AI may reduce average effort while increasing the difficulty and consequence of the moments that still require a person.

That is a very different operating proposition from saving time.

The takeover problem

Bainbridge understood that an operator cannot simply re-enter a process at the moment something goes wrong. Effective intervention depends on a working understanding of the system’s current state: what has happened, which possibilities have already been considered and how the present condition developed. That understanding accumulates while someone participates in the work.

When automation removes that participation, it also removes the opportunity to build context. This may be the defining problem of embedded AI.

An agent works across a series of records, messages, rules and applications. It reaches a point of uncertainty and hands the task to a human. The human receives the output, but not necessarily the journey.

Without those answers, escalation is not really a handoff. It is an interruption accompanied by missing context.

An Explain button will not be enough. Operators need a usable history of the agent’s actions, inputs, assumptions and changes. They need to reconstruct the situation quickly, distinguish facts from generated interpretations and see where their intervention can still matter. The handoff is part of the product, not an edge case.

The oversight fiction

There is another irony in asking people to supervise AI. Bainbridge pointed out that automation was introduced because it could perform certain tasks more consistently or quickly than an operator. Yet the operator was still expected to monitor whether it was performing those tasks correctly.

That becomes problematic when the automated system is processing more information, making decisions faster or using methods the person cannot follow in real time. Human oversight may still exist on the workflow diagram, but it may not exist in any meaningful operational sense.

We should be careful with the phrase human in the loop. A person who receives 200 AI-generated decisions and clicks approve is technically in the loop. Operationally, they may be little more than a confirmation step.

Review is itself work. It requires time, attention, evidence and a clear standard for determining whether an answer is acceptable. If reviewing a decision takes as much effort as producing it independently, people will eventually stop reviewing closely. If the AI is usually correct, vigilance will fall further.

This is not primarily a character flaw or a training problem. It is a predictable response to the design of the work. The important question is whether the person has the information, time, skill and authority required to change the outcome.

The erosion of operator skill

Automation also changes what people learn. When routine work disappears, so does the repetition through which people develop judgment. Experienced operators often recognize weak signals because they have seen hundreds of normal cases. They understand the written procedure and the rhythm of the system.

Bainbridge worried that early automated systems were effectively borrowing expertise from people who had first learned the work manually. The next generation of operators might be expected to supervise a system they had never learned to operate themselves.

That concern feels immediate now. What happens when a junior analyst learns to review AI-generated work but rarely constructs an analysis from the underlying data? What happens when a developer becomes skilled at accepting generated code without developing an equivalent ability to diagnose it? What happens when a service agent sees only escalations and never handles enough ordinary cases to understand what ordinary looks like?

The question is not whether people are becoming less intelligent. It is whether the work still gives them the experience required to develop the intelligence we expect them to contribute.

This leads to one of Bainbridge’s sharpest conclusions: the most successful automated systems, those that require human intervention least often, may require the greatest investment in human training. That sounds counterintuitive until we consider the job we have created. We are asking someone who intervenes rarely to perform exceptionally when they do.

When failure becomes less visible

Embedded AI introduces another operational risk: it can make degradation difficult to see. A traditional system often fails noisily. It stops, throws an error or refuses to complete the transaction. An AI system may continue producing fluent, plausible output while its performance deteriorates. The system can remain operational while becoming less reliable.

Bainbridge observed that automation could camouflage emerging problems by correcting around them until the situation was difficult to control. Embedded AI can create a cognitive version of the same effect. A polished summary can conceal missing evidence. A plausible classification can hide a changed business condition. A successful-looking workflow can accumulate small errors downstream.

Good AI operations must make uncertainty and failure visible. Systems need clear boundaries, traceable actions, meaningful alerts and graceful ways to stop. People must be able to inspect what the agent is doing before a weak signal becomes a consequential decision.

This is becoming a recognized industry problem. A 2026 NIST report on monitoring deployed AI systems identifies human-AI feedback loops, the burden of human monitoring and the difficulty of scaling oversight alongside rapid deployment as unresolved challenges.[3] More than forty years later, we are still confronting Bainbridge’s question: who monitors the automation, and what makes that monitoring humanly possible?

Questions before enabling the toggle

The practical lesson is not that we should resist embedded AI. Bainbridge was asking designers to treat the human and the technology as one operating system. That changes the questions we ask.

Before enabling an AI capability, I want to know what work will disappear and what work will remain. Will the remaining tasks be rarer, more ambiguous or more consequential? Who will receive them, and will that person still have enough contact with the underlying process to understand them?

I want the handoff designed before the happy path. What context will the operator receive? Can they see what the agent did, why it acted and what changed? Can they reverse its actions? How much time will they have?

I want to know how we will preserve capability. Which skills must remain available inside the organization even if they are used infrequently? Do people need deliberate manual practice, simulations or rotations through the underlying work?

I also want responsibility to match authority. If an operator remains accountable for an AI-mediated decision, that person must have a realistic ability to examine and alter it. Responsibility without visibility or control is liability assignment, not governance.

A must-read for the embedded AI era

Ironies of Automation deserves to be brought back into the light because it gives us language for what is happening now.

We are turning cognitive automation into a feature and distributing it through the software people already use. Because adoption can happen with a toggle, the organizational consequences can arrive without a corresponding redesign of work, training or accountability.

The danger is not simply that AI will make mistakes. Every operating system, including one made entirely of people, makes mistakes.

The deeper danger is that we will remove people from routine participation while continuing to depend on their judgment during exceptional moments. We will call them supervisors without giving them meaningful visibility. We will hold them accountable for processes they can no longer fully understand. We will interpret the presence of an approval button as evidence of control.

Bainbridge’s paper gives us a practical starting point. Ask what kind of human work the AI’s success will create. Ask what the operator will know at the moment of intervention. Ask whether the system is preserving the skills on which its resilience depends. Ask whether the person supposedly in control has any practical way to exercise it.

Embedded AI may arrive as a toggle. Its ironies will not.