From Human in the Loop to Human Oversight

In today's world of agentic AI, the human is a surprisingly central topic. On the one hand, this is amazing. From a governance, legal and ethical perspective, the human needs to be in control, deciding, with enough information, if it's desirable to utilize an AI system. On the other hand, it's a disaster. From an operational perspective, a human can create a bottleneck and a false sense of safety without leveraging their most valuable contribution — information about how the system should operate. In this blog we will review the three biggest problems of human-in-the-loop and how the role of humans needs to change in order to capture the full benefit of agentic AI.

Human-in-the-loop as the catch-all patch

Human-in-the-loop is the most common and most widely used control to safeguard AI systems. It's positioned as a catch-all control to safeguard agents that are often poorly understood. Organisations reason that the system will be safe because the person that previously did the job will review the output. Therefore, the quality of delivery should stay the same. In addition, the system will bring operational efficiency because the person only has to review the output, which is faster than creating it.

This is not the way forward.

First of all, an agent with a human-in-the-loop is not as scalable as one without. Yes, review is faster than creating from scratch. Nevertheless, reviewing an output still takes time, and you need a skilled person to do so. This means there is a limited capacity in how many requests can be reviewed.

An agent on its own is scalable to the amount of compute that you have available. This has been growing exponentially over the last years and is unlikely to be a bottleneck for organisations leveraging AI for operational efficiency. The human review is scalable to the amount of skilled man-hours you have available. Training people with competences for tasks you are actively trying to automate is challenging. In doing so, the human review will become the bottleneck for the scalability of your agent.

Secondly, human-in-the-loop is positioned as a "safe by design" setup. The reasoning is: "A person has reviewed the outcome, so it will be correct." This belief ignores non-rational characteristics and heuristics that are common to our thinking and judgement. Humans have a tendency to trust automated processes — automation bias. In addition, complacency can kick in: if the agent has answered correctly ten times in a row, the assumption is that the eleventh time will be correct too. Add some time pressure and a few too many tasks, and clicking the "approve" button without any real review becomes the default action.

Even when human-in-the-loop is the best bet for a safe and scalable setup, in a lot of cases one important element is still missing: the feedback loop. A human is correcting the output of the agent, correcting that singular instance. But is this information also flowing back to the agent, so the system can be improved as a whole and make the human reviews less necessary over time?

Moving from human-in-the-loop to human oversight

To succeed with agentic AI, we need to move away from human-in-the-loop (as the catch-all solution) and move towards human oversight. This still means that people are involved and take accountability. To do so, humans need to understand the system as a whole, instead of a single prediction, and make a clear decision on whether the system meets the risk-reward trade-off. In some specifically selected settings, a human review on individual outcomes or cases could still be warranted.

Moving to human oversight means three things. First, a clear view on the agent's operational actions is needed. Every action the agent takes needs to be logged, so it can be tracked, audited and understood. This can then be compared to the agent's intended purpose to see if the system works.

Secondly, "unwanted actions" or unintended behaviour needs to be blocked without human involvement. The agent shouldn't take actions outside of its intended purpose, such as leaking data or hacking a competitor.[1] Tests are required to check how well the system is able to block these unwanted actions. This in turn will inform the risk management decision that the system is safe (enough) to be deployed in production.

Third, human-in-the-loop should be reduced to the most crucial moments where other options fail, either because unwanted actions are difficult to block or because the agent struggles with obtaining the correct output. This review needs to be triggered automatically, and the outcome of this review should feed back to the system to improve how it works.

Setting this up agent by agent, with custom logging and hand-written filters, quickly becomes a project of its own. A runtime governance layer such as Kyvvu makes it structural and scalable: every agent action is logged and checked against your policies in real time, safe actions go through, and unwanted ones are flagged or blocked before they happen. It's one of the reasons we recently partnered with Kyvvu.

Conclusion

In today's world we rely too much on human-in-the-loop in order to deploy agents. This blog argues this is undesired because of its limited scalability and its patchy risk mitigation. To succeed in the agentic world, the role of the human needs to evolve to that of oversight, deciding which systems get deployed in production, following up on whether agents still behave as anticipated, and only reviewing and correcting the outcomes in the most sensitive cases. This will improve the scalability, reliability and safety of our AI systems.

[1] Blocking unwanted actions can be done in many ways, such as architecturally, through prompts, machine-learning-based filtering or with deterministic filters in the harness.