From Human in the Loop to Human Oversight

In today's world of agentic AI, the human is a surprisingly central topic. On the one hand, this is amazing. From a governance, legal and ethical perspective, the human needs to be in control, deciding, with enough information, if it's desirable to utilize an AI system. On the other hand, it's a disaster. From an operational perspective, a human can create a bottleneck and a false sense of safety without leveraging their most valuable contribution — information about how the system should operate.

In this blog wewill review the three biggest problems of human-in-the-loop and how the role ofhumans needs to change in order to capture the full benefit of agentic AI.

Human-in-the-loop as the catch-all patch

Human-in-the-loopis the most common and most widely used control to safeguard AI systems. It'spositioned as a catch-all control to safeguard agents that are often poorlyunderstood. Organisations reason that the system will be safe because theperson that previously did the job will review the output. Therefore, thequality of delivery should stay the same. In addition, the system will bringoperational efficiency because the person only has to review the output, whichis faster than creating it.

This is not theway forward.

First of all,an agent with a human-in-the-loop is not as scalable as one without. Yes,review is faster than creating from scratch. Nevertheless, reviewing an outputstill takes time, and you need a skilled person to do so. This means there is alimited capacity in how many requests can be reviewed.

An agent on itsown is scalable to the amount of compute that you have available. This has beengrowing exponentially over the last years and is unlikely to be a bottleneckfor organisations leveraging AI for operational efficiency. The human review isscalable to the amount of skilled man-hours you have available. Training peoplewith competences for tasks you are actively trying to automate is challenging.In doing so, the human review will become the bottleneck for the scalability ofyour agent.

Secondly,human-in-the-loop is positioned as a "safe by design" setup. Thereasoning is: "A person has reviewed the outcome, so it will becorrect." This belief ignores non-rational characteristics and heuristicsthat are common to our thinking and judgement. Humans have a tendency to trustautomated processes — automation bias. In addition, complacency can kick in: ifthe agent has answered correctly ten times in a row, the assumption is that theeleventh time will be correct too. Add some time pressure and a few too manytasks, and clicking the "approve" button without any real reviewbecomes the default action.

Even whenhuman-in-the-loop is the best bet for a safe and scalable setup, in a lot ofcases one important element is still missing: the feedback loop. A human iscorrecting the output of the agent, correcting that singular instance. But isthis information also flowing back to the agent, so the system can be improvedas a whole and make the human reviews less necessary over time?

Moving from human-in-the-loop to human oversight

To succeed withagentic AI, we need to move away from human-in-the-loop (as the catch-allsolution) and move towards human oversight. This still means that people areinvolved and take accountability. To do so, humans need to understand thesystem as a whole, instead of a single prediction, and make a clear decision onwhether the system meets the risk-reward trade-off. In some specificallyselected settings, a human review on individual outcomes or cases could stillbe warranted.

Moving to humanoversight means three things. First, a clear view on the agent's operationalactions is needed. Every action the agent takes needs to be logged, so it canbe tracked, audited and understood. This can then be compared to the agent'sintended purpose to see if the system works.

Secondly,"unwanted actions" or unintended behaviour needs to be blockedwithout human involvement. The agent shouldn't take actions outside of itsintended purpose, such as leaking data or hacking a competitor.[1] Tests arerequired to check how well the system is able to block these unwanted actions.This in turn will inform the risk management decision that the system is safe(enough) to be deployed in production.

Third,human-in-the-loop should be reduced to the most crucial moments where otheroptions fail, either because unwanted actions are difficult to block or becausethe agent struggles with obtaining the correct output. This review needs to betriggered automatically, and the outcome of this review should feed back to thesystem to improve how it works.

Setting this upagent by agent, with custom logging and hand-written filters, quickly becomes aproject of its own. A runtime governance layer such as Kyvvu makes itstructural and scalable: every agent action is logged and checked against yourpolicies in real time, safe actions go through, and unwanted ones are flaggedor blocked before they happen. It's one of the reasons we recently partneredwith Kyvvu [add link to partnership article].

Conclusion

In today'sworld we rely too much on human-in-the-loop in order to deploy agents. Thisblog argues this is undesired because of its limited scalability and its patchyrisk mitigation. To succeed in the agentic world, the role of the human needsto evolve to that of oversight, deciding which systems get deployed inproduction, following up on whether agents still behave as anticipated, andonly reviewing and correcting the outcomes in the most sensitive cases. Thiswill improve the scalability, reliability and safety of our AI systems.

[1] Blockingunwanted actions can be done in many ways, such as architecturally, throughprompts, machine-learning-based filtering or with deterministic filters in theharness.

‍