What Building a Voice Agent Taught Us About Conversation Design and Testing

Over the past three months, we have been developing and piloting a voice agent designed to conduct structured conversations at scale. So far, around 80 to 100 participants have completed 200 to 250 conversations, each typically lasting two to three minutes. In the longer term, the solution is intended to support up to 20 000 conversations per year.

The goal is straightforward: automate a process that would otherwise require significant human capacity, while still collecting information consistently through a natural spoken interaction.

What looked relatively simple on paper quickly became a broader engineering and conversation design challenge. The difficult part was not getting the agent to speak. It was making sure the conversation remained reliable when real users hesitated, interrupted, changed direction or responded in ways we had not anticipated.

Voice Agent Is Not Just a Chatbot That Speaks

It is tempting to approach a voice agent as a chatbot with an additional speech layer. Spoken conversations, however, behave very differently from text-based interactions.

Users hesitate, correct themselves, interrupt the agent, provide information earlier than expected or combine several ideas in one answer. Timing also matters much more. A delay that goes unnoticed in chat can feel uncomfortable in a spoken interaction, while a response that reads well on screen may feel unnecessarily long when spoken aloud.

We saw this clearly during the pilot. One seemingly small parameter, how long the agent should wait before interpreting a pause as the end of a user's turn, proved highly specific to the use case. Interrupt too quickly and the agent cuts users off. Wait too long and the interaction starts to feel unnatural.

The challenge was further complicated by the fact that the agent had to follow a predefined dialogue structure. It needed to keep the conversation on track while still responding naturally to what users actually said.

That meant we could not evaluate the solution solely by checking whether it produced the expected response. We also had to assess whether the conversation as a whole remained natural, coherent and aligned with the intended flow.

Real Users Create Scenarios You Cannot Fully Predict

The pilot quickly surfaced conversation patterns that would have been difficult to anticipate through internal testing alone.

Some users answered very concisely, while others gave much longer explanations. Information appeared earlier or later in the conversation than expected. Users changed their answers halfway through a sentence, added context that had not been requested or responded in ways that required the agent to decide how closely to follow the predefined script.

For us, these were not simply edge cases to eliminate. They became direct input for improving the dialogue logic.

We adjusted how the agent followed the script, how it responded to unexpected input and how it continued the conversation without losing the intended structure. We also refined interruption timing based on actual user behavior rather than assumptions made during development.

Following these iterations and another round of testing, around 80% of participants were able to complete all dialogue tasks end to end.

That does not mean the solution is finished. The project is still in its pilot phase. But the result gives us a much clearer indication of where the dialogue already performs reliably and where further refinement is needed.

Turning Pilot Insights into Automated Tests

The pilot produced more than qualitative feedback. It also generated concrete scenarios that we could incorporate into our testing process.

Whenever a relevant unexpected dialogue path emerged, we analyzed it, determined the desired behavior and translated it into a repeatable test where possible.

This created a practical feedback loop: real conversations revealed new scenarios, those scenarios informed improvements to the dialogue, and automated regression tests helped protect the resulting behavior.

The project now contains around 40 to 50 automated tests covering important dialogue paths and previously observed scenarios.

This matters because manually repeating every possible conversation after each modification is neither scalable nor cost-effective. Human testing remains essential for judging whether an interaction feels natural, but automated testing allows us to repeatedly verify known behavior as prompts, dialogue logic and supporting components evolve.

The pilot therefore did not replace automated testing. It made our automated testing more representative of how people actually interact with the voice agent.

From a Controlled Pilot to a More Reliable Solution

One of the main lessons reinforced by this project is that a successful voice agent is not created by trying to predict every possible conversation in advance.

We started with a defined dialogue and validated the most important paths internally. We then introduced the solution to real users through a controlled pilot, observed how they interacted with it and used those conversations to improve both the dialogue and the test suite.

Three months into the pilot, 80 to 100 participants have completed 200 to 250 conversations, the solution is supported by 40 to 50 automated tests, and around 80% of participants in a follow-up test were able to complete all dialogue tasks end to end.

There is still work ahead before the solution can support its intended longer-term scale of up to 20 000 conversations per year. But the pilot is already providing something we could not get from internal testing alone: real conversation patterns that can be translated into concrete improvements and repeatable tests.

That is helping us move from a working voice agent towards one that can operate more reliably at scale.

‍