We have been tinkering with LLMs and building voice applications for the past few months. As we were trying to improve our voice bots performance (interruptions, latency of tts, llm etc) it became very tedious to test our bot over and over again speaking to it.

With real users we would observe edge cases that we didn’t handle before. After making changes it was difficult to recreate that exact scenario (accent, noise, slow speech or gaps between sentences) and test changes.

So we built a way to use any existing call recording and replay ‘user’ portion of the call to the bot. Since LLMs won’t repeat the exact text or follow same flow on each run, it became important to handle those deviations. So took sometime experimenting and came up with this approach:

1. Segment call in to audio snippets of dialogue 2. Create a transcript and understand flow, intent, details of the call. 3. In response to LLM, identify correct user audio snippet to replay based on the flow/context (lot of work here :D) 4. In cases where it is a new question or reconfirmation by bot, generate new text+audio.

I think this is pretty cool and would love for you to try it over your VoiceAI bots and give feedback to improve it.

Earlier we also built a visualisation for analysing audio calls (player, transcript and basic errors) and included it. Here is a quick demo of how both look: https://youtu.be/j3kRhSxD5P0

What’s next? Thinking of creating more voice based automated tests. Open to ideas and would love to know how you improve your VoiceBots.