The chain of a voice agent
A voice agent is a text agent with hearing and a voice added. Speech recognition turns the client’s words into text, the model decides what to answer and which tool to call, and speech synthesis turns the answer into a voice.
Every block exists in open form. Whisper, the speech recognition model from OpenAI, works with many languages, Russian among them; its code and weights are open under the MIT license.
A framework, a ready skeleton of a program, links the blocks. Pipecat, an open Python framework for real-time voice agents, comes under the BSD-2 license: it processes audio, calls the model and generates speech in one pipeline.
Each block adds its own delay and its own errors. If recognition mishears, the model will honestly answer a different question, so a voice agent’s log keeps both the audio and the transcript.
The model inside is the same as in a text agent: with instructions, tools and, if needed, a knowledge base. What the turnkey service can do is described on the page of the AI voice agent for inbound calls.
Delay and voice quality
Time is the main difficulty with voice. Stivers and co-authors (PNAS, June 2009) studied conversations in 10 languages: most often the next turn starts 0–200 milliseconds after the previous one.
In that time the agent has to see that the client has finished, recognize the phrase, get the model’s answer and start speaking. So the audio streams through the whole chain instead of waiting for each step to finish.
Turn-taking is a separate task: telling the end of a phrase from a pause when the client is thinking, and going quiet if the client interrupts. LiveKit Agents, an open framework under Apache 2.0, has end-of-turn detection for this.
Voice quality is tested on your calls, not on a demo: noise, accents, names of services, streets and people. Recognition that works well on a clean recording can confuse product codes and names over the phone.
Telephony and call recording
For the agent to take calls, it is connected to telephony. Usually this is SIP, the internet telephony protocol: the PBX passes the call to the agent as if to another employee.
LiveKit Agents takes and makes calls through a SIP integration. If you run your own PBX, you can use Asterisk, an open PBX and telephony toolkit under the GPL-2.0 license.
Transfer to a person works the way an experienced receptionist does it: the agent switches the call and briefly passes on the gist, so the employee doesn’t start the conversation over. For this, the PBX must be able to transfer a call on the agent’s command.
A call recording is needed to review errors and disputed cases: what the client said, what the agent answered, where it passed the call to a person. It is stored in the same place as the rest of the client data.
We don’t do outbound calling in our pilot: it is a different product with different risks. How telephony connects to our agents is covered in the integrations section.
Personal data in calls
A call almost always contains personal data: a phone number, a name, and in a clinic also health information. The call recording and its transcript contain it too.
Under 152-FZ, Russia’s personal data law, recording, organizing and storing the data of Russian citizens in databases outside the country is not allowed (Part 5, Art. 18). Since 1 March 2023, an operator must notify Roskomnadzor, the Russian data regulator, before transferring data abroad.
For a voice agent, this means that recognition, the model, synthesis and call recordings run on servers in Russia if client data passes through them. Cloud speech recognition abroad does not fit such calls.
Decide two things about call recordings in advance: how long to keep them and which employees may listen to them. The fewer people with access, the lower the risk of a leak.
How we store data and access is described on the page about data security.
Build or order
For a trial on your own, there are open frameworks: Pipecat under BSD-2 and LiveKit Agents under Apache 2.0. A Python developer can build a browser conversation on them and show how the agent hears and answers.
A working line is still far off: connection to the PBX, delay on a real network, recognition over the phone, call transfer to an employee, writing to the CRM, storage under 152-FZ. This is team-level work, not a beginner’s project.
Our voice agent pilot on inbound calls: 14 days, from $1,499, with the metric in the contract: the share of calls the agent takes without a person, and the number of missed calls.
A typical voice agent is also built from open blocks, but it takes more work to connect and test than a text agent. If someone charges for it as a unique development, ask what the price is made of, and pay for a result by the metric, not for hours.
How to compare building it yourself and ordering it, for any type of agent, is covered in the decision table.