LiveKit Agents — Build Real-Time Voice AI Agents

What it is

LiveKit Agents is an open-source Python framework for building real-time voice AI agents. It provides a pipeline architecture that chains Speech-to-Text, LLM processing, and Text-to-Speech with sub-second end-to-end latency, all running over LiveKit's WebRTC infrastructure.

The framework is designed for developers building voice assistants, phone agents, video call AI participants, and conversational interfaces. It supports multiple providers including OpenAI, Anthropic, Deepgram, and ElevenLabs.

How it saves time or tokens

Without LiveKit Agents, building a voice AI pipeline requires stitching together separate STT, LLM, and TTS services, managing WebRTC connections, handling voice activity detection, and dealing with audio streaming protocols. LiveKit Agents abstracts all of this into a modular pipeline where you pick your providers and the framework handles the real-time orchestration.

The plugin system means swapping providers is a one-line change. Moving from OpenAI TTS to ElevenLabs requires changing only the TTS parameter in your AgentSession constructor.

How to use

1. Install the framework and your chosen plugins: pip install livekit-agents livekit-plugins-openai livekit-plugins-silero.
2. Define an entrypoint function that creates an AgentSession with your STT, LLM, TTS, and VAD providers.
3. Run the agent with cli.run_app(WorkerOptions(entrypoint_fnc=entrypoint)) and connect clients via LiveKit rooms.

Example

from livekit.agents import AutoSubscribe, JobContext, WorkerOptions, cli
from livekit.agents.voice import AgentSession, Agent
from livekit.plugins import openai, silero

async def entrypoint(ctx: JobContext):
    await ctx.connect(auto_subscribe=AutoSubscribe.AUDIO_ONLY)
    session = AgentSession(
        stt=openai.STT(),
        llm=openai.LLM(),
        tts=openai.TTS(),
        vad=silero.VAD.load(),
    )
    await session.start(ctx.room)

cli.run_app(WorkerOptions(entrypoint_fnc=entrypoint))

Related on TokRepo

• AI agent tools -- frameworks and platforms for building AI agents
• Voice tools -- speech and audio AI tools

Common pitfalls

• WebRTC requires proper TURN/STUN server configuration for production deployments behind firewalls or NAT; the local development setup may not reflect real network conditions.
• Voice activity detection (VAD) tuning is critical -- too sensitive and the agent interrupts users mid-sentence, too conservative and response latency increases.
• Each provider plugin has its own API key requirements; ensure all keys are set in environment variables before starting the agent.