Open-LLM-VTuber — Build Your Own AI Companion with Live2D and Voice

Introduction

Open-LLM-VTuber lets developers and creators build AI characters that listen, think, and speak with an animated Live2D avatar. It connects any LLM backend to speech recognition and text-to-speech engines, producing a real-time conversational experience rendered through expressive 2D character models.

What Open-LLM-VTuber Does

• Drives Live2D character animations synchronized with AI-generated speech
• Supports multiple LLM backends including OpenAI, Ollama, and local models
• Integrates speech-to-text engines for real-time voice input from microphone
• Connects to various TTS providers for natural-sounding AI voice output
• Provides a web-based interface for interacting with the AI companion in a browser

Architecture Overview

The system runs as a Python server with a web frontend. Audio from the user is captured by the browser, transcribed via a configurable ASR engine (Whisper, Faster Whisper, or cloud APIs), and sent to the chosen LLM for response generation. The LLM output is streamed to a TTS engine, and the resulting audio is synchronized with Live2D lip-sync and expression data using a motion mapping layer. The frontend renders the Live2D model via PixiJS and plays audio in sync.

Self-Hosting & Configuration

• Clone the repo and install Python dependencies from requirements.txt
• Configure your preferred LLM provider in conf.yaml (OpenAI, Ollama, or any OpenAI-compatible API)
• Set up a TTS engine such as Edge TTS, VITS, or ElevenLabs in the config
• Choose an ASR provider for voice input (Whisper, Faster Whisper, or cloud services)
• Place Live2D model files in the models directory and reference them in the config

Key Features

• Modular architecture lets you swap LLM, TTS, and ASR components independently
• Real-time lip-sync and expression mapping for natural character animation
• Supports both local and cloud AI backends for flexibility in deployment
• Browser-based interface requires no client installation
• Extensible persona system for defining character personality and behavior

Comparison with Similar Tools

• SillyTavern — focuses on text-based character chat; Open-LLM-VTuber adds voice and animated avatar interaction
• AI Vtuber — typically relies on specific platforms; Open-LLM-VTuber is fully self-hosted and provider-agnostic
• GPT-SoVITS — specializes in voice cloning; Open-LLM-VTuber integrates voice with visual avatars and conversation
• Character.AI — proprietary cloud service; Open-LLM-VTuber runs entirely on your own infrastructure

FAQ

Q: Which LLMs work with Open-LLM-VTuber? A: Any model accessible via OpenAI-compatible API, including Ollama for local models and cloud providers.

Q: Can I use my own Live2D model? A: Yes. Place your model files in the models directory and update the configuration to reference them.

Q: Does it work offline? A: Yes, if you use local models for LLM (Ollama), TTS (VITS), and ASR (Whisper).

Q: What are the hardware requirements? A: A modern CPU handles the pipeline. GPU is recommended only if running local LLM or TTS models.

Sources

• https://github.com/Open-LLM-VTuber/Open-LLM-VTuber