EmotiVoice — Multi-Voice Text-to-Speech with Emotion Control

Introduction

EmotiVoice is a text-to-speech engine developed by NetEase Youdao that goes beyond standard TTS by adding explicit emotion control to speech synthesis. Users can specify emotional tones like happy, sad, angry, or surprised, and the engine generates speech that naturally conveys those emotions across multiple built-in speaker voices.

What EmotiVoice Does

• Generates speech from text with controllable emotional expression
• Supports over 2000 built-in speaker voices for diverse output
• Handles both Chinese and English text input natively
• Provides emotion control through simple text prompts or labels
• Offers both batch processing and interactive web UI modes

Architecture Overview

EmotiVoice uses a two-stage architecture. The first stage converts text and emotion prompts into mel spectrograms using a modified Transformer model with emotion embedding layers. The second stage uses a HiFi-GAN vocoder to convert the spectrograms into high-fidelity waveforms. Emotion control is injected as a learned embedding concatenated with the text encoder output, allowing the model to modulate prosody, pitch, and energy according to the specified emotion.

Self-Hosting & Configuration

• Requires Python 3.8+ and a CUDA GPU for fast synthesis
• Install dependencies via pip and download pre-trained model weights
• Configure speaker ID and emotion via command-line arguments or config file
• Deploy the included Gradio web interface for interactive experimentation
• Supports Docker deployment for containerized production use

Key Features

• Fine-grained emotion control beyond what standard TTS systems offer
• Over 2000 pre-built speaker voices covering diverse demographics
• Bilingual support for Chinese and English in a single model
• Emotion can be specified as labels (happy, sad) or natural language prompts
• Fast inference on GPU with real-time factor well below 1.0

Comparison with Similar Tools

• Bark — Generates expressive audio with non-verbal sounds, but emotion control is less precise than EmotiVoice
• Coqui TTS — Broad model support and multilingual, but lacks EmotiVoice's explicit emotion embedding system
• ChatTTS — Optimized for conversational speech, while EmotiVoice focuses on controllable emotional expression
• Piper — Extremely fast and lightweight, but designed for neutral speech without emotion control

FAQ

Q: What emotions are supported? A: The model supports common emotions including happy, sad, angry, surprised, fearful, disgusted, and neutral, plus custom emotion prompts.

Q: Can I add custom voices? A: The released model includes over 2000 voices. Adding entirely new voices requires fine-tuning with target speaker data.

Q: Does it work in real-time? A: On a modern GPU, EmotiVoice achieves real-time synthesis speeds suitable for interactive applications.

Q: Is commercial use allowed? A: EmotiVoice is released under an open-source license. Check the repository for specific terms regarding commercial deployment.

Sources

• https://github.com/netease-youdao/EmotiVoice
• https://replicate.com/jasonppy/emotivoice