VoiceCraft — Zero-Shot Speech Editing and Text-to-Speech

Introduction

VoiceCraft is a neural codec language model developed at UT Austin that achieves state-of-the-art performance in both zero-shot speech editing and zero-shot text-to-speech. Given a short audio reference, it can edit words within existing speech or generate entirely new speech that sounds like the reference speaker, making it a versatile tool for audio post-production and voice cloning.

What VoiceCraft Does

• Edits spoken words in audio recordings while preserving speaker identity and prosody
• Generates new speech from text that matches a reference speaker's voice
• Requires only a few seconds of reference audio for zero-shot voice cloning
• Handles both insertion and replacement edits in existing recordings
• Produces natural-sounding results with smooth transitions at edit boundaries

Architecture Overview

VoiceCraft is built on a token-based language model that operates over neural audio codec tokens (from Encodec). It uses a novel token rearrangement procedure during training that teaches the model to handle both generation and editing in a unified framework. For speech editing, the model conditions on the original audio tokens and generates replacements for the edited segment. For TTS, it autoregressively generates all tokens conditioned on a short prompt.

Self-Hosting & Configuration

• Requires Python 3.9+ and a CUDA GPU with at least 16 GB VRAM
• Install dependencies via pip and download pre-trained model checkpoints
• Provide reference audio and its transcript for speech editing tasks
• Configure generation parameters like temperature and top-p sampling
• Supports both CLI scripts and a Gradio web UI for interactive use

Key Features

• Unified model handling both speech editing and text-to-speech generation
• Zero-shot voice cloning from as little as 3 seconds of reference audio
• Seamless edit boundaries that sound natural without audible artifacts
• Multiple model sizes available trading off quality and inference speed
• Gradio-based demo interface for easy experimentation

Comparison with Similar Tools

• Bark — Generates expressive speech with non-verbal sounds, but lacks VoiceCraft's speech editing capability
• VALL-E — Pioneered zero-shot TTS with codec tokens, but VoiceCraft extends the approach to speech editing
• so-vits-svc — Focused on singing voice conversion, while VoiceCraft handles speech editing and TTS
• OpenVoice — Specializes in cross-lingual voice cloning, while VoiceCraft excels at in-context editing

FAQ

Q: What is speech editing versus TTS? A: Speech editing modifies specific words in an existing recording while keeping everything else intact. TTS generates entirely new audio from text.

Q: How much reference audio is needed? A: As little as 3 seconds of clean speech from the target speaker, though more reference audio generally improves quality.

Q: Can it handle multiple languages? A: The released models are primarily trained on English speech. Community efforts are adapting it for other languages.

Q: Is the output suitable for production use? A: The quality is suitable for many professional applications, though edge cases with very different speaking styles may require fine-tuning.

Sources

• https://github.com/jasonppy/VoiceCraft
• https://jasonppy.github.io/VoiceCraft_web/