# VoiceCraft — Zero-Shot Speech Editing and Text-to-Speech

> A neural codec language model for zero-shot speech editing and text-to-speech that can seamlessly edit spoken audio or generate speech that matches a reference speaker with only a few seconds of audio.

## Install

Save as a script file and run:

# VoiceCraft — Zero-Shot Speech Editing and Text-to-Speech

## Quick Use
```bash
git clone https://github.com/jasonppy/VoiceCraft.git
cd VoiceCraft
conda create -n voicecraft python=3.9 && conda activate voicecraft
pip install -r requirements.txt
python inference_speech_editing.py --orig_audio ref.wav --orig_transcript "original text" --target_transcript "edited text"
```

## Introduction
VoiceCraft is a neural codec language model developed at UT Austin that achieves state-of-the-art performance in both zero-shot speech editing and zero-shot text-to-speech. Given a short audio reference, it can edit words within existing speech or generate entirely new speech that sounds like the reference speaker, making it a versatile tool for audio post-production and voice cloning.

## What VoiceCraft Does
- Edits spoken words in audio recordings while preserving speaker identity and prosody
- Generates new speech from text that matches a reference speaker's voice
- Requires only a few seconds of reference audio for zero-shot voice cloning
- Handles both insertion and replacement edits in existing recordings
- Produces natural-sounding results with smooth transitions at edit boundaries

## Architecture Overview
VoiceCraft is built on a token-based language model that operates over neural audio codec tokens (from Encodec). It uses a novel token rearrangement procedure during training that teaches the model to handle both generation and editing in a unified framework. For speech editing, the model conditions on the original audio tokens and generates replacements for the edited segment. For TTS, it autoregressively generates all tokens conditioned on a short prompt.

## Self-Hosting & Configuration
- Requires Python 3.9+ and a CUDA GPU with at least 16 GB VRAM
- Install dependencies via pip and download pre-trained model checkpoints
- Provide reference audio and its transcript for speech editing tasks
- Configure generation parameters like temperature and top-p sampling
- Supports both CLI scripts and a Gradio web UI for interactive use

## Key Features
- Unified model handling both speech editing and text-to-speech generation
- Zero-shot voice cloning from as little as 3 seconds of reference audio
- Seamless edit boundaries that sound natural without audible artifacts
- Multiple model sizes available trading off quality and inference speed
- Gradio-based demo interface for easy experimentation

## Comparison with Similar Tools
- **Bark** — Generates expressive speech with non-verbal sounds, but lacks VoiceCraft's speech editing capability
- **VALL-E** — Pioneered zero-shot TTS with codec tokens, but VoiceCraft extends the approach to speech editing
- **so-vits-svc** — Focused on singing voice conversion, while VoiceCraft handles speech editing and TTS
- **OpenVoice** — Specializes in cross-lingual voice cloning, while VoiceCraft excels at in-context editing

## FAQ
**Q: What is speech editing versus TTS?**
A: Speech editing modifies specific words in an existing recording while keeping everything else intact. TTS generates entirely new audio from text.

**Q: How much reference audio is needed?**
A: As little as 3 seconds of clean speech from the target speaker, though more reference audio generally improves quality.

**Q: Can it handle multiple languages?**
A: The released models are primarily trained on English speech. Community efforts are adapting it for other languages.

**Q: Is the output suitable for production use?**
A: The quality is suitable for many professional applications, though edge cases with very different speaking styles may require fine-tuning.

## Sources
- https://github.com/jasonppy/VoiceCraft
- https://jasonppy.github.io/VoiceCraft_web/

---
Source: https://tokrepo.com/en/workflows/asset-b4d25dcf
Author: Script Depot