Pack TTS + STT

What's in this pack

This is the components catalog for voice apps. Where the Voice AI Stack pack gives you the realtime substrate (LiveKit, Moshi, OpenAI Realtime, Zonos) for speech-to-speech agents, this pack gives you the discrete STT and TTS engines you compose into the classic cascade architecture: microphone → STT → LLM → TTS → speaker.

The cascade is not dead. It's the right call when you need:

• Precise control over the LLM step — tool calls, structured output, RAG retrieval, anything where you need to inspect or transform the text.
• Cost-sensitive workloads at scale — speech-to-speech models are still 3-5x more expensive per minute than a well-tuned cascade.
• Non-realtime use cases — transcription pipelines, audiobook generation, podcast post-production, voiceovers for video. Latency is not the constraint.
• Self-hosted or air-gapped deployments — every component here has an open-source option you can run on your own GPU or even CPU.

Ten picks, grouped by layer:

Install in this order

# 1. Pick your STT first — it sets your latency floor
tokrepo install whisper-cpp           # local, CPU
# or
tokrepo install faster-whisper        # GPU, batch + streaming
# or
tokrepo install whisperx              # transcription with diarization

# 2. Add the TTS engine matched to your quality bar
tokrepo install elevenlabs-python-sdk # ship quality, pay per char
# or
tokrepo install coqui-tts             # self-host, decent quality
# or
tokrepo install kokoro                # lightweight, runs anywhere

# 3. Optional — voice cloning for branded narrators
tokrepo install openvoice

The TokRepo CLI drops a skill into your repo per asset. For Claude Code, Cursor, or Codex CLI the skills include working Python snippets and dependency lists; you wire them into your own app loop.

How the cascade actually fits together

[ Microphone / audio file ]
        │
        ▼
[ STT — Whisper variant ]
        │  text + word timestamps
        ▼
[ LLM — your choice ]
        │  reply text + tool calls
        ▼
[ Text normalizer ]
        │  numbers, dates, emoji stripped
        ▼
[ TTS — ElevenLabs / Coqui / Bark / Kokoro ]
        │  streaming audio frames
        ▼
[ Speaker / output file ]

A few things every shipping cascade does right:

1. Stream both ends. STT emits partial hypotheses every ~200ms; TTS emits audio after the first ~100ms of LLM output. Wire the LLM to stream tokens. End-to-end perceived latency drops from "send-then-wait" to "trickle".
2. Normalize before TTS. $1,234.56 reads as one-comma-two-three-four-point-five-six on most engines. A 20-line normalizer for currency, dates, abbreviations, and URLs is worth a week of "why does my agent sound dumb".
3. Cache the boot. Whisper-large takes ~3 seconds to load weights cold. Keep the model warm in a long-lived process; the first transcription should not pay this cost.

Tradeoffs you'll hit

• Whisper-large vs medium vs tiny. Tiny runs on a Raspberry Pi; large needs a GPU. Most production teams settle on medium plus VAD-aware chunking — it's the accuracy/cost knee. Faster Whisper makes large affordable; whisper.cpp makes tiny/base usable on CPU.
• ElevenLabs vs open-source TTS. ElevenLabs sounds noticeably better but costs $30-330/month plus per-character overages. Coqui + StyleTTS 2 reach "good enough for production" but require GPU. The cutoff: under 100k chars/day, run ElevenLabs; above, self-host.
• Bark vs Kokoro vs StyleTTS. Bark is expressive (laughs, music, effects) but slow and not always controllable. Kokoro is fast and tiny but neutral-sounding. StyleTTS 2 is human-level natural but needs the most VRAM. Match the engine to the artifact — Bark for game NPCs, Kokoro for IVR, StyleTTS for audiobooks.
• Voice cloning ethics. OpenVoice and ElevenLabs both support consent-based cloning. Always require explicit opt-in and log the consent. Unconsented cloning is the one easy way to lose a deal or a lawsuit.

Common pitfalls

• No VAD on the STT input. Sending continuous silence to Whisper produces hallucinated transcripts ("Thank you for watching!" is the famous one). Run a 30-line webrtcvad or silero-vad filter before Whisper. This single change kills the most common cascade bug.
• Sending the whole LLM reply to TTS at once. You're paying for the full LLM latency and the full TTS latency sequentially. Stream the LLM tokens into a sentence-buffer; flush a sentence to TTS the moment a . , ? , or ! arrives.
• Ignoring sample rate mismatches. Whisper expects 16kHz mono. TTS engines output 22.05/24/48 kHz. Resample at the boundaries; mismatched rates produce chipmunk or sub-bass artifacts that QA will blame on the model.
• Treating WhisperX as a drop-in for Whisper. WhisperX needs pyannote for diarization, which means a Hugging Face token and a license agreement. Plan the auth before you depend on it in production.
• Forgetting to log audio + transcript pairs. Voice apps regress silently — a TTS update or STT version bump can quietly degrade quality. Sample 1% of sessions, store the audio and transcript, and review weekly. Without this you'll only hear about regressions from angry users.