TTS Model Fixes Throat Singing Bug, Improves 50%

While OpenAI’s Whisper dominates speech recognition, Coqui AI’s XTTS v2 has carved out territory in the opposite direction: generating natural-sounding speech from text. The latest v2.0.3 release addresses a peculiar bug that caused the model to produce throat singing-like artifacts and delivers a claimed 50% performance improvement across multiple languages.

Key Specs

XTTS v2.0.3 operates as a multilingual text-to-speech system supporting 17 languages, including English, Spanish, French, German, Italian, Portuguese, Polish, Turkish, Russian, Dutch, Czech, Arabic, Chinese, Japanese, Hungarian, Korean, and Hindi. The model requires just 6 seconds of reference audio to clone a voice, significantly less than earlier iterations that needed 30+ seconds.

The throat singing bug emerged when users processed certain phonetic combinations in Turkic and Mongolian languages. The model would generate harmonic overtones characteristic of traditional throat singing rather than standard speech patterns. Engineers traced the issue to an overactive formant synthesis layer that misinterpreted certain vowel sequences as intentional harmonic requests.

Performance improvements stem from three architectural changes. First, the team implemented mixed-precision inference, allowing the model to use 16-bit floating-point operations where full 32-bit precision isn’t necessary. Second, they optimized the attention mechanism in the transformer backbone, reducing computational overhead by 23%. Third, a new caching system stores frequently-accessed phoneme embeddings, eliminating redundant calculations during batch processing.

The model runs on GPUs with at least 8GB VRAM for real-time synthesis. CPU inference remains possible but operates at roughly 0.3x real-time speed on modern processors. The complete model weights clock in at 1.8GB, making it deployable on edge devices with sufficient memory.

Who Benefits

Content creators producing multilingual videos gain the most immediate value. A single voice sample can now generate narration across all 17 supported languages with consistent vocal characteristics, eliminating the need for multiple voice actors or separate recording sessions for international markets.

Accessibility developers building screen readers and assistive technologies benefit from the reduced latency. The 50% speed improvement brings synthesis closer to real-time requirements for interactive applications, particularly important for users who rely on audio feedback for navigation.

Game developers working on character dialogue systems can leverage the improved voice cloning for rapid prototyping. Instead of booking voice talent for early development builds, teams can generate placeholder dialogue that maintains character consistency across thousands of lines.

Researchers studying phonetics and language acquisition now have access to a model that handles edge cases more reliably. The throat singing fix demonstrates improved handling of complex phonetic environments, making the system more suitable for linguistic analysis tools.

Quick Start

Installation requires Python 3.9 or newer and PyTorch 2.0+. The package installs via pip:

pip install TTS==0.22.0

from TTS.api import TTS

# Initialize model
tts = TTS("tts_models/multilingual/multi-dataset/xtts_v2")

# Generate speech from text with voice cloning
tts.tts_to_file(
    text="Neural networks transform text into natural speech patterns.",
    speaker_wav="reference_audio.wav",
    language="en",
    file_path="output.wav"
)

The reference audio should be clean speech without background noise, ideally 6-10 seconds long. WAV format at 22050 Hz sample rate produces optimal results, though the model accepts various audio formats through automatic resampling.

For production deployments, the team recommends running the model behind an API server. The official Coqui TTS repository at https://github.com/coqui-ai/TTS includes FastAPI integration examples that handle request queuing and GPU memory management.

Alternatives

Bark from Suno AI offers comparable multilingual support with additional capabilities for non-speech sounds like laughter and music. However, it requires significantly more computational resources and produces less consistent voice cloning results with short reference samples.

Eleven Labs provides a commercial API with superior voice quality for English but limited language support and higher per-character pricing. The service excels at emotional expression but lacks the local deployment option that XTTS v2 provides.

Meta’s Voicebox demonstrates stronger zero-shot capabilities but remains unavailable for public use due to safety concerns. Published benchmarks show similar quality to XTTS v2 on standard metrics, though direct comparisons remain difficult without access to the model weights.

StyleTTS 2 achieves competitive quality for English synthesis with lower computational requirements but supports only single-language deployment. Teams requiring multilingual output would need to maintain separate model instances for each language.