tts-chapters: multi-voice audiobook pipeline
A Python pipeline that turns a novel chapter into narrated audio, attributing each line of dialogue to a speaker and giving every character its own voice.
Overview
tts-chapters takes a chapter of plain prose and produces a single loudness-normalized MP3 or WAV in which the narrator and each character speak in distinct voices. It runs as a CLI or as a desktop or web GUI built with Flet, with one tab per pipeline stage so attribution and voices can be checked and edited before rendering. Synthesis runs locally through the native audio.cpp engine, and an OpenAI-compatible LLM endpoint (a local server works) handles the ambiguous cases. Source not public.
Pipeline
| Stage | Implementation | Purpose |
|---|---|---|
| Parse | Sentence and quote splitter | Separates narration from dialogue into script units |
| Attribute | Rule layer, then LLM refiner | Assigns a speaker to every dialogue line |
| Voice | VoiceManager with a JSON voice registry |
Reuses, designs or falls back to a voice per character |
| Synthesize | audio.cpp subprocess, batched requests | Generates audio for every unit |
| Assemble | pyloudnorm, NumPy, soundfile | Normalizes loudness, adds gaps, concatenates |
Technical deep dive
Speaker attribution in two tiers. Most dialogue can be attributed cheaply. A deterministic layer matches explicit dialogue tags against a character registry of canonical names, aliases and descriptive patterns, each with its own confidence weight. Only lines it is unsure about go to the LLM, along with the known characters, a short window of surrounding narration and recent dialogue, so back-and-forth exchanges between two speakers resolve correctly. The registry feeds both tiers, so adding an alias improves both. New characters are registered automatically as they appear.
Voice design under a fixed vocabulary. A character without a voice gets one designed from an LLM-written description. The default model only accepts a comma-separated selection from a fixed set of tags (gender, age, pitch, accent, whisper), so the LLM is prompted with that exact vocabulary, and its reply is filtered against it, with a safe default if nothing valid remains. Designed voices and their reference clips are cached in the registry and reused across chapters. Descriptors are stored per backend.
Backend choice. I first used Python TTS packages, but two backends pinned conflicting
transformers versions in the same environment. I replaced them with audio.cpp, a C++/ggml engine
driven as a subprocess with no Python dependencies. The wrapper tracks capabilities per checkpoint
variant rather than per model family, because one family ships separate checkpoints that are each
locked to a single task (design, cloning or synthesis). The settings dialog can therefore never
offer a design-only checkpoint for synthesis.
Synthesis robustness. Consecutive units from the same speaker are merged, up to a character cap and never across scene breaks, so a paragraph is spoken as one utterance instead of stitched sentences. Lines are sent in batches with a timeout that scales with batch size. audio.cpp aborts a whole batch over one bad request, so a failed batch is bisected and each half retried. Isolating one bad line costs a logarithmic number of extra calls instead of falling back to one call per line. Cached voices are checked for a playable file before use, so a deleted reference clip is regenerated instead of silently dropping that speaker’s lines.
Key challenge: failures in TTS output are often quiet. A reference clip paired with a placeholder transcript instead of its real one made the engine leak stray words at clip boundaries. A missing cached voice file once removed the whole narration track with no error. Most of the work was tracing those artifacts back to mismatched state between the voice registry and the backend, then making the pipeline check that state rather than trust it.