HuggingFace speech-to-speech: Build Local Voice Agents with Open-Source Models

HuggingFace's speech-to-speech is #15 on GitHub's weekly trending chart (weekly growth of 3823 stars). As of 2026-08-06 it sits at 11,350 stars, 1402 forks, 132 open issues, 107 watchers, Apache-2.0 license, pure Python. The repo was created on 2024-08-07, with the most recent push today (2026-08-06). One-sentence pitch: build local voice agents with open-source models via a streaming VAD->STT->LLM->TTS pipeline that exposes an OpenAI Realtime-compatible WebSocket API, with every component swappable. It's already in production-as the conversation backend for thousands of Reachy Mini robots.

What It Is

A low-latency, fully modular voice-agent pipeline: VAD -> STT -> LLM -> TTS, exposed through an OpenAI Realtime-compatible WebSocket API at ws://localhost:8765/v1/realtime. Every component is swappable. Default config: local Parakeet TDT for STT, an OpenAI-compatible API for the LLM (default model gpt-5.4-mini), and local Qwen3-TTS for speech output. The LLM slot speaks OpenAI-compatible protocols, so you can point it at a hosted provider (OpenAI, HF Inference Providers, OpenRouter), a self-hosted server (vLLM, llama.cpp), or a local transformers / mlx-lm backend for a fully local, fully open stack. Any OpenAI Realtime-compatible client can connect-swap the base_url and you're done.

What Pain It Solves

Cloud voice agents (OpenAI Realtime and similar) hit three unavoidable pain points. First, cost: per-minute audio pricing adds up fast under heavy use. Second, latency: the audio-up-cloud-inference-audio-back round trip is hard to compress below 300ms, leaving the conversation feeling unsteady. Third, privacy: user audio leaves the machine, which directly violates compliance in healthcare, finance, and customer-service recording scenarios. Add vendor lock-in-the cloud provider changes the API, the pricing, or sunsets a model, and you're stuck. speech-to-speech's answer is to make all four components swappable and optionally fully local: Silero VAD v5 for VAD, Parakeet TDT default for STT, Qwen3-TTS default for TTS, and the LLM slot pointing at either a cloud provider or a local llama.cpp server. Want fully local? Point the LLM at local. Want convenience? Call OpenAI's API. Want data to never leave the machine? Run all four components on local open-source models. The protocol layer is OpenAI Realtime-compatible, so existing Realtime clients work without code changes-just swap the endpoint.

Core Capability: Open-Source Models for a Local Voice Agent (ASR + LLM + TTS Streaming)

Four stages, each running in its own thread, connected by queues:

1. VAD (Voice Activity Detection): Silero VAD v5 detects speech boundaries and turn-taking.
2. STT (Speech to Text, aka ASR): transcribes the user's turn, with optional live partial transcripts.
3. LLM: generates the response, streaming text and tool calls.
4. TTS: synthesizes audio and streams it back to the client.

What does "fully modular" buy you in practice? You can tune each component independently to hit your hardware budget and latency target. Latency-sensitive? Swap the LLM for a small quantized model and TTS for the lighter Kokoro-82M. Chinese-focused? Swap STT to Paraformer and TTS to ChatTTS. On a Mac? One flag (--local_mac_optimal_settings) shifts everything to the MLX ecosystem. Components are decoupled by queues, so swapping one doesn't touch the other three-this is the fundamental difference from a monolithic, black-box voice SDK.

Supported components (from the README):

Four run modes:

Other capabilities: Smart Turn v3.2 endpointing (uses the huggingface.co/pipecat-ai/smart-turn-v3 model, validates Silero's end-of-speech decisions using content and prosody to cut down on false cuts-where a mid-sentence pause gets misread as the user finishing-enabled only in --mode realtime, on by default, auto-downloads the quantized CPU model from HF Hub on first use); LLM Proxy (--enable_llm_proxy exposes the remote LLM as a plain OpenAI-compatible endpoint so a client can run side tasks like summaries, titles, or background agents concurrently with the voice conversation, never interrupted by new speech); multi-language (depends on STT/TTS backends-Parakeet TDT covers 25 European languages, Whisper broad multilingual, Paraformer Chinese-oriented by default, Qwen3-TTS multilingual with --qwen3_tts_language auto, ChatTTS English + Chinese; --language auto lets STT detect each segment's language and forward to the LLM); Pocket TTS (from Kyutai Labs, streaming TTS with voice cloning, 8 voice presets: alba, marius, javert, jean, fantine, cosette, eponine, azelma).

Three-Minute Setup

# 1. Install (Python 3.10+)
pip install speech-to-speech

# 2. Run (default LLM uses OpenAI Responses API, needs OPENAI_API_KEY)
export OPENAI_API_KEY=sk-xxx
speech-to-speech
# Starts OpenAI Realtime-compatible server at ws://localhost:8765/v1/realtime
# Defaults: local Parakeet TDT STT + OpenAI-compatible LLM + local Qwen3-TTS

# 3. From a second terminal, talk to it
python scripts/listen_and_play_realtime.py --host 127.0.0.1 --port 8765

Those three lines are the fastest path: local STT and TTS are already running, only the LLM goes through OpenAI's cloud. Speak into your mic, Parakeet TDT transcribes it, OpenAI generates a reply, Qwen3-TTS synthesizes it back as audio-a complete voice conversation loop. To keep the LLM on your own machine (no OpenAI key needed), serve Gemma 4 with llama.cpp:

# Terminal 1: llama.cpp serving Gemma 4
llama-server -hf ggml-org/gemma-4-E4B-it-GGUF -np 2 -c 65536 -fa on --swa-full

# Terminal 2: speech-to-speech pointed at the local LLM server
speech-to-speech \
    --model_name "ggml-org/gemma-4-E4B-it-GGUF" \
    --responses_api_base_url "http://127.0.0.1:8080/v1" \
    --responses_api_api_key ""

One-shot Mac local optimal settings:

speech-to-speech --local_mac_optimal_settings
# Equivalent to --device mps + Parakeet TDT + MLX LM + Qwen3-TTS (mlx-audio 6bit) + --mode local

Docker (requires NVIDIA Container Toolkit first):

docker compose up
# Starts llama.cpp + Gemma 4 + TCP socket server, exposes 8080/12345/12346

Who It's For + Pitfalls

For: teams building conversation backends for hardware devices (robots, smart speakers, kiosks)-already the production backend for thousands of Reachy Mini robots; scenarios where data cannot leave the machine (healthcare, finance, customer-service recordings); developers looking to avoid cloud voice API per-minute pricing; teams that want the OpenAI Realtime protocol without locking into OpenAI.

All seven pitfalls below come from explicit README warnings or install notes-knowing them upfront saves real debugging time. Seven pitfalls. First, Qwen3-TTS on Linux uses the GGML backend, and the default PyPI wheel targets CUDA 12.8 runtime-if your machine doesn't have the CUDA 12 runtime, install the matching wheel first: pip install "qwentts-cpp-python==0.3.1+cu130" -f https://huggingface.co/datasets/andito/qwentts-cpp-python-wheels/tree/main/whl/cu130 (CUDA 13.x), or +cu124 (CUDA 12.4), or +cpu (CPU-only fallback); Mac users go through mlx-audio and don't hit this. Second, the LLM is the latency bottleneck-README's words: "The LLM is the most compute-intensive and highest-latency component." A single forward pass through a large model can dominate end-to-end response time. For low latency, pick a small model (gpt-5.4-mini, Gemma 4 E4B, Qwen3-4B) or use a low-latency inference provider like Cerebras or Groq; for local, run a quantized model via llama.cpp. Third, macOS and Linux defaults differ (Qwen3-TTS uses GGML on non-macOS, mlx-audio on macOS), and the two paths have different bugs-cross-platform deployments need testing on both. Fourth, DeepFilterNet conflicts with Pocket TTS (DeepFilterNet needs numpy<2, Pocket TTS needs numpy>=2, can't coexist)-install DeepFilterNet manually only in environments where you're not using Pocket TTS. Fifth, the LLM Proxy is off by default, does no authentication, and does no throttling; the /v1/chat/completions or /v1/responses paths exposed by --enable_llm_proxy should only be enabled on a trusted network or behind a gateway that owns access control-the README warns this explicitly. Sixth, Smart Turn v3.2 is only supported in --mode realtime; pass --no_smart_turn when selecting another mode. Seventh, Direct Audio Input (bypassing STT to send VAD-cut audio segments straight to an audio-input model) only supports --llm_backend chat-completions, not responses-api, and the default gpt-5.4-mini doesn't accept audio-you must explicitly set --model_name to a model that does.

How It Compares

Only stating what the README can verify. vs OpenAI Realtime API: speech-to-speech implements the core OpenAI Realtime event set (inbound: input_audio_buffer.append, session.update, conversation.item.create, response.create, response.cancel; outbound: speech start/stop, streaming transcription, audio deltas, tool calls, response.done). The difference is that all four components are swappable, can be fully local, and can be fully open-source, while OpenAI Realtime is a cloud service tied to OpenAI models; any OpenAI Realtime client works after swapping the base_url with near-zero code changes, so migration cost is low-this protocol compatibility is its biggest engineering leverage. vs pipecat: the Smart Turn v3.2 endpointing model comes from huggingface.co/pipecat-ai/smart-turn-v3, so speech-to-speech reuses this model from the pipecat ecosystem, but the README doesn't position it as a pipecat replacement and makes no comparison-only the verifiable part is stated here. vs single-stage STT/TTS tools (Whisper.cpp, ChatTTS, Faster Whisper, Whisper MLX, etc.): speech-to-speech is an end-to-end pipeline that strings VAD+STT+LLM+TTS together and exposes a standard Realtime API, while single tools handle only one stage-in fact ChatTTS, Faster Whisper, and Whisper MLX are all optional backends of speech-to-speech, so the relationship is containment, not competition. Worth noting: speech-to-speech ships no hosted version-you deploy and operate it yourself, which is the fundamental division of labor versus a pure cloud service like OpenAI Realtime: one sells a service, the other sells a blueprint. One-liner: if you want the OpenAI Realtime-compatible protocol, full-local full-open-source, swappable four components, and a conversation backend for hardware devices, pick speech-to-speech.

References

• HuggingFace speech-to-speech GitHub repo (11,350 stars, Apache-2.0, Python): https://github.com/huggingface/speech-to-speech
• speech-to-speech README (sourced 2026-08-06): https://github.com/huggingface/speech-to-speech/blob/main/README.md
• Reachy Mini robot intro (production scenario mentioned in README): https://huggingface.co/blog/reachy-mini
• Smart Turn v3.2 model (endpointing): https://huggingface.co/pipecat-ai/smart-turn-v3
• Parakeet TDT STT model: https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3
• Qwen3-TTS model: https://huggingface.co/Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice