Primary question: Does this OpenAI Realtime-compatible server remove the need for a hosted voice API while letting you keep swapping models per stage?
RepoDaily adoption score
RepoDaily rates this as 92/100 (strong) for adoption: evidence, installation path, production risk, differentiation, license clarity, and AI/agent fit are scored from the article sources and adoption notes.
5 source(s) across 3 source category/categories, plus a RepoDaily-specific evidence module when available.
6 workflow step(s), 5 next-action step(s), and 6 command/install signal(s) were detected.
Trending momentum is +837 stars, with maintenance/release/issue signals counted when present.
Risk is marked medium, with 5 security note(s) and 4 explicit skip condition(s).
3 opportunity lens item(s), 4 alternative(s), and 3 type-specific section(s) support differentiation.
License source or license wording is present.
8 AI/agent-related signal(s) were detected in the article text and metadata.
Project overview
huggingface/speech-to-speech is a Python voice-agent runtime that packages a four-stage pipeline — Voice Activity Detection, Speech-to-Text, a Large Language Model, and Text-to-Speech — behind a single WebSocket server. The README calls the cascade VAD -> STT -> LLM -> TTS, with each component running in its own thread and connected by queues. The whole stack is exposed through an OpenAI Realtime-compatible WebSocket API at ws://localhost:8765/v1/realtime, so any client built against OpenAI's Realtime surface can be pointed at a self-hosted instance.
The repository positions itself as fully modular. The LLM slot speaks OpenAI-compatible protocols, which means you can point it at a hosted provider, Hugging Face Inference Providers, or a local vLLM or llama.cpp server for an entirely on-device stack. The default install ships Parakeet TDT for STT, Qwen3-TTS for speech output, and Silero VAD v5 for turn detection. On macOS, TTS uses the mlx-audio backend; on Linux and Windows it uses the GGML backend by default.
The project claims real production weight: the README states the pipeline runs as the conversation backend for thousands of Reachy Mini robots. The package is published to PyPI as speech-to-speech, currently at version 0.2.11 with Development Status 3 - Alpha in its classifiers. The license is Apache 2.0, and the repository includes both a Dockerfile and a docker-compose.yml that pair the pipeline with a llama.cpp CUDA server running Gemma 4.
What makes this worth attention this week is the combination of OpenAI Realtime wire compatibility, per-stage swappability, and an explicit local-first default. Most voice-agent repos force you to choose a single STT or TTS vendor early. This one treats those choices as CLI flags and lets the LLM stay on your own GPU through a local OpenAI-compatible endpoint.
Why it is trending now
- 837 stars in the trending period with a rank of 5 signals concentrated interest in self-hostable voice runtimes, not just model releases.
- The README headline — a modular pipeline exposed as an OpenAI Realtime-compatible server — directly addresses teams trying to migrate off hosted Realtime endpoints.
- Production claim is concrete: the pipeline runs as the conversation backend for thousands of Reachy Mini robots.
- Default stack is fully local (Parakeet TDT, Qwen3-TTS, Silero VAD v5), so a single pip install plus an LLM endpoint gets a working voice agent without paid STT or TTS.
- Optional extras in pyproject.toml cover ChatTTS, faster-whisper, kokoro, paraformer, pocket-tts, WebRTC, and whisper-mlx, which widens the audience beyond a single model family.
Problem it solves
- Building a voice agent usually means stitching VAD, STT, LLM, and TTS libraries with custom queueing, streaming, and interruption handling.
- Hosted Realtime-style APIs create vendor lock-in and send every utterance — including audio — to a third party.
- Swapping one stage (for example, replacing Qwen3-TTS with kokoro) typically requires rewriting the conversation loop and the transport layer.
- Local-first stacks exist, but they rarely expose a wire protocol that existing Realtime-compatible clients already speak.
How it works
- Install with pip install speech-to-speech (Python 3.10+ required) and set OPENAI_API_KEY, or point the LLM backend at a local llama.cpp / vLLM server.
- Run speech-to-speech to start the WebSocket server at ws://localhost:8765/v1/realtime using Parakeet TDT for STT, an OpenAI-compatible LLM, and Qwen3-TTS for output.
- VAD (Silero VAD v5) detects speech boundaries and handles turn-taking; STT transcribes the user turn with optional live partials.
- The LLM stage streams text and tool calls back through the OpenAI-compatible responses API.
- TTS synthesizes audio and streams it back to the client. On macOS the backend is mlx-audio; on non-macOS it is the GGML backend.
- Connect any OpenAI Realtime-compatible client, or use scripts/listen_and_play_realtime.py --host 127.0.0.1 --port 8765 from a source checkout to talk to it locally.
Product demo and interface preview
Architecture read: four threads, one Realtime-shaped wire
The pipeline is a cascade of four components running in separate threads and connected by queues: VAD, STT, LLM, TTS. The README lists Silero VAD v5 for voice activity detection, STT that supports live partial transcripts, an LLM that streams text and tool calls, and TTS that synthesizes and streams audio back. The wire surface is an OpenAI Realtime-compatible WebSocket at ws://localhost:8765/v1/realtime.
The swappability is the architectural point. Every stage has multiple interchangeable backends selected via CLI flags. The LLM slot specifically speaks OpenAI-compatible protocols, so a hosted provider, Hugging Face Inference Providers, a vLLM server, or a llama.cpp server all look the same to the pipeline. The README's llama.cpp example serves Gemma 4 with: llama-server -hf ggml-org/gemma-4-E4B-it-GGUF -np 2 -c 65536 -fa on --swa-full, then points the pipeline at it with --responses_api_base_url http://127.0.0.1:8080/v1 and an empty API key.
The docker-compose.yml mirrors that design. It runs a ghcr.io/ggml-org/llama.cpp:server-cuda container with Gemma 4 and exposes port 8080, then runs the speech-to-speech pipeline with --llm_backend responses-api, --model_name ggml-org/gemma-4-E4B-it-GGUF, and --responses_api_base_url http://llama:8080/v1. The pipeline container exposes ports 12345 and 12346 for socket send/recv, and both containers request an NVIDIA GPU with device_ids: ['0'].
Deployment notes: Dockerfile, CUDA base, and the compose wiring
- Dockerfile base image is nvidia/cuda:12.8.1-cudnn-runtime-ubuntu24.04 — a GPU is assumed, not optional.
- The container installs libportaudio2, libsndfile1, python3, python3-pip, and python3-venv, then uses uv to sync dependencies from pyproject.toml.
- The Dockerfile runs nltk.download('punkt_tab') and nltk.download('averaged_perceptron_tagger_eng') at build time, which is required for sentence and POS handling.
- docker-compose.yml pairs the llama.cpp server (port 8080) with the pipeline container (ports 12345 and 12346), and both mount ./cache/ at /root/.cache/ to share model downloads.
- Default init prompt in compose is the string 'You are a helpful assistant' passed via --init_chat_prompt, with --init_chat_role set to system.
- The Dockerfile uses uv sync --python /usr/bin/python3 --no-dev, so dev dependencies (ruff, mypy, pytest, pytest-asyncio) are excluded from the runtime image.
Integration surface: CLI flags and optional extras
The entry point in pyproject.toml is speech_to_speech.s2s_pipeline:main, installed as the speech-to-speech script. The README shows the default invocation starting a realtime server; the compose file shows the socket mode with --mode socket, --recv_host 0.0.0.0, --send_host 0.0.0.0, and --llm_backend responses-api.
pyproject.toml defines optional dependency groups that double as feature toggles: chattts, facebook-mms, faster-whisper, kokoro, language-detection, mlx-lm, paraformer, pocket, webrtc, websocket, and whisper-mlx. Several are platform-gated — kokoro is explicitly excluded on macOS, whisper-mlx and the full mlx stack are macOS-only, and paraformer pulls in funasr, modelscope, and onnxruntime with a Python version constraint.
Core dependencies are heavy and specific: openai==2.28.0, transformers==5.6.2 on macOS and >=4.57.0 elsewhere, torch==2.11.0 on macOS and >=2.4.0 elsewhere, plus fastapi, httpx, uvicorn, websockets, and sounddevice. The numpy and soundfile pins differ between Darwin and non-Darwin, which matters for teams building multi-arch images.
Who should pay attention?
Good fit if
- Teams migrating OpenAI Realtime clients to a self-hosted endpoint without rewriting client code.
- Robotics and kiosk builders who need a local voice loop and can provide an NVIDIA GPU.
- ML engineers who want to benchmark different STT or TTS backends behind a stable conversation surface.
- Apple Silicon developers willing to use the mlx-audio and mlx-lm stack for a fully local voice agent.
Skip for now if
- CPU-only or low-memory servers — the Dockerfile assumes CUDA 12.8.1 and an NVIDIA device reservation.
- Teams that need a stable, non-Alpha API contract; the package is classified Development Status :: 3 - Alpha.
- Projects that require a single static binary or a non-Python embedding target.
- Use cases that need enterprise support, SLAs, or a long-term compatibility promise.
Risks and cautions
Apache-2.0, published to PyPI, and backed by a real production claim, but Alpha status, GPU defaults, and platform-divergent pins raise integration and upgrade risk.
- pyproject.toml classifies the package as Development Status :: 3 - Alpha, so APIs and flags can still change.
- The Dockerfile base is nvidia/cuda:12.8.1-cudnn-runtime-ubuntu24.04 and the compose file reserves an NVIDIA GPU, which excludes CPU-only hosts.
- Several dependencies are pinned differently on macOS versus other platforms (torch, torchaudio, transformers, numpy, sounddevice, soundfile), complicating multi-arch CI.
- Version 0.2.11 is recent, and the README references components like Gemma 4 and Qwen3-TTS that are themselves on fast release cadences.
- Production validation is tied to Reachy Mini robots; no public benchmark covers latency or accuracy on generic hardware.
- Apache 2.0 license with the LICENSE file shipped in the repository and referenced by pyproject.toml as license-files = ["LICENSE"].
- The server binds to localhost (ws://localhost:8765/v1/realtime) by default in the README quickstart, but docker-compose uses 0.0.0.0 for recv and send hosts — review network exposure before deploying.
- The llama.cpp local backend example passes an empty API key (--responses_api_api_key ""), which is correct for a local server but should not be reused against a hosted endpoint.
- No authentication layer is described in the README; any Realtime-compatible client that can reach the WebSocket can converse with the pipeline.
- Dependencies include openai==2.28.0, transformers, torch, and websockets — standard supply-chain review and pinning are appropriate before production use.
Alternatives to compare
| Approach | When to use | Trade-off |
|---|---|---|
OpenAI Realtime API (hosted) | You want zero infrastructure and accept sending audio to a third-party endpoint. | Per-minute usage pricing; no GPU to provision. |
| You already standardize on LiveKit for WebRTC transport and want a programmatic voice-agent framework. | Open source; self-hosted infrastructure required. | |
pipecat | You need a broader Python framework for multi-stage voice and video pipelines beyond the Realtime wire format. | Open source; you run the infrastructure. |
vLLM or llama.cpp (LLM only) | You already have STT and TTS wired and only need a local OpenAI-compatible LLM backend. | Open source; GPU recommended for usable latency. |
What this trend reveals
Realtime client migration tooling
Because the server speaks OpenAI Realtime, a thin migration shim that re-points existing clients at ws://localhost:8765/v1/realtime and records latency per stage could turn this into an audit tool for teams leaving the hosted API.
Run the default pip install plus a local llama.cpp server, then drive both endpoints with the same Realtime client and compare first-audio latency and cost.
Stage swap benchmark harness
The optional extras (faster-whisper, kokoro, paraformer, pocket, chattts, facebook-mms) make it cheap to benchmark different STT and TTS backends behind a fixed conversation loop.
Pick two STT extras and two TTS extras, hold the LLM constant, and measure end-to-end latency and audio quality on a fixed set of prompts.
Robotics and kiosk baseline
The README's claim that the pipeline runs behind thousands of Reachy Mini robots gives hardware vendors a reference architecture for on-device conversation.
Reproduce the docker-compose stack on a single NVIDIA device, then strip the compose file down to the minimum flags a kiosk image would actually need.
RepoDaily verdict
huggingface/speech-to-speech is a credible local-first voice-agent runtime: Apache-2.0, Realtime-compatible, modular by design, and backed by a concrete production claim. Treat it as Alpha infrastructure with real GPU and platform-divergence constraints, but it is one of the fastest ways today to stand up a self-hosted voice loop that existing OpenAI Realtime clients can talk to without a rewrite.