Local speech-to-text behind an OpenAI-compatible API

voiced is a small gateway that serves speech-to-text on an Apple Silicon Mac through the same HTTP API as OpenAI's transcription endpoint. Transcription runs locally with whisper.cpp. Any client built on an OpenAI SDK can use it by changing its base URL.

The text-to-speech route is reserved in the API and returns HTTP 501 until it is implemented. This post covers the speech-to-text side: why a gateway is needed at all, how it manages models and memory, and where its limits are.

Why a gateway

whisper.cpp ships an HTTP server, whisper-server, and on a Mac it installs with Homebrew. Three properties of it shaped voiced, and the README lists them directly.

• whisper-server exposes /inference. OpenAI clients call POST /v1/audio/transcriptions with a different set of form fields.
• One whisper-server process serves one model. Offering two models means running two processes, and something has to start, stop and restart them.
• GPU acceleration through Metal needs native macOS execution. A Linux container on a Mac has no access to it, so Docker is not an option.

voiced solves all three in one process: an HTTP server that speaks the OpenAI API and supervises one whisper-server child per model.

Architecture

voiced is written in TypeScript and compiled with Bun into a single executable. It runs as a per-user launchd agent, io.byteink.voiced, which starts at login and is restarted if it exits.

client (OpenAI SDK)
   │  POST /v1/audio/transcriptions
   ▼
voiced  :2022
   ├── whisper-server  127.0.0.1:2023   ggml-large-v3-turbo.bin
   ├── whisper-server  127.0.0.1:2024   ggml-large-v3.bin
   └── one child per ggml-*.bin in ~/.voiced/models/

At startup, voiced scans ~/.voiced/models/ for ggml-*.bin files and assigns each model a port, counting up from 2023. The children bind to 127.0.0.1 only; the gateway is the only thing clients talk to. The model field of each request selects the child. whisper-1, the model name most OpenAI clients send by default, is an alias for large-v3-turbo when that model is installed.

Models load on first use

Assigning a port does not start a process. A whisper-server child holds its whole model in memory, and large-v3 alone is 2.9 GB on disk. Keeping every installed model resident would cost that memory around the clock for a workload that is idle most of the time. So voiced keeps two sets: models that are installed, and children that are running.

A request for a model with no running child goes through ensureChild:

1. If a child is running, use it.
2. If another request is already starting this model, wait on the same promise. Two concurrent requests never race two processes onto one port.
3. Otherwise, spawn whisper-server and poll it every 250 ms until it answers, for up to 120 seconds by default. A cold start reads the model off disk, so this is measured in seconds.

Each request increments an in-flight counter on its child and decrements it in a finally block, so an error or a client disconnect still releases it. When the counter reaches zero, an idle timer starts, five minutes by default. When it fires, the timer checks again that nothing is in flight, then stops the child. A steady stream of requests keeps a model loaded; a quiet period frees the memory.

Crashes are handled separately from eviction. When a child exits, voiced respawns it after two seconds only if it is still the registered process for that model. An evicted or removed child has already been taken out of the map, so its exit is not treated as a crash.

/health reports both sets. An empty loaded list is the normal idle state:

{ "ok": true, "models": ["large-v3", "large-v3-turbo"], "loaded": [] }

The trade-off is latency on the first request after an idle period. VOICED_IDLE_MS=0 keeps a model resident once it has been used.

Translating requests

For a normal transcription, voiced reads the multipart form, keeps file, language, prompt, temperature and response_format, drops everything else, and posts the result to the child's /inference. response_format defaults to json, as OpenAI's does. The child runs with --convert, so it accepts the usual audio formats through ffmpeg.

One detail comes from how whisper-server accepts connections. It serves one inference at a time and refuses a connection that arrives while it is busy, most visibly right after a cold start. voiced retries connection failures only, with exponential backoff from 250 ms to 4 seconds, about 7.75 seconds in total. An HTTP response of any status is the child answering and is passed through. An abort is the client leaving and is never retried.

Backpressure

Every transcription passes through a single admission gate. The default limit is four concurrent requests. At the limit, voiced does not queue: it returns 503 with Retry-After: 1 immediately, so a client sees backpressure instead of waiting on a pile-up. The limit is set by VOICED_MAX_CONCURRENCY, and voiced limit N changes it on the running server and persists it across restarts.

The gate matters most for diarization, where each request starts its own ffmpeg process and its own diarizer process. Without a ceiling, a burst of requests would start as many processes as there were requests.

Cancellation is threaded through every stage. When a client disconnects, ffmpeg and the diarizer are killed and the request is logged with status 499. The one thing voiced cannot stop is a decode already running inside whisper-server, because its HTTP library cannot interrupt an in-flight handler. The model becomes free one decode later.

Speaker diarization

Passing diarize=true with response_format=verbose_json adds a speaker field to every segment:

{ "id": 0, "start": 0.0, "end": 5.64, "text": " a pencil…", "speaker": "SPEAKER_00" }

A diarized request decodes the upload to 16 kHz mono WAV with ffmpeg, then runs transcription and diarization in parallel on the same file. Each transcript segment gets the speaker whose turn overlaps it most, or the nearest turn if none overlaps. A request without diarize is unchanged.

Two engines are available, and neither needs Python:

• Sortformer (sortformer-v2.1, sortformer-v2): NVIDIA's Sortformer model, run by a small Rust sidecar, voiced-diarize, on ONNX Runtime. It has better turn detection and a hard limit of four speakers.
• sherpa: sherpa-onnx clustering with pyannote segmentation and WeSpeaker embeddings. It handles any number of speakers with lower turn accuracy.

Both engines print the same line format, so the server's parser works with either. Diarization is full-file only: speaker labels are computed over the whole recording and do not stay consistent across separate short requests.

Trust boundary and limits

The gateway listens on all interfaces and has no authentication. The README states the model: a private network such as Tailscale is the trust boundary. The two admin endpoints, which change the concurrency limit and reload the model set, accept connections from loopback only.

Other limits, from the README:

• Apple Silicon Macs only. The Mac must be awake. launchd restarts the process, but macOS sleep stops it.
• Memory scales with loaded models. Each loaded model holds its full weights in RAM.
• No streaming. A whole file goes in and a whole transcript comes out.
• No text-to-speech yet. /v1/audio/speech returns 501 so clients that hardcode the route get a clear error instead of a 404.

Try it

Install with Homebrew, which pulls in whisper-cpp and ffmpeg, then download a model and start the agent:

brew install byteink/tap/voiced
voiced add large-v3-turbo
voiced start
curl http://127.0.0.1:2022/health

voiced ls lists installed models and the catalogue, which runs from base.en at 142 MB to large-v3 at 2.9 GB. voiced doctor checks the data directories, binaries, models, launchd agent and HTTP endpoint, and exits non-zero on any failure.

Point an OpenAI client at it through the environment:

OPENAI_BASE_URL=http://<mac-hostname>:2022/v1
OPENAI_API_KEY=any-non-empty-string
OPENAI_AUDIO_MODEL=whisper-1

The API key can be any non-empty string, because OpenAI SDKs require one and voiced ignores it. To add speaker labels, install and select a diarization engine:

voiced diarize add sortformer-v2.1
voiced diarize use sortformer-v2.1

voiced follows the same principle as ThinkByte, our iPhone assistant: the model runs on hardware you own and the data stays there. For AI work that has to run under the same constraint, see AI consulting. The source is on GitHub, and our other public tools are on the open-source page.