I built a 20ms voice layer for AI agents on Mac — here's the architecture

Pavel Taran · Written April 30, 2026 · 9 min read

I've been writing software for 24 years. Most of that was typing. The last six months have been different — I do most of my work talking to a fleet of AI agents, and typing has become the bottleneck.

When you stop being the person who writes code and start being the person who directs code, the tool that translates your intent into the agents' input matters more than your editor. The voice-to-agent layer is the new keyboard. It deserves the same engineering rigor as a good keyboard does.

Existing dictation tools failed me on a single number: latency. So I built one that doesn't.

This is the engineering story of how.

Why 20ms is the threshold

When you press a hotkey to start talking, three things have to happen before the first syllable is captured:

1. The OS notices the keyboard event.
2. Your audio device starts capturing.
3. The inference pipeline is warm and ready to consume audio.

If any of these takes more than 20ms, you lose the first word.

This sounds dramatic. It isn't. The neuroscience is straightforward: humans don't experience their own speech with conscious deliberation. The first sound comes out around 150-200ms after the intent to speak. If your tool takes 300ms to start capturing, the user has to either (a) wait, or (b) accept that the first word is gone.

Option (a) breaks flow. Option (b) breaks trust.

Most "press to talk" tools sit somewhere in the 200-400ms range. You learn to wait a beat, then speak. In a focused work session — when your cognitive load is on the content you're dictating, not the tool — that beat disappears, and the first word goes with it.

20ms is the threshold below which the tool stops existing. The audio device is already capturing by the time your brain has decided to speak. There's no waiting. There's no learned pause. The interface is just gone.

That was the design target.

What I tried first

Like any reasonable person, I bought the best-rated dictation app on the market before considering writing my own.

I was disappointed.

The press-to-talk latency was somewhere between 250ms and 500ms depending on cold/warm state. Worse, hands-free mode (talk continuously, text appears progressively) didn't actually progress — text only appeared at the very end of a session, after you'd stopped speaking entirely. You'd dictate two minutes into the void, then the entire transcript would drop in at once.

For general-purpose dictation — writing emails, replying to Slack — these tools are fine. They're optimized for a workflow where you click a UI element, take a breath, then start dictating. The user composes in chunks. The tool catches up.

That's not the workflow I needed. I needed:

• Press hotkey, speak immediately (no learned pause)
• Text appearing in chunks during the session at natural pauses, not held until the end of the entire session
• Multiple hotkeys routing to different destinations — terminal, remote SSH session, an API endpoint
• No cloud dependency for the audio path (privacy + latency floor)

None of the existing tools cleared all four. So I started writing.

Architecture

The latency budget breaks down like this:

Hotkey press
    │
    ├─ 1ms   OS keyboard event hook (CGEventTap)
    ├─ 3ms   Audio device wake + first sample
    ├─ 5ms   Apple Neural Engine model warm-up (already loaded, just activated)
    ├─ 8ms   First inference window (sliding 100ms audio buffer)
    ├─ 2ms   Text injection into focused app
    │
    └─ Total: ~19ms cold start

Every box on that diagram took weeks to optimize.

Why Apple Neural Engine, not GPU or CPU

Apple Silicon ships with three compute engines: CPU, GPU, and the Neural Engine (ANE). For Whisper-class models, the ANE is roughly 3-5× more energy-efficient than the GPU and roughly 10× more efficient than the CPU. More importantly for this use case: ANE wakes up faster.

A GPU inference on Metal has a shader compilation hop (cached after first run, but still measurable on cold start). The ANE, being a fixed-function block compiled ahead-of-time via Core ML, is essentially "always warm". You pay the warmup cost once at app launch, never again.

For a tool that lives in your menu bar and gets used in 5-second bursts dozens of times an hour, this matters more than peak throughput. Steady-state latency on the GPU might be 8ms; the ANE might be 12ms. But if the GPU has to recompile its shader cache because you switched models or formats, that's 50-200ms gone. The ANE doesn't have this failure mode.

Why not Swift

I started in Swift. It's the obvious choice on macOS — first-party language, AppKit/SwiftUI integration, Core ML wrappers in the standard library.

I spent two weeks on it and couldn't make it work consistently.

The problem was concurrent event handling. The pipeline has many fast-firing events: audio buffers arriving every ~10ms, hotkey state changes, inference completion callbacks, text injection events, microphone permission changes, system audio routing changes (Bluetooth disconnect/reconnect). In Swift, the natural way to coordinate these is async/await + actors. In practice, the event timing was unpredictable. Sometimes everything ran at <30ms; sometimes a stall would hit 200ms because of GCD priority inversion or main-thread contention.

I couldn't make the timing deterministic. And without determinism, the 20ms target was unreachable on the worst-case path.

I rewrote the core in Rust. The audio pipeline, hotkey listener, and text injector are all Rust. Core ML inference is wrapped through a thin FFI shim. SwiftUI is used only for the menu bar UI — not on any latency-critical path.

Rust gave me three things Swift didn't:

1. Deterministic memory layout — no surprise allocations, no garbage collection pauses, predictable cache behavior.
2. Explicit concurrency model — tokio with single-threaded runtime for the audio path, work-stealing pool only for non-time-critical tasks.
3. No GCD — async work uses my own pinned threads with real-time priority where the OS allows it.

The rewrite took three weeks. The tail latency dropped from "sometimes 200ms" to "never above 25ms". I haven't seen a stall longer than 30ms in three months of daily use.

The app is the only program in my menu bar that has never glitched on me. Under any load. That's the Rust dividend.

Hands-free streaming

Press-to-talk solves one problem. Hands-free solves a different one.

When you're pacing the kitchen with Bluetooth headphones, you don't want to keep a hand on a hotkey. You want to talk continuously, see text appear in chunks as you go, and stop talking to make it stop.

The naive implementation:

audio (whole session) ──→ buffer until user ends ──→ one big inference ──→ paragraph appears

This is what most tools do. The user experience: dictate for 30 seconds, see nothing, end the session, the entire transcript appears at once. You can't correct yourself mid-thought because you can't see what you said.

The reason this is the default: ASR models like Whisper need real audio context for best accuracy. Single-pass batch inference is simpler. Most tools take the easy path.

The better implementation is chunk-on-pause streaming:

audio in ──→ Voice Activity Detection
                │
                ├─ user pauses (>1s silence) → close current chunk
                ↓
        finalize chunk → ANE inference → display chunk → ready for next chunk

When you pause naturally between sentences or stop to think, the chunk you just spoke gets finalized, transcribed, and inserted into the active app. You speak the next sentence; that gets its own chunk; appears at the next pause. Your text grows in chunks at the natural rhythm of your speech, not at the end of the session.

Why this is hard: the chunk boundary detection has to be tuned. Too aggressive (close on every brief 200ms pause) → fragmented chunks, broken sentences. Too lax (wait 3+ seconds) → the user experiences the same "talk into the void" problem. The sweet spot is configurable per user (default 1.0s silence) but the VAD has to be reliable across noisy environments and different microphones.

Building this on Core ML required some unhappy plumbing — the Whisper-family ASR models aren't natively streaming, so each chunk goes through its own forward pass. The trick is keeping the chunks short enough to feel responsive (each transcribes in <100ms on the Apple Neural Engine) while long enough to give the model linguistic context. About 800 lines of Rust to manage the VAD + chunk lifecycle + post-processing. Worth it.

What this enabled

Latency under 20ms isn't an end in itself. It's a prerequisite. Once the tool stops being slow enough to think about, three things become possible:

1. Voice for everything, not just dictation. When you trust the tool to capture instantly, you start using it for things that aren't traditional "dictation". Quick Slack replies. Search queries. Filenames. Variable names while pairing with an AI agent. The tool becomes a general-purpose voice → text fastlane, not a special-occasion dictation app.

2. Multi-channel routing. With low enough latency, you can have multiple hotkeys routing to different destinations. ⌘+space → text at cursor. ⌘+1 → SSH session to my main agent on a Linux box. ⌘+2 → POST request to Linear with the dictated text packaged as a JSON issue. Each hotkey has different latency requirements; the tight cold-start budget makes all of them feel instant.

3. Hands-free becomes a real mode, not a gimmick. When chunks land at the natural pauses in your speech instead of waiting for you to end the session, you can hold a continuous conversation with a terminal agent while pacing the room. The text grows on screen at the rhythm of your thought. The first time you experience it, you understand why this is a different category from "dictation" as it has historically existed.

What I gave up to get here

The honest tradeoffs:

• Mac only. No plans for Windows or Linux. The Apple Neural Engine optimization is load-bearing for the latency budget. I'd have to rewrite the inference pipeline for any platform without ANE, and the resulting tool would be measurably worse. Not interested in shipping a worse tool.
• Apple Silicon required, macOS 14+. Older Intel Macs don't have the ANE; older macOS versions don't have the Core ML APIs I depend on. ~5% of Mac users excluded.
• 200MB resident RAM when active. The model has to stay loaded for cold-start to be cold-start. This is fine on any modern Mac; mentioned for completeness.
• Bluetooth adds 50-150ms of audio capture latency depending on codec. Wired headphones or built-in mic stay under 5ms. Bluetooth still feels instant subjectively because human speech production has its own ~150ms motor planning latency, but the engineering ceiling there is the radio, not the software.

These tradeoffs are why this tool is faster than the cross-platform alternatives. They're also why this tool will never be cross-platform.

What it looks like

This is the app running in my menu bar:

The hotkey is configurable. The model is configurable (local or cloud, depending on tradeoffs you want — local is private, cloud is slightly higher accuracy on rare words). The behavior of each hotkey is configurable through a plain-text config file in ~/.config/speechbutton/.

That config file is the secret weapon — it's intentionally readable by AI agents. You can hand the file to Claude Code or Cursor and ask "set up a hotkey that takes my dictation and posts it as a Linear issue", and the agent will figure out the right config block. There's a CLAUDE.md next to the config that documents the schema for exactly this.

Try it

It's called SpeechButton. Free up to 5 minutes of transcription per day. Pro tier removes the limit at $7.99/month, $69.99/year, or $129.99 one-time (Lifetime, up to 3 Macs). macOS 14+, Apple Silicon required.

Install with one line:

$ brew install --cask speechbutton/tap/speechbutton

Or download from speechbutton.com — just press a key on the home page and you'll see what 20ms feels like.

If you want to talk about voice OS for AI agent workflows, I'm @SpeechButton on X. Open to engineering questions and willing to nerd out about latency budgets.

This is part of an essay series on how AI-agent-driven development changes the developer toolchain. The next post is about hotkey routing — how I send dictated text to 30 different destinations using 3 modifiers.

SpeechButton

I built SpeechButton — a 20ms push-to-talk voice layer for Mac. Try it if you want: speechbutton.com.