talk
A realtime voice interface over my personal knowledge system — and the build that taught me what unobserved really costs.
What this is
Tap an orb on my phone, get briefed on the top item in my queue, talk it through, and have the decisions land back in Life OS before I've put the phone down. "Defer that to Friday and log that I talked to Cooper about it" ... both writes verifiably in the system, spoken, hands free.
The persona is Nigel, my chief of staff across every surface where the assistant talks to me, not just this one. One line describes him: he holds the whole picture, says the part that matters, and stops. His voice is cedar. He was derived from evidence rather than invented, which matters because the first attempt at his persona file was 684 lines of research report wearing a persona's clothes, and I threw it out.
Underneath, the durable piece isn't the voice vendor. It's a gateway: sixteen verbs over MCP, each classified by risk, behind bearer auth. The voice model on top is swappable. Life OS never learns which vendor is talking.
Why it's built this way
Single-hop speech-to-speech, not an assembled pipeline. An STT → LLM → TTS chain is three network hops and three failure modes stacked, and it can't do instant barge-in. Target was 300–800ms to first audio, so the assembled version was out before it was in.
Risk tiers are enforced in the gateway, never in the prompt. This is the decision I'd defend hardest. Anything irreversible (send an email, delete, publish) is physically absent from the voice toolset. The model cannot call it, because it isn't there. What voice can do is stage a proposed action that I approve from a visual card later. A prompt saying "don't send email" is a suggestion to a system that guesses for a living. An absent tool is a fact.
Measured latency for the gateway's own share: 44–71ms per tool call, queue building 56ms server-side. Comfortable against 300–800ms. Retrieval inside a spoken turn gets a 50–300ms budget with a ceiling around one second, every retrieval tool carries a timeout, and the fallback is spoken, because dead air is the failure the user actually experiences.
The build log, failures left in
By 2026-07-20
— Gateway live behind TLS. 23 end-to-end checks over public HTTPS passing: every tool against real data, the confirmation gate, idempotent replay, staged-not-sent, a revoked token correctly refusing.
2026-07-21/22
— iOS Safari echo cancellation broke. Tapping the remote WebRTC stream into WebAudio to drive the UI meters defeats AEC's reference path: Nigel hears himself through the speaker, reads it as me interrupting, and cuts himself off mid-sentence, in a loop. On Apple WebKit, never connect the remote stream to an AudioContext. Drive the meters synthetically, set playsinline, use far-field noise reduction.
2026-07-22
— Same trap, wider than I'd understood. Connecting anything to AudioContext.destination during a live call, including a 180ms confirmation blip, changes the audio session route and defeats AEC. Output, not just input. Cues now play through a detached <audio> element with a synthesized WAV data URI. I still shipped a same-day regression of exactly this on 2026-07-24, which is why there's a source-scanning guard test now.
2026-07-24
— "The model is over-talking" turned out to be a client bug, not a prompt problem. A watchdog armed on every tool call and disarmed only on events that don't fire when the model calls a tool and speaks in the same response, so it sent an unbidden follow-up. Two identical usage reports for one turn was the tell. I nearly rewrote the prompt instead.
2026-07-25 · THE ONE THAT CHANGED EVERYTHING ELSE
Auditing costs, I found that of 145 voice sessions, 117 had no cost record at all. Not wrong numbers. No record. And the sessions that did have numbers were a blend, because the verification scripts call /session/start identically to the real client, so my own test traffic was structurally indistinguishable from real use. Out of that came the eight-point observability standard I now apply everywhere, the sharpest point being that production and test separate by a declared field, never inferred, never defaulting to production.
⚠ What would break
Apple's audio session. The whole AEC fix is a set of rules about what you must not connect while a call is live, guarded by a test that reads source code rather than listening to audio. A WebKit change, or a clever workaround someone adds, degrades every call with no red test.
The text path records no turn text. Voice turns get written; the chat path writes metrics only, so a text session's trace has tool calls, tokens, and latency, and not one word of what was said. Reconstructing a probe run from the admin view was impossible for exactly that reason. An honest partial pass on the diagnosable-trace requirement, logged rather than papered over.
What I learned
Models claim tool actions they didn't take, roughly a quarter of the time under repetition, on both surfaces I tested. "Staged that for you" without staging. "Filed under Station" when the link had failed. This is the most useful thing I know about shipping agents, and asking nicely doesn't fix it. What works has three parts: a prompt rule naming the exact failure ("if you haven't called the tool yet, call it now, before you describe it"), tools returning explicit failure flags plus a line to say out loud so honesty is in-band, and a post-session pass reconciling every claim against the event record.
Where else AI was confidently wrong: a client library's tool objects carry their JSON schema on .parameters, and my code read .inputSchema with an empty-dict fallback. Every tool reached the model with an empty schema. The model guessed argument names, confidently, and every call failed validation... which looked like a model problem for far too long. Caught by a regression test that now asserts tool definitions have non-empty properties. Silent fallbacks are how a config bug wears a costume.
What this demonstrates
Realtime voice over a private knowledge system, with safety enforced structurally instead of conversationally. Retrieval budgeted to human patience. And the part I'd most want judged: my own instrumentation had been lying by omission for 117 sessions, and rather than patch that one project I wrote the standard that now blocks anything unobservable from shipping anywhere.
I no longer trust a system's own account of what it did. I trust the event record, and if there isn't one, that's the bug.