Build log
·
Running a private LLM on a Jetson AGX Orin
I tried running private speech-to-text and text-to-speech on a 64GB Jetson. It worked, but a one-to-two second delay told me a lot about when local voice AI makes sense.
Why I wanted voice that never leaves the building
Most voice AI works by shipping your audio off to someone else’s cloud. That’s fine for a lot of things, but plenty of the small businesses, nonprofits, and schools I work with would rather their conversations stay put. So I wanted to see if I could run private speech-to-text and text-to-speech entirely on my own hardware.
The setup
The box is an NVIDIA Jetson AGX Orin with 64GB of memory. The pipeline has three pieces:
Whisper for speech-to-text
Piper for the voice coming back out
Pipecat to tie it all together into one conversation loop
How it actually felt
It worked, which was the exciting part. But there was a noticeable delay of a second or two before every reply, and that’s right where a conversation starts to feel unnatural. As far as I can tell, most of the lag came from the GPU generating the voice, not from the transcription.
What I tried
[Fill in from your notes: the tweaks you tried to cut the delay, and what each one did.]
Verdict: a tinkerer’s project, for now
Running private voice AI locally is very doable, and it’s a great way to learn how these pipelines fit together. But for a proof of concept or an MVP, I’d reach for a commercial platform like Vapi first. Get the experience right, prove people actually want it, and go local once privacy is the thing that really matters.
Want the next one in your inbox?