Real-time voice assistant

What you'll build

Step-by-step build

1. 1

   Prove the pipeline non-streaming first

   Record a short clip, run Whisper to get text, send it to Groq, and speak the reply with your TTS. It will feel slow, and that is fine. You now have a correct baseline and can attack latency one stage at a time.

2. 2

   Open a realtime channel

   Replace request/response with a WebSocket between the browser and the FastAPI service. This persistent connection lets audio flow up and text and audio flow back down continuously, which is the foundation everything else builds on.

3. 3

   Stream speech-to-text

   Send microphone audio in small chunks and feed them to Whisper incrementally so partial transcripts appear while the user is still talking. Decide when an utterance is finished, either with a short silence timeout or voice activity detection.

4. 4

   Start the model early

   As soon as you have a confident end-of-utterance transcript, send it to Groq and stream the answer tokens back. Groq's speed is doing heavy lifting here; the faster the first token, the more alive it feels.

5. 5

   Stream text-to-speech

   Do not wait for the full answer. Buffer the model's output to the first sentence boundary, synthesize that with your TTS, and start playing it while later sentences are still being generated and synthesized. Overlapping generation and speech is the core trick.

6. 6

   Handle barge-in

   When the user starts speaking while the assistant is talking, stop playback immediately, cancel the in-flight TTS and LLM streams, and start a new turn. A real conversation is interruptible; without this it feels robotic.

7. 7

   Manage turns and session state

   Keep per-session conversation history in Redis so multi-turn context works, and track whose turn it is so the two audio directions do not collide. Clean, explicit turn-taking prevents the assistant from talking over itself.

8. 8

   Measure and cut latency

   Instrument each stage: time to first transcript, time to first token, time to first audio. Find the biggest number and shrink it, whether by smaller audio chunks, an earlier TTS start, or model choice. Optimize what you can measure.

9. 9

   Build the client UI

   A single talk button or always-on mic, a live transcript of both sides, and a clear speaking indicator. Handle microphone permissions and the case where audio playback is blocked until the user interacts with the page.

10. 10

    Deploy and demo

    Host the Python service on the AWS free tier and the client on Vercel. Record a short clip of a real back-and-forth, including an interruption, and put it in the README with the measured latency numbers so the responsiveness is undeniable.

Done when

• ✓You speak and the assistant begins replying in about a second or less, not after a long pause.
• ✓It starts speaking its answer before the full answer has finished generating.
• ✓You can interrupt mid-sentence and it stops and listens immediately.
• ✓A multi-turn exchange keeps context so follow-up questions make sense.
• ✓Someone else can load the live URL, grant mic access, and have a real conversation.

Ship it

Run the FastAPI pipeline on the AWS free tier with Whisper and the TTS self-hosted, and serve the Next.js client from Vercel over a secure WebSocket. Because latency is the whole point, put the measured time-to-first-audio and a short recorded conversation, interruption included, right at the top of the README.

What it proves: You can build a low-latency, streaming, realtime AI system where audio and text flow both ways at once, one of the hardest and most sought-after AI engineering skills.

Paste this into Cursor, Claude, or ChatGPT and build it step by step.

You are my senior AI engineer pair. Help me build a "Real-time voice assistant" step by step. The goal is low latency so it feels like a live conversation. Stack: Next.js client for audio capture and playback, a FastAPI service to orchestrate the pipeline, WebSockets for a two-way channel, Whisper (faster-whisper, free/local) for speech-to-text (STT), Groq (free LLM API) for fast token streaming, Piper or Coqui (free, self-hosted) for text-to-speech (TTS), and Redis for session state.

Requirements:
1. Stream microphone audio to the server and transcribe it in chunks, not one blob.
2. Send a finished utterance to the LLM and stream the answer tokens back.
3. Start TTS on the first sentence so the assistant speaks before the full answer exists.
4. Barge-in: if the user speaks while the assistant is talking, stop playback and cancel in-flight streams.
5. Multi-turn memory in Redis and clear turn-taking so the two audio directions never collide.
6. Instrument time-to-first-transcript, first-token, and first-audio.

Work in this order: prove the pipeline non-streaming, then the WebSocket, then streaming STT, then streaming LLM, then streaming TTS, then barge-in, then latency tuning, then the UI. STOP after each step so I can test. Do not write the whole app at once.