Build a voice assistant in Python in under an hour: Whisper, ChatGPT, and TTS

A voice assistant sounds like a large engineering project, but the core loop is just three API calls chained together: transcribe speech to text, send that text to a language model, then convert the response back to speech. This tutorial builds a working command-line version of that loop using OpenAI's Whisper for transcription, ChatGPT for the response, and OpenAI's text-to-speech endpoint for the audio output.

What you will need

• Python 3.9 or later installed
• An OpenAI API key with billing enabled (this project uses three separate endpoints, each billed per use). As of 2025, you can also run local models through Ollama, which raised $88 million to push open-weight models as an alternative to proprietary APIs, per Ollama's funding details.
• A microphone, and a way to play audio back from your machine

Step 1: Set up the environment

Create a project folder and install the required packages:

pip install openai sounddevice scipy python-dotenv

Store your API key in a .env file rather than hardcoding it into the script. This matters more than it sounds. API keys committed to code end up leaked in git repositories more often than developers expect.

OPENAI_API_KEY=your-key-here

Step 2: Record and transcribe audio with Whisper

The recording step captures a few seconds of microphone input and saves it as a WAV file, which is then sent to Whisper's transcription endpoint:

import sounddevice as sd, scipy.io.wavfile as wav, os
from openai import OpenAI
from dotenv import load_dotenv
load_dotenv()
client = OpenAI()

def record_audio(filename="input.wav", duration=5, fs=44100):
    print("Recording...")
    audio = sd.rec(int(duration * fs), samplerate=fs, channels=1)
    sd.wait()
    wav.write(filename, fs, audio)

def transcribe(filename="input.wav"):
    with open(filename, "rb") as f:
        result = client.audio.transcriptions.create(model="whisper-1", file=f)
    return result.text

Five seconds is enough for a short question. For longer input, either raise the duration or, for a production version, replace the fixed-duration recording with silence detection so it stops automatically once you finish speaking.

Step 3: Send the transcript to ChatGPT

With text in hand, the chat completion call is straightforward. Keep the system prompt tight, since voice responses should be shorter and more conversational than typical chat output meant for reading on screen. This is where the tutorial meets the real world: getting the medium right matters as much as the model. The missing "ums" and "uhs" that finally make AI speech sound human are a reminder that nuance is everything in voice, as explored in research on AI speech naturalness.

def get_response(user_text):
    completion = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": "You are a voice assistant. Keep answers under 3 sentences, conversational, no markdown."},
            {"role": "user", "content": user_text}
        ]
    )
    return completion.choices[0].message.content

The "no markdown" instruction matters here specifically because a bulleted list makes no sense read aloud. This is a case where the prompt has to account for the output medium, not just the content.

Step 4: Convert the response to speech

OpenAI's text-to-speech endpoint takes that response text and returns audio directly:

def speak(text, filename="output.mp3"):
    response = client.audio.speech.create(model="tts-1", voice="alloy", input=text)
    response.stream_to_file(filename)
    os.system(f"start {filename}" if os.name == "nt" else f"afplay {filename}")

The voice parameter accepts several preset options: alloy, echo, fable, onyx, nova, and shimmer, each with a distinct tone. Try a few against your use case. A customer-facing assistant and a personal note-taking tool do not need the same voice. Platforms like Sarvam Samvaad have already proven the demand for voice AI that works across languages and contexts, as seen in coverage of voice AI in 11 Indian languages.

Step 5: Wire it together

if __name__ == "__main__":
    record_audio()
    text = transcribe()
    print(f"You said: {text}")
    reply = get_response(text)
    print(f"Assistant: {reply}")
    speak(reply)

Run the script, speak a question when prompted, and the full loop, record, transcribe, respond, speak, completes in a few seconds. For a deeper look at how this pipeline fits into the broader AI agent ecosystem, see how a single agent roams across messaging apps with persistent memory.

Where this breaks, and how to fix it

The most common issue is clipped audio. If the fixed five-second window cuts off before you finish talking, the transcription will be incomplete and the response will answer the wrong question. Increasing the duration is the fast fix. Adding a voice activity detection library like webrtcvad is the correct one. Latency is the other constraint worth planning around. Three sequential API calls typically add up to two to four seconds of round trip time. Noticeable but usually acceptable for a personal tool, less so for anything meant to feel truly real time. The gap between prototype and production is still human, as our analysis of vibe coding revealed.