How We Achieved 377ms Voice AI Latency: A Technical Case Study

When we started building Edesy's voice AI platform, our initial prototype had 2-second response latency. Conversations felt unnatural—the AI would pause awkwardly after every user input, breaking the flow.

Today, our production system achieves 377ms average latency with Gemini Live 2.5 HD. This is the story of how we got there.

Why Latency Matters in Voice AI

Human conversation has a natural rhythm. Studies show that response delays over 500ms feel unnatural, and delays over 1 second feel robotic.

The latency budget:

• 0-200ms: Feels instant
• 200-500ms: Feels natural
• 500-1000ms: Feels slow but acceptable
• 1000ms+: Feels like talking to a machine

Our goal: Sub-500ms consistently, with <400ms as the stretch target.

The Starting Point: 2-Second Latency

Our initial architecture was straightforward:

User speaks →
  Twilio captures audio →
    Deepgram STT (200ms) →
      GPT-4 processing (800ms) →
        Google TTS (200ms) →
          Audio back to Twilio (100ms) →
            User hears response

Total: ~1,300ms + network overhead = ~2,000ms

Each component added latency, and network round-trips compounded the problem.

Phase 1: Optimizing the Traditional Pipeline

Before adopting native audio LLMs, we optimized the traditional STT→LLM→TTS pipeline.

1.1 Streaming Everything

Problem: Waiting for complete transcription before processing.

Solution: Stream partial transcripts to the LLM. The AI starts generating responses while the user is still speaking.

User: "I want to book an app—"
STT interim: "I want to book"
LLM starts: "Sure, I can help you book..."
User: "—ointment for tomorrow"
STT final: "I want to book an appointment for tomorrow"
LLM continues: "...an appointment. What time works for you?"

Impact: Reduced perceived latency by ~300ms.

1.2 Choosing the Right STT

We tested Deepgram, Google Chirp, Azure, AssemblyAI, and Whisper.

Results:

Decision: Deepgram Nova-2 for English (fastest), Google Chirp for Hindi (most accurate).

Impact: Saved ~50-150ms on STT.

1.3 LLM Optimization

GPT-4 was accurate but slow (~800ms). We experimented with:

• GPT-4o: Faster (~400ms) with similar quality
• GPT-3.5-turbo: Much faster (~200ms) but lower quality
• Claude 3 Haiku: Fast (~250ms) and high quality
• Groq Llama: Ultra-fast (~100ms) on dedicated hardware

Decision: GPT-4o as default, with Groq for latency-critical outbound campaigns.

Impact: Saved ~400ms on LLM processing.

1.4 TTS Optimization

Google Cloud TTS and Azure Neural were both ~100ms. We added:

• Caching common phrases: "Hello, how can I help you?" cached as audio
• Pre-generating greetings: First response ready before call connects
• Parallel generation: Start TTS while LLM is still generating (first sentence)

Impact: Saved ~100ms and eliminated greeting latency.

Phase 1 Result: ~800ms

The optimized traditional pipeline achieved:

• STT: 150ms
• LLM: 400ms
• TTS: 100ms
• Network: 150ms
• Total: ~800ms

Good, but not great.

Phase 2: Native Audio LLMs

The real breakthrough came with native audio-to-audio models.

What Changed

Native audio LLMs (Gemini Live, OpenAI Realtime) process audio directly—no STT or TTS step:

Traditional: Audio → STT → LLM → TTS → Audio
Native:      Audio → Native Audio LLM → Audio

This eliminates ~400ms of processing.

Gemini Live 2.5 HD on Vertex AI

Google offers Gemini Live through two backends:

1. Google AI Studio: Standard API
2. Vertex AI: Enterprise API with optimized infrastructure

We tested both:

That's a 76% improvement just by using Vertex AI.

Why the difference? Vertex AI uses Google's internal network and optimized serving infrastructure. AI Studio goes through public APIs.

The 377ms Breakdown

With Gemini Live 2.5 HD on Vertex AI:

Audio capture: ~20ms
Network to Vertex AI: ~50ms
Gemini processing: ~250ms
Network return: ~50ms
Audio playback start: ~7ms
Total: ~377ms

This is native audio processing—no separate STT or TTS.

OpenAI Realtime Comparison

OpenAI's Realtime API achieved ~500ms average:

Audio capture: ~20ms
Network to OpenAI: ~80ms
GPT-4o processing: ~300ms
Network return: ~80ms
Audio playback: ~20ms
Total: ~500ms

Still excellent, but Gemini Live + Vertex AI wins on latency.

Phase 3: Infrastructure Optimization

Beyond model selection, infrastructure choices matter.

3.1 WebSocket Connection Pooling

Problem: New WebSocket connections add 100-200ms.

Solution: Maintain warm connection pools to providers. Reuse connections across calls.

Impact: Saved ~150ms on first response per call.

3.2 Geographic Optimization

Problem: Network latency to US-based providers from India.

Solution:

• Use Vertex AI's us-central1 region (fastest for Gemini)
• For STT/TTS fallback, use regional endpoints where available
• Consider edge processing for initial audio capture

Impact: Saved ~50ms on round-trips.

3.3 Audio Format Optimization

Problem: High-quality audio means more data, more latency.

Solution: Use telephony-optimized formats:

• 8kHz sample rate (sufficient for voice)
• u-law or a-law encoding
• Small frame sizes (20ms)

Impact: Reduced audio transfer time by ~30%.

3.4 Silence Detection (VAD)

Problem: Detecting when user stops speaking takes time.

Solution: Optimized Voice Activity Detection:

• Silero VAD with tuned parameters
• Language-specific profiles (Hindi speakers pause differently than English)
• Aggressive end-of-speech detection without cutting off users

Configuration:

{
  "vadProfile": "low_latency",
  "confidence": 0.7,
  "stopSecs": 0.6
}

Impact: Reduced turn-taking delay by ~200ms.

The Final Architecture

Our production architecture for minimum latency:

┌─────────────────────────────────────────────────────┐
│                    User Phone                        │
└─────────────────┬───────────────────────────────────┘
                  │ Audio stream
┌─────────────────▼───────────────────────────────────┐
│              Twilio/Exotel (WebSocket)               │
└─────────────────┬───────────────────────────────────┘
                  │ 8kHz u-law
┌─────────────────▼───────────────────────────────────┐
│          Edesy Voice Pipeline (India)                │
│  ┌─────────────────────────────────────────────┐    │
│  │          Silero VAD (low_latency)            │    │
│  └─────────────────┬───────────────────────────┘    │
│                    │                                 │
│  ┌─────────────────▼───────────────────────────┐    │
│  │    Connection Pool Manager (warm sockets)    │    │
│  └─────────────────┬───────────────────────────┘    │
└────────────────────┼────────────────────────────────┘
                     │
┌─────────────────���──▼────────────────────────────────┐
│      Vertex AI (us-central1) - Gemini Live 2.5      │
│                                                      │
│  Audio In → Native Processing → Audio Out            │
│             (~250ms total)                           │
└────────────────────┬────────────────────────────────┘
                     │ Audio response
                     ▼
                User Phone

Total: 377ms average

Latency by Use Case

Different use cases have different latency profiles:

Lessons Learned

1. Native Audio is Transformative

The single biggest improvement came from switching to native audio LLMs. No amount of pipeline optimization matches eliminating the pipeline.

2. Provider Backend Matters

Vertex AI vs AI Studio was a 76% difference for the same model. Always test provider-specific optimizations.

3. Don't Optimize Prematurely

We spent weeks optimizing STT streaming before native audio LLMs made it irrelevant. Wait for stable requirements before deep optimization.

4. Measure Real-World Latency

Lab benchmarks don't capture real-world variance. We measure P50, P95, and P99 on production traffic.

5. User Perception > Raw Numbers

A 400ms response that starts smoothly feels faster than a 350ms response that stutters. Optimize for perceived latency.

Current Benchmarks

Our production metrics (March 2026):

These are measured end-to-end: user stops speaking → user hears response.

What's Next

We're exploring:

• Edge audio processing - Move initial processing closer to users
• Predictive response - Start generating likely responses before user finishes
• Custom model fine-tuning - Faster responses for specific domains

The goal: Sub-300ms consistently.

Conclusion

Achieving 377ms voice AI latency required:

1. Adopting native audio LLMs (Gemini Live 2.5 HD)
2. Using optimized infrastructure (Vertex AI)
3. Tuning every component (VAD, audio format, connection pooling)
4. Measuring relentlessly and optimizing what matters

The result: conversations that feel natural, not robotic.

Want to experience 377ms voice AI? Try Edesy's platform with a free trial.