Core Concepts

Before diving into implementation, it's important to understand the core concepts that make Edesy Voice Agent work.

The Voice Agent Pipeline

A voice agent is essentially a real-time processing pipeline that transforms speech into intelligent responses:

┌─────────────────────────────────────────────────────────────────────┐
│                         Voice Agent Pipeline                         │
├─────────────────────────────────────────────────────────────────────┤
│                                                                      │
│   ┌─────────┐   ┌─────────┐   ┌─────────┐   ┌─────────┐   ┌──────┐ │
│   │  Audio  │──▶│   VAD   │──▶│   STT   │──▶│   LLM   │──▶│ TTS  │ │
│   │  Input  │   │         │   │         │   │         │   │      │ │
│   └─────────┘   └─────────┘   └─────────┘   └─────────┘   └──┬───┘ │
│                      │                            │           │     │
│                      │         Interruption       │           │     │
│                      └────────────────────────────┘           │     │
│                                                               ▼     │
│                                                         ┌──────────┐│
│                                                         │  Audio   ││
│                                                         │  Output  ││
│                                                         └──────────┘│
└─────────────────────────────────────────────────────────────────────┘

Frames

Frames are the fundamental unit of data in the pipeline. Everything - audio, text, control signals - flows through the system as frames.

Frame Types

Frame Flow Example

User says: "What's the status of order twelve thirty four?"

Frame 1: InputAudioFrame { audio: [bytes...], timestamp: 0ms }
Frame 2: InputAudioFrame { audio: [bytes...], timestamp: 20ms }
...
Frame N: InterimTranscriptionFrame { text: "What's the status" }
Frame N+1: InterimTranscriptionFrame { text: "What's the status of order" }
Frame N+2: TranscriptionFrame { text: "What's the status of order 1234?", is_final: true }
Frame N+3: LLMResponseFrame { text: "Let me check that for you..." }
Frame N+4: FunctionCallFrame { name: "get_order_status", args: { order_id: "1234" } }
Frame N+5: LLMResponseFrame { text: "Your order 1234 is out for delivery..." }
Frame N+6: TTSAudioFrame { audio: [bytes...] }

Sessions

A session represents a single call/conversation with state:

type Session struct {
    // Identity
    CallSid     string    // Unique call identifier
    StreamSid   string    // Audio stream identifier
    UserIdPin   string    // Internal reference

    // Configuration
    Agent       *AgentConfig
    Language    string
    Providers   ProviderConfig

    // State
    Transcript  []Message
    Variables   map[string]string
    StartTime   time.Time
    Status      CallStatus

    // Channels
    STTChannel  chan string
    TTSChannel  chan string
    EventChan   chan Event
}

Session Lifecycle

1. INITIALIZING
   └── WebSocket connected, loading agent config

2. GREETING
   └── Playing initial greeting message

3. LISTENING
   └── Waiting for user speech

4. PROCESSING
   └── STT → LLM → TTS pipeline active

5. SPEAKING
   └── Playing TTS audio to user

6. TRANSFERRING (optional)
   └── Connecting to human agent

7. ENDING
   └── Cleanup, save recordings, log disposition

Providers

Providers are pluggable components that handle specific tasks:

Provider Interface

Each provider type implements a standard interface:

// STT Provider
type STTProvider interface {
    Connect(ctx context.Context) error
    SendAudio(audio []byte) error
    ReceiveTranscript() <-chan TranscriptResult
    Close() error
}

// TTS Provider
type TTSProvider interface {
    Synthesize(ctx context.Context, text string) (<-chan []byte, error)
    Close() error
}

// LLM Provider
type LLMProvider interface {
    Generate(ctx context.Context, messages []Message, tools []Tool) (<-chan string, error)
    Close() error
}

// Call Provider (Telephony)
type CallProvider interface {
    ProcessInput(audioChan chan []byte, sttOutputChan chan string)
    ProcessOutput(ttsOutputChan chan string)
    SendClear() error
    Close() error
}

Provider Selection

Providers are selected per-agent based on configuration:

{
  "agent": {
    "name": "Customer Support",
    "language": "en",
    "sttProvider": "deepgram",
    "ttsProvider": "cartesia",
    "llmProvider": "openai"
  }
}

Voice Activity Detection (VAD)

VAD determines when the user is speaking vs. silent:

Audio Signal:
─────┬──────────────────┬────────────────┬──────────
     │   User Speech    │    Silence     │   Speech
     │                  │                │
VAD: ─────█████████████──────────────────█████─────
          ↑            ↑                 ↑
     Speech Start  Speech End      Speech Start

VAD Parameters

Interruptions (Barge-In)

When a user speaks while the bot is talking, we need to handle the interruption gracefully:

Timeline:
──────────────────────────────────────────────────────────▶

Bot speaking:  "Your order is currently being processed..."
                              │
User speaks:          "When will it arrive?"
                              │
                              ▼
                    ┌─────────────────┐
                    │ InterruptionFrame │
                    └─────────────────┘
                              │
               ┌──────────────┼──────────────┐
               ▼              ▼              ▼
        Clear TTS      Cancel LLM    Process new input
         buffer        generation    "When will it arrive?"

Interruption Handling Strategies

Context Management

The LLM needs conversation context to generate relevant responses:

type ConversationContext struct {
    SystemPrompt   string            // Agent personality/instructions
    Messages       []Message         // Conversation history
    Variables      map[string]string // Dynamic variables
    Tools          []Tool            // Available functions
    CurrentIntent  string            // Detected user intent
}

type Message struct {
    Role      string    // "user", "assistant", "system", "tool"
    Content   string    // Message text
    Timestamp time.Time
    ToolCalls []ToolCall // For assistant messages with function calls
}

Variable Substitution

Variables allow dynamic content in prompts and responses:

System Prompt:
"You are a customer support agent for {{company_name}}.
The customer's name is {{customer_name}} and their
order number is {{order_id}}."

Variables:
{
  "company_name": "Acme Corp",
  "customer_name": "John",
  "order_id": "ORD-12345"
}

Result:
"You are a customer support agent for Acme Corp.
The customer's name is John and their
order number is ORD-12345."

Events and Webhooks

The system emits events at key points for external integration:

Event Types

Webhook Integration

// Webhook payload example
{
  "event": "call.ended",
  "timestamp": "2024-01-15T10:30:00Z",
  "call_sid": "CA123456",
  "data": {
    "duration": 120,
    "disposition": "SUCCESS",
    "recording_url": "https://...",
    "transcript": [...]
  }
}

Next Steps

Now that you understand the core concepts:

1. Configuration Guide - Set up your first agent
2. Pipeline Deep Dive - Understand frame processing
3. Provider Setup - Configure STT/TTS/LLM providers