voice-agents – Agents & Automation skill for AI agents

Voice Agents

Voice agents represent the frontier of AI interaction - humans speaking

naturally with AI systems. The challenge isn't just speech recognition

and synthesis, it's achieving natural conversation flow with sub-800ms

latency while handling interruptions, background noise, and emotional

nuance.

This skill covers two architectures: speech-to-speech (OpenAI Realtime API,

one low-latency architecture option) and pipeline (STT→LLM→TTS, more control,

easier to debug). Key insight: latency is the constraint. Humans expect

responses in 500ms. Every millisecond matters.

No market-adoption statistic or universal latency benchmark is established by this skill.

Detailed Guide

Read [the detailed guide](references/detailed-guide.md) before executing this skill. It retains the complete procedure and reference material. Treat its safety, prerequisites, and validation requirements as mandatory. For focused work, load the relevant sections; for end-to-end work, read the guide completely.

Legacy Pipeline Sketch (not a production implementation)

import { Deepgram } from '@deepgram/sdk';
import { ElevenLabsClient } from 'elevenlabs';
import OpenAI from 'openai';

// Initialize clients
const deepgram = new Deepgram(process.env.DEEPGRAM_API_KEY);
const elevenlabs = new ElevenLabsClient();
const openai = new OpenAI();

async function processVoiceInput(audioStream) {
  // 1. Speech-to-Text (Deepgram Nova-3)
  const transcription = await deepgram.transcription.live({
    model: 'nova-3',
    punctuate: true,
    endpointing: 300,  // ms of silence before end
  });

  transcription.on('transcript', async (data) => {
    if (data.is_final && data.speech_final) {
      const userText = data.channel.alternatives[0].transcript;
      // Do not log raw private transcripts by default.

      // 2. LLM Processing
      const completion = await openai.chat.completions.create({
        model: 'gpt-4o-mini',
        messages: [
          { role: 'system', content: 'You are a concise voice assistant.' },
          { role: 'user', content: userText }
        ],
        max_tokens: 150,  // Keep responses short for voice
      });

      const agentText = completion.choices[0].message.content;
      // Log only authorized redacted operational metadata.

      // 3. Text-to-Speech (ElevenLabs)
      const audioStream = await elevenlabs.textToSpeech.stream({
        voice_id: 'voice_id_here',
        text: agentText,
        model_id: 'eleven_flash_v2_5',  // Lowest latency
      });

      // Stream to user
      playAudioStream(audioStream);
    }
  });

  // Pipe audio to transcription
  audioStream.pipe(transcription);
}

Optimization Tips:

• Start TTS while LLM still generating (streaming)
• Pre-compute first response segment during user speech
• Use Flash/turbo models for latency

Voice Activity Detection Pattern

Detect when user starts/stops speaking

When to use: All voice agents need VAD for turn-taking

VOICE ACTIVITY DETECTION (VAD):

VAD Types:
1. Energy-based: Simple, fast, noise-sensitive
2. Model-based: Silero VAD, more accurate
3. Semantic VAD: Understands meaning, best for conversation

When to Use

• User mentions or implies: voice agent
• User mentions or implies: speech to text
• User mentions or implies: text to speech
• User mentions or implies: whisper
• User mentions or implies: elevenlabs
• User mentions or implies: deepgram
• User mentions or implies: realtime api
• User mentions or implies: voice assistant
• User mentions or implies: voice ai
• User mentions or implies: conversational ai
• User mentions or implies: tts
• User mentions or implies: stt
• User mentions or implies: asr

Limitations

• Use this skill only when the task clearly matches the scope described above.
• Do not treat the output as a substitute for environment-specific validation, testing, or expert review.
• Stop and ask for clarification if required inputs, permissions, safety boundaries, or success criteria are missing.

Version, privacy and implementation boundaries

These mixed-version SDK sketches are conceptual and were not executed. Deepgram constructor/transcription.live, ElevenLabs stream signatures, the beta Realtime client and the named Silero JavaScript package must be checked against the installed official SDKs before use; do not infer missing package/runtime availability. smart_format formats text and is not noise reduction; model keyword/keyterm flags vary. A model-VAD probability threshold cannot be set from raw amplitude without a calibrated mapping. Confidence scores/latency figures are not universally comparable.

Define end-of-user-turn to first audible output and measure p50/p95 on representative devices/networks/languages. STT first partial, LLM first token and TTS first audio times are not simply additive end-to-end guarantees. Do not send partial/stale hypotheses to irreversible tools: serialize turns, deduplicate final transcripts, buffer complete TTS-safe segments, maintain history, and cancel/discard obsolete generation/audio. On barge-in truncate unheard assistant content in server history as well as stopping local playback; a checkpoint alone is insufficient.

Clients need appropriate ephemeral/session credentials, not long-lived API keys. Obtain authorized microphone/recording/voice/recipient scope, minimize audio/transcript retention and external-provider transfers, and avoid voice cloning/impersonation without rights. Bound sockets, retries, queues and costs; clean up listeners/streams on disconnect. Spoken names/numbers and claimed identity are not authentication. Validate tenant/recipient/action and obtain transaction-specific confirmation where needed. Text cleanup must preserve decimal numbers, currency, URLs and code semantics; SSML requires supported engines and escaping. A voice prompt is not a tool-authorization boundary. No call, recording or provider request was made.