> ## Documentation Index > Fetch the complete documentation index at: https://voxray-cac3ed72.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Your First Voice Agent > Build a working voice agent that greets users and answers questions — from a blank config to a live conversation in under 10 minutes. By the end of this page you will have a Voxray server running locally, connected to real STT, LLM, and TTS providers, and able to hold a spoken conversation in your browser. We use OpenAI for all three pipeline stages because it requires only one API key, but every field documented here maps directly to a provider-agnostic config key you can swap later. ## Minimal working config Copy `config.example.json` to `config.json`, then replace its contents with the minimal config below. The example file contains every available option — useful as a reference, but noisy when learning. Start here instead. ```json config.json theme={null} { "host": "localhost", "port": 3042, "transport": "websocket", "provider": "openai", "stt_provider": "openai", "llm_provider": "openai", "tts_provider": "openai", "model": "gpt-4o-mini", "stt_model": "gpt-4o-mini-transcribe", "tts_voice": "alloy", "api_keys": { "openai": "YOUR_OPENAI_API_KEY" }, "turn_detection": "silence", "turn_stop_secs": 3.0, "vad_min_volume": 0.2, "plugins": [] } ``` Replace `YOUR_OPENAI_API_KEY` with a real key before starting the server. Alternatively, set the `OPENAI_API_KEY` environment variable and omit the `api_keys` block entirely — Voxray falls back to environment variables automatically. ## Config field walkthrough ### Server: `host` and `port` ```json theme={null} "host": "localhost", "port": 3042 ``` `host` controls which network interface Voxray binds to. `localhost` means only processes on the same machine can connect — correct for local development. Change to `0.0.0.0` when you deploy to a server and need external traffic. `port` defaults to `8080` in Voxray's built-in defaults, but `3042` is the value in `config.example.json` and is used throughout this documentation. The browser client at `web/index.html` points to this port by default; if you change it, update the client's server URL to match. ### Transport: `websocket` to start simple ```json theme={null} "transport": "websocket" ``` Voxray supports three transport values: | Value | What it enables | | --------------- | ----------------------------------------------------------- | | `"websocket"` | WebSocket only (`/ws`) — no CGO, no C compiler needed | | `"smallwebrtc"` | WebRTC only (`/webrtc/offer`) — requires CGO + gcc for Opus | | `"both"` | Both transports on the same HTTP server | Start with `"websocket"`. It has no build dependencies, works in every browser, and makes debugging straightforward: you can inspect WebSocket frames directly in Chrome DevTools. Switch to `"both"` or `"smallwebrtc"` once your pipeline logic is correct and you want lower-latency audio delivery. ### Provider selection: `provider`, `stt_provider`, `llm_provider`, `tts_provider` ```json theme={null} "provider": "openai", "stt_provider": "openai", "llm_provider": "openai", "tts_provider": "openai" ``` `provider` is a **fallback default**. When Voxray resolves which provider to use for each pipeline stage, it checks the task-specific key first (`stt_provider`, `llm_provider`, `tts_provider`) and falls back to `provider` only if the task-specific key is absent or empty. This means you can set `"provider": "openai"` as a catch-all and then override individual stages selectively. For example, to use Groq for STT (faster Whisper) but keep OpenAI for LLM and TTS: ```json theme={null} "provider": "openai", "stt_provider": "groq" ``` The priority chain per stage is: task-specific key → `provider` → error. ### Model selection: `model`, `stt_model`, `tts_voice` ```json theme={null} "model": "gpt-4o-mini", "stt_model": "gpt-4o-mini-transcribe", "tts_voice": "alloy" ``` `model` sets the LLM model. `stt_model` sets the transcription model; when omitted, Voxray uses the provider's default. `tts_voice` selects the speaker voice — valid values depend on your TTS provider: | Provider | Example voices | | ---------- | --------------------------------------------------- | | OpenAI | `alloy`, `nova`, `echo`, `fable`, `onyx`, `shimmer` | | ElevenLabs | Use the voice ID from ElevenLabs dashboard | | Sarvam | `anushka`, `arvind`, and other Sarvam voice names | | Google | Voice names from Cloud TTS (e.g. `en-US-Neural2-A`) | There is no `tts_model` in this minimal config because OpenAI's TTS API selects the model based on the voice. For providers that distinguish model from voice (Sarvam, ElevenLabs), add `"tts_model": "..."` alongside `tts_voice`. ### API keys: `api_keys` ```json theme={null} "api_keys": { "openai": "sk-..." } ``` `api_keys` is a flat map from provider name to key. Voxray checks this map first, then falls back to environment variables using standard names (`OPENAI_API_KEY`, `GROQ_API_KEY`, etc.). You never need both; the map is convenient for local dev and environment variables are better for CI and production. For multi-provider configs, list each provider's key under its name: ```json theme={null} "api_keys": { "groq": "gsk_...", "openai": "sk-...", "elevenlabs": "el_..." } ``` Never commit `config.json` with real API keys to version control. Add `config.json` to `.gitignore` and use environment variables in CI/CD pipelines. ### Turn detection: `turn_detection` and `turn_stop_secs` ```json theme={null} "turn_detection": "silence", "turn_stop_secs": 3.0 ``` `turn_detection` controls when Voxray decides the user has finished speaking and sends the accumulated audio to STT. There are two values: * `"none"` — no automatic turn detection; the client signals turn end explicitly. Useful when building custom UI. * `"silence"` — **silence gate**: Voxray listens for `turn_stop_secs` seconds of continuous silence after speech activity, then closes the turn and flushes to STT. `turn_stop_secs: 3.0` means the agent waits for 3 seconds of silence before it considers the user done speaking. This is the most important tuning knob for conversational feel. Lower values (1.0–1.5s) feel snappier but cut off users who pause mid-sentence. Higher values (4–5s) feel more patient but add latency. Start at 3.0 and adjust based on your use case and user feedback. ### VAD: `vad_min_volume` ```json theme={null} "vad_min_volume": 0.2 ``` Voice activity detection (VAD) runs continuously on incoming audio and decides whether a given audio chunk contains speech. `vad_min_volume` sets the minimum RMS amplitude a chunk must exceed before VAD considers it "speech" — chunks below this threshold are treated as silence regardless of content. The practical effect: **`vad_min_volume` is your noise floor gate.** If the agent is triggering on air conditioning, keyboard clicks, or ambient room noise, raise `vad_min_volume` (try 0.3–0.4). If the agent is missing quiet speech or the user's microphone has low gain, lower it (try 0.1–0.15). The default in `config.example.json` is `0.25`; `0.2` in this minimal config is slightly more sensitive, which works well for most laptop microphones in a quiet room. ### Plugins: `plugins` ```json theme={null} "plugins": [] ``` Plugins extend the pipeline with additional processors — echo cancellation, wake-word detection, frame filtering, audio gain, and more. Start with an empty list: your agent will work without any plugins and the config stays readable. Once the core pipeline is working, add plugins one at a time so you can isolate their effect. Available built-in plugins include `echo`, `frame_filter`, `wake_check_filter`, `stt_mute_filter`, `audio_filter`, `interruption_controller`, `external_chain`, and `rtvi`. Each has its own options block under `plugin_options`. See the Extensions documentation for details. ## Adding a system prompt Voxray's config file controls pipeline behavior — provider selection, transport, VAD parameters — but **the system prompt lives in the LLM context**, not in `config.json`. The LLM context is assembled per-session from the first message in the conversation context passed by the client. When a client connects over WebSocket, it sends an initial context payload before audio begins. That payload includes a `messages` array. The first message in that array with `role: "system"` becomes the system prompt the LLM sees on every turn. A minimal context payload sent by the client looks like: ```json theme={null} { "type": "context", "messages": [ { "role": "system", "content": "You are a friendly voice assistant named Aria. Keep your responses concise — one to three sentences — because the user is listening rather than reading. Always greet new users with 'Hello! How can I help you today?'" } ] } ``` Keep system prompts short and speech-aware. Unlike chat interfaces, voice agents read responses aloud, so avoid bullet points, markdown formatting, code blocks, or long enumerations. Write in complete spoken sentences and instruct the LLM to do the same. The reference browser client at `web/index.html` (or `tests/frontend/webrtc-voice.html` for WebRTC) sends this context automatically when you configure it. For custom clients, send the context message immediately after the WebSocket connection is established, before sending any audio frames. ## Starting and testing your agent From the repository root, build Voxray for WebSocket-only use (no CGO required): ```bash theme={null} go build -o voxray ./cmd/voxray ``` On Windows: ```powershell theme={null} go build -o voxray.exe ./cmd/voxray ``` If you have `make` available, `make build` does the same thing. ```bash theme={null} ./voxray -config config.json ``` You should see output similar to: ``` INFO voxray starting transport=websocket host=localhost port=3042 INFO listening on :3042 ``` The server is ready when you see `listening on :3042`. If the port is already in use, change `port` in `config.json` to another value (e.g. `3043`) and restart. Open `web/index.html` directly in your browser (drag the file into a browser tab, or use a local file server): ```bash theme={null} # macOS open web/index.html # Linux xdg-open web/index.html # Or serve it with Python to avoid any file:// restrictions python -m http.server 8000 --directory web # then open http://localhost:8000/index.html ``` Alternatively, use the WebRTC test client (requires a voice build): ```bash theme={null} cd tests/frontend && python -m http.server 3000 # Open http://localhost:3000/webrtc-voice.html ``` Set the server URL field in the client to `ws://localhost:3042/ws`. Click the microphone button to begin a session. Wait for the connection indicator to turn green, then say **"Hello"** clearly. Pause for at least `turn_stop_secs` (3 seconds) after speaking. Voxray's silence gate needs that pause to detect end of turn and dispatch your audio to STT. The agent should respond with audio within 2–3 seconds of the silence gate triggering. Confirm each stage completed by checking the server logs. A successful round-trip looks like: ``` INFO vad speech_detected INFO stt transcript="Hello" INFO llm response_started INFO tts audio_chunk_sent INFO turn complete ``` If any stage is missing, that log line points to where the pipeline stopped. ## Verification checklist Run through this checklist after your first start. Every item corresponds to a distinct system layer — checking all five rules out entire categories of problems. * [ ] Server log shows `listening on :3042` within 2 seconds of startup * [ ] Browser connects without 4xx or 5xx errors (open DevTools → Network tab → filter by WS) * [ ] VAD indicator in the browser client shows activity when you speak (confirms audio is reaching the server) * [ ] Server log shows `stt transcript=` after you stop speaking and the silence gate fires * [ ] Agent responds with audio within 2–3 seconds of the transcript appearing in logs ## Common first-time issues **Agent responds with a robotic or wrong voice** Your `tts_voice` value does not match a valid voice ID for your TTS provider. Each provider has its own voice name format — OpenAI uses short names like `alloy` and `nova`, ElevenLabs uses UUIDs or slug names from your dashboard, and Sarvam uses names like `anushka`. Check the provider's documentation for the exact list of valid voice identifiers and update `tts_voice` to match. **Agent does not respond after you speak** The most common cause is the silence gate not firing. Make sure `turn_detection` is set to `"silence"` and that you paused speaking for the full `turn_stop_secs` duration (3 seconds by default). The gate requires a continuous unbroken silence — if your environment has frequent ambient noise bursts (fans, typing), the gate keeps resetting. Try raising `vad_min_volume` to 0.3 or 0.4 so low-amplitude background noise does not count as speech. If you are certain you paused long enough, check that the WebSocket connection is established (green indicator in the client) and that the server is receiving audio frames (look for `vad` lines in server logs). **No audio from the agent** Two possible causes: 1. **Browser speaker permission denied** — open browser Settings and verify microphone and speaker permissions are granted for the page. Some browsers also require a user gesture (a click) before playing audio; the microphone button in the client satisfies this. 2. **TTS provider API key is invalid or missing** — check the server log for a TTS error line. If the API key in `api_keys.openai` is wrong, the TTS call will return a 401 and Voxray logs the error. Fix the key and restart the server. **Server exits immediately with "invalid config format"** Your `config.json` has a JSON syntax error. Run `cat config.json | python3 -m json.tool` or paste it into [jsonlint.com](https://jsonlint.com) to find the offending line. **Port 3042 already in use** Another process is bound to that port. Either stop it (`lsof -i :3042` on macOS/Linux) or change `port` in your config to a free port. ## Next steps Understand how the STT → LLM → TTS pipeline works, what frames are, and how turn detection and VAD fit together. Step-by-step tutorials from a zero-dependency echo bot to a production multi-provider pipeline with telephony. Run Voxray in production: Docker, environment variables, TLS termination, Prometheus metrics, and horizontal scaling. Connect Voxray to telephony (Twilio, Telnyx, Plivo), WebRTC, Daily.co, and external LLM chains.