RAG Voice Agent: Knowledge-Aware Speech Fish Audio

A RAG voice agent retrieves relevant knowledge before answering, then speaks the grounded result. The architecture matters because retrieval adds latency and uncertainty to a conversation that already has a strict turn-taking budget.

Fish Audio supplies the speech layer: transcription for the caller and expressive streaming TTS for the answer. Your RAG system owns the source data, retrieval, ranking, prompt, tools and answer policy.

What is the minimal RAG voice-agent loop?

The loop has six steps:

1. Transcribe the caller.
2. Rewrite ambiguous questions into a retrieval query.
3. Retrieve the smallest useful set of approved passages.
4. Generate an answer constrained to those passages.
5. Stream the answer through Fish Audio.
6. Log the evidence and outcome without retaining sensitive data unnecessarily.

The AI voice agent guide covers the speech loop. This page focuses on the retrieval layer and the failures it introduces.

Which knowledge should be retrievable?

Retrieve from authoritative, versioned sources: product documentation, current policy, approved support answers and account data exposed through narrow tools.

Do not mix obsolete drafts, marketing claims and legal policy into one unranked index. Retrieval quality starts with content ownership and freshness.

Each result should carry source name, version, effective date, audience and permission scope. A voice agent should never read a passage the caller is not allowed to see.

How do you keep retrieval fast enough for voice?

Precompute embeddings, store compact metadata and retrieve fewer passages. Set a hard timeout and a fallback answer such as checking with a human rather than silently waiting.

Do not retrieve the entire knowledge base into the prompt. More context increases model time and can make the answer less precise.

The latency budget is shared across speech to text, retrieval, the model turn and Fish Audio’s time to first audio. The real-time TTS guide explains how to measure p95.

How should answers be grounded?

Constrain the answer to retrieved evidence and approved actions. When the evidence conflicts, say so and escalate rather than selecting the most convenient passage.

For spoken citations, name the source only when it helps the caller verify or trust the answer. Put the detailed citation in the transcript, support ticket or follow-up message.

Test adversarial retrieval: an outdated page, a near-duplicate policy and a query that retrieves account data for the wrong user.

When should the agent say it does not know?

Say it does not know when retrieval confidence is low, sources conflict or the request needs an action outside policy. A clear “I can connect you with someone who can check” is safer than an invented answer.

Track no-answer rate and human transfer separately. A rise in transfers can indicate stale knowledge, bad chunking or a permission problem.

How much does the speech cost?

One Fish Audio credit covers about 1,000 characters. A concise 900-character support answer is approximately one credit. API access starts with the $99 Studio pack, which contains 15,000 credits.

Retrieval infrastructure, embedding updates, model tokens and telephony are separate. Use the TTS API pricing guide for the speech budget.

What should the evaluation set contain?

Use real questions with known correct answers, then add:

• Questions with two relevant policies.
• Questions that should trigger a tool.
• Requests for another customer’s data.
• Outdated document traps.
• Questions that require a human.
• Interruptions after the first answer sentence.

Record retrieval evidence, final answer and whether a person corrected it. Optimize end-to-end task success, not only similarity search.

How should chunks and permissions be designed?

Chunk by meaningful policy or procedure, not by fixed character count alone. Each chunk should state the rule, the condition and the exception without requiring three neighbouring passages to make sense.

Apply permission filters before ranking. A highly relevant document is still forbidden if the caller is not authorized to see it. Keep current and retired documents distinguishable, and make the effective date available to the model and the audit log.

When sources conflict, route to review. The voice layer should not become the place where an outdated policy accidentally wins because its sentence is easier to speak.

Key takeaways

• RAG supplies knowledge; Fish Audio supplies speech.
• Retrieve fewer authoritative passages and enforce permission scope.
• Set hard retrieval timeouts and safe no-answer behavior.
• Speak concise source context and leave full citations in text.
• Measure task success, transfer rate and p95 latency together.

Frequently asked questions

What is a RAG voice agent?

It combines retrieval-augmented generation with a speech loop: retrieve relevant knowledge, let the model answer with that context, then speak the approved response.

Does Fish Audio provide retrieval or a vector database?

No. Fish Audio provides the speech layer. Your application owns documents, retrieval, ranking, model prompts, permissions and answer policy.

How do you keep a RAG voice agent fast?

Keep answers short, precompute indexes, return fewer stronger passages, set retrieval timeouts and stream speech in small units after the answer has enough context.

How do you handle citations in voice?

Speak the source when it changes the decision or can be checked later. Do not read long citation blocks aloud; expose them in a transcript or follow-up link instead.

How much does the speech cost?

Fish Audio consumes one credit for about 1,000 characters, and API access comes with the $99 Studio pack. Retrieval, models and telephony are separate costs.