Telephony Text to Speech API: PCM & Latency Guide (2026)

A telephony text to speech API converts spoken agent replies into audio that a phone platform can stream to a caller. Production quality depends less on the model demo than on PCM format, sample rate, first-audio latency, interruption and the boundary between telephony orchestration and speech generation.

Fish Audio supplies expressive speech through its API. Amazon Connect, Twilio, Genesys, Agora and Telnyx own different combinations of media transport, agent logic, provider selection and telephony billing. Verify the current platform contract before claiming one-click support.

What belongs in a telephony TTS architecture?

The minimum stack is a media transport, an agent response source, a TTS request, a PCM conversion path and playback. Real deployments also need cancellation, retries, provider fallback, call recording controls and cost attribution.

Keep the responsibilities separate:

• Agent platform: turn detection, dialogue, tools, transfer and fallback policy.
• Media layer: SIP, WebSocket, codec conversion, jitter buffer and playback.
• Fish Audio: text-to-speech generation, voice identity and prosody.
• Adapter: authentication, request mapping, PCM normalization and request logging.

If one vendor supplies every layer, the architecture is simpler but less portable. A modular stack requires explicit ownership for each failure mode.

Which telephony platforms support custom TTS?

Support varies. Some platforms expose a custom HTTP or WebSocket audio service; others provide a controlled provider list, marketplace approval or media-stream escape hatch.

Review the official Agora custom TTS, Twilio ConversationRelay, Amazon Connect third-party TTS and Genesys TTS pages. Product provider lists and marketplace rules change.

What PCM settings cause the most failures?

Wrong header, sample rate, channel count, signedness or byte order are the most common telephony defects. A generated file can play correctly in a browser and still produce static on a SIP path because the browser decoded a container the adapter declared as raw PCM.

An 8 kHz mono stream is common for PSTN narrowband audio. Real-time AI pipelines often use 16 kHz or 24 kHz internally and resample before telephony playback. Use the transport's required rate and avoid repeated resampling.

Document the complete audio contract:

The Agora integration guide shows a direct PCM adapter path. The existing Twilio text to speech guide covers a media-stream bridge in more detail.

How should you measure telephony TTS latency?

Measure from the caller's final syllable to the first audible reply. Break the result into STT endpointing, LLM first token, Fish Audio time to first byte, adapter conversion, network transport, jitter buffer and playback.

Record p50 and p95 rather than only the average. A median under one second can hide a slow tail where one in twenty callers waits several seconds.

Set separate budgets for a short acknowledgement, a tool-driven explanation and an error message. A pre-recorded acknowledgement can start sooner than a dynamic answer and keeps the caller informed while retrieval finishes.

The real-time text to speech guide covers streaming and chunking, while the voice agent guide explains the complete speech loop.

How do you implement barge-in for generated speech?

The telephony platform must cancel queued and in-flight audio when the caller starts speaking. The adapter must propagate cancellation to Fish Audio and stop sending bytes to the media transport.

Test four stages: TTS request in flight, first audio already played, LLM still generating and fallback active. In each case, verify that audio stops, the transcript records the interruption and the next answer does not repeat stale content.

Avoid client-side-only muting. If the server continues generating and queuing audio, the next turn can play old speech or incur unnecessary Fish Audio credits.

How should custom TTS and platform TTS fall back?

Define failures by class: timeout, rate limit, invalid voice, authentication, malformed audio, exhausted credits and no first byte. Return explicit errors to the agent platform so it can choose the fallback.

A fallback voice should preserve language, role, gender presentation and approximate pace. If no acceptable fallback exists, use a short scripted handoff rather than changing personas mid-sentence.

See the voice-agent TTS fallback guide for a provider-neutral fault-injection matrix.

How should a telephony project budget TTS usage?

Fish Audio credits follow generated characters, not call duration. A one-minute reply of roughly 900 characters commonly uses about one credit, and the Studio pack includes 15,000 credits plus API access.

Add retries and cancelled generations that still consume provider resources. Telephony, SIP, media streams, speech recognition, LLM and agent-platform fees remain separate lines.

Use the TTS API pricing guide for character-volume examples. Do not compare Fish Audio's prepaid rate with an all-inclusive per-minute telephony bundle without separating the underlying services.

What should the launch test include?

Pilot short confirmations, a long tool-driven answer, explicit interruption, a provider timeout and a fallback. Include names, numbers, account identifiers, prices and two languages.

Review one telephone recording from each region and transport. Measure first audio, total speech duration, failed generations, fallback rate and cost per completed call. Test the system after a pod restart or credential rotation before deeming it stable.

Key takeaways

• Telephony TTS is an audio-contract problem as much as a voice-model choice.
• Custom TTS support varies by platform; verify the current provider or media-stream path.
• Raw PCM must use the exact rate, channel count and byte order expected by the transport.
• Measure p95 from the caller's last syllable to the first audible reply.
• Test interruption and fallback through the real SIP or WebSocket transport.

Frequently asked questions

What is a telephony text to speech API?

It converts agent text into audio that a phone platform, SIP gateway or media stream can play to a caller, usually through streaming PCM at a telephony-compatible sample rate.

Can Fish Audio be used with Amazon Connect or Twilio?

Use the platform-supported custom media or provider path and verify its current contract. Fish Audio is the speech engine; the telephony platform must accept the audio format and routing model.

What sample rate should a phone agent use?

Use the rate required by the media transport. An 8 kHz narrowband stream is common for PSTN audio, while 16 kHz PCM is common for real-time AI pipelines before telephony conversion.

Why does phone TTS sound noisy or distorted?

The usual causes are a header where raw PCM is expected, wrong channel count, signedness or byte order, a sample-rate mismatch, or transcoding after synthesis.

Does a telephony provider include TTS interruption handling?

Only where the platform documents barge-in and media cancellation. The integration must still test whether in-flight audio stops and the next turn uses the right transcript.