FireRedTTS‑2: Stream Voice Cloning for Long‑Form Podcasts and Chatbots

FireRedTTS-2 is a streaming text-to-speech (TTS) system designed for long-form conversations. It excels at multi-speaker dialogue, maintaining consistent and natural prosody throughout. Optimized for podcasts and conversational AI, the system can generate up to three minutes of continuous back-and-forth speech between four distinct voices—and can be extended further with additional training data. It supports zero-shot voice cloning for English, Chinese, Japanese, Korean, French, German, and Russian, enabling language switching mid-sentence while preserving the speaker's vocal identity.

1. Long-Form Dialogue Generation

The model generates three-minute conversations featuring up to four speakers in a single pass. With more diverse training data, both the duration and the number of speakers can be scaled. For podcasts or multi-speaker chatbots, this eliminates the need for fragmented recording cycles; FireRedTTS-2 creates a coherent exchange from a single command.

2. Multilingual Capabilities

Seven languages are supported natively. Zero-shot cloning allows you to replicate a specific voice using only a few seconds of reference audio. Even when switching languages within the same sentence, the vocal characteristics remain stable and the rhythm remains natural.

3. Ultra-Low Latency Streaming

A 12.5Hz streaming speech tokenizer and a dual-Transformer architecture process interleaved text and audio sequences efficiently. On an NVIDIA L20 GPU, the first audio packet is delivered in as little as 140 milliseconds, providing near-instantaneous output without compromising audio quality.

4. Consistent Output Stability

Whether a single speaker narrates for several minutes or four speakers engage in rapid dialogue, the voice profile remains consistent. The system maintains high similarity scores and keeps word error rates (WER) and character error rates (CER) remarkably low, ensuring that the speech rhythm does not drift over time.

5. Random Voice Generation

The system can also generate speech with randomized vocal traits. This is useful for building synthetic datasets for ASR (Automatic Speech Recognition) models or stress-testing voice-interactive systems, reducing the reliance on expensive, studio-recorded corpora.

Installing FireRedTTS-2 and Fetching Models

1. Clone the Repository and Set Up the Environment

Clone the repository and create a dedicated Conda environment.

git clone https://github.com/FireRedTeam/FireRedTTS2.git
cd FireRedTTS2

conda create --name fireredtts2 python==3.11
conda activate fireredtts2

2. Install Dependencies

Install the appropriate PyTorch build followed by the project’s required packages.

pip install torch==2.7.1 torchvision==0.22.1 torchaudio==2.7.1 --index-url https://download.pytorch.org/whl/cu126

pip install -e .
pip install -r requirements.txt

3. Download Pretrained Weights

Use Git LFS to download the model files from Hugging Face.

git lfs install
git clone https://huggingface.co/FireRedTeam/FireRedTTS2 pretrained_models/FireRedTTS2

Basic Usage

1. Monologue Mode

Used for a single voice reading extended passages. This mode works with either randomized voices or voice cloning.

import os
import sys
import torch
import torchaudio
from fireredtts2.fireredtts2 import FireRedTTS2

device = "cuda"
lines = [
    "Hello everyone, welcome to our newly launched FireRedTTS2. It supports multiple languages including English, Chinese, Japanese, Korean, French, German, and Russian. Additionally, this TTS model features long-context dialogue generation capabilities.",
    "如果你厌倦了千篇一律的AI音色，不满意于其他模型语言支持不够丰富，那么本项目将会成为你绝佳的工具。",
    "ランダムな話者と言語を選択して合成できます",
    "이는 많은 인공지능 시스템에 유용합니다、예를 들어, 제가 다양한 음성 데이터를 대량으로 생성해 여러분의 ASR 모델이나 대화 모델에 풍부한 데이터를 제공할 수 있습니다.",
    "J'évolue constamment et j'espère pouvoir parler davantage de langues avec plus d'aisance à l'avenir.",
]

fireredtts2 = FireRedTTS2(
    pretrained_dir="./pretrained_models/FireRedTTS2",
    gen_type="monologue",
    device=device,
)

# Random voice generation
for i in range(len(lines)):
    text = lines[i].strip()
    audio = fireredtts2.generate_monologue(text=text)
    out_name = str(i) + ".wav"
    all_audio = audio.unsqueeze(0).cpu()
    torchaudio.save(out_name, all_audio, 24000)

# Voice cloning (Uncomment to use a reference clip)
# for i in range(len(lines)):
#     text = lines[i].strip()
#     audio = fireredtts2.generate_monologue(
#         text=text,
#         prompt_wav="path/to/reference.wav",
#         prompt_text="corresponding transcript",
#         temperature=0.75,
#         topk=20,
#     )
#     out_name = str(i) + ".wav"
#     all_audio = audio.unsqueeze(0).cpu()
#     torchaudio.save(out_name, all_audio, 24000)

2. Dialogue Mode

For scenarios where multiple speakers interact. You must provide reference clips and transcripts for each speaker.

import os
import sys
import torch
import torchaudio
from fireredtts2.fireredtts2 import FireRedTTS2

device = "cuda"

fireredtts2 = FireRedTTS2(
    pretrained_dir="./pretrained_models/FireRedTTS2",
    gen_type="dialogue",
    device=device,
)

text_list = [
    "[S1]那可能说对对，没有去过美国来说去去看到美国线下。巴斯曼也好，沃尔玛也好，他们线下不管说，因为深圳出去的还是电子周边的会表达，会发现哇对这个价格真的是很高呀。都是卖三十五美金、四十美金，甚至一个手机壳，就是二十五美金开。",
    "[S2]对，没错，我每次都觉得不不可思议。我什么人会买三五十美金的手机壳？但是其实在在那个target啊，就塔吉特这种超级市场，大家都是这样的，定价也很多人买。",
    "[S1]对对，那这样我们再去看说亚马逊上面卖卖卖手机壳也好啊，贴膜也好，还包括说车窗也好，各种线材也好，大概就是七块九九或者说啊八块九九，这个价格才是卖的最多的啊。因为亚马逊的游戏规则限定的。如果说你卖七块九九以下，那你基本上是不赚钱的。",
    "[S2]那比如说呃除了这个可能去到海外这个调查，然后这个调研考察那肯定是最直接的了。那平时我知道你是刚才建立了一个这个叫做呃rean的这样的一个一个播客，它是一个英文的。然后平时你还听一些什么样的东西，或者是从哪里获取一些这个海外市场的一些信息呢？",
    "[S1]嗯，因为做做亚马逊的话呢，我们会关注很多行业内的东西。就比如说行业有什么样亚马逊有什么样新的游戏规则呀。呃，物流的价格有没有波动呀，包括说有没有什么新的评论的政策呀，广告有什么新的打法呀？那这些我们会会关关注很多行业内部的微信公众号呀，还包括去去查一些知乎专栏的文章呀，以及说我们周边有很多同行。那我们经常会坐在一起聊天，看看信息有什么共享。那 this is a 关注内部的一个方式。",
]

prompt_wav_list = [
    "examples/chat_prompt/zh/S1.flac",
    "examples/chat_prompt/zh/S2.flac",
]

prompt_text_list = [
    "[S1]啊，可能说更适合美国市场应该是什么样子。那这这个可能说当然如果说有有机会能亲身的去考察去了解一下，那当然是有更好的帮助。",
    "[S2]比如具体一点的，他觉得最大的一个跟他预想的不一样的是在什么地方。",
]

all_audio = fireredtts2.generate_dialogue(
    text_list=text_list,
    prompt_wav_list=prompt_wav_list,
    prompt_text_list=prompt_text_list,
    temperature=0.9,
    topk=30,
)

torchaudio.save("chat_clone.wav", all_audio, 24000)

3. Web UI

If you prefer a graphical interface over coding, you can launch the Gradio demo.

python gradio_demo.py --pretrained-dir "./pretrained_models/FireRedTTS2"

This opens a local browser tab where you can paste text, upload reference audio, and generate files directly to your machine.