Index-TTS-LoRA: Fine-Tuning Voice Models for Natural Speech Synthesis

Index-TTS-LoRA is a specialized toolkit designed for Bilibili’s Index-TTS model. It supports LoRA (Low-Rank Adaptation) fine-tuning for both single and multi-speaker configurations, resulting in synthesized speech with improved naturalness and more fluid prosody. This comprehensive pipeline handles everything from audio token extraction and speaker conditioning to training and inference.

Extract Audio Tokens & Speaker Conditions

To begin the process, execute the following command:

python tools/extract_codec.py --audio_list ${audio_list} --extract_condition

The audio_list file should contain pairs of audio file paths and their corresponding transcriptions, separated by a tab (\t). For example:

/path/to/audio.wav  小朋友们，大家好，我是凯叔，今天我们讲一个龟兔赛跑的故事。

Once the extraction is complete, the processed files and speaker_info.json will be saved in finetune_data/processed_data/. Below is a sample speaker_info.json file:

[
    {
        "speaker": "kaishu_30min",
        "avg_duration": 6.6729,
        "sample_num": 270,
        "total_duration_in_seconds": 1801.696,
        "total_duration_in_minutes": 30.028,
        "total_duration_in_hours": 0.500,
        "train_jsonl": "/path/to/kaishu_30min/metadata_train.jsonl",
        "valid_jsonl": "/path/to/kaishu_30min/metadata_valid.jsonl",
        "medoid_condition": "/path/to/kaishu_30min/medoid_condition.npy"
    }
]

Training

To start the fine-tuning process, run:

python train.py

Inference

Generate speech using the following command:

python indextts/infer.py

Speech Synthesis Examples

2. Model Evaluation

For a deeper dive into the evaluation metrics, refer to the 2025 Benchmark of Mainstream TTS Models: Who Is the Best Voice Synthesis Solution? The table below details the Word Error Rate (WER) across various test sets.

Note: The zero-shot results for the base model were obtained using prompts taken directly from the test set.