|
|
--- |
|
|
library_name: transformers |
|
|
pipeline_tag: audio-to-audio |
|
|
tags: |
|
|
- speech |
|
|
- audio |
|
|
- voice-conversion |
|
|
- speecht5 |
|
|
- hifigan |
|
|
- speechbrain |
|
|
- xvector |
|
|
- sagemaker |
|
|
datasets: |
|
|
- cmu-arctic |
|
|
base_model: |
|
|
- microsoft/speecht5_vc |
|
|
- microsoft/speecht5_hifigan |
|
|
- speechbrain/spkrec-ecapa-voxceleb |
|
|
license: other |
|
|
language: en |
|
|
--- |
|
|
|
|
|
# Model Card for `speech-conversion` |
|
|
|
|
|
> Any‑to‑any **voice conversion** (speech‑to‑speech) powered by Microsoft’s SpeechT5 voice‑conversion model. Convert a source utterance into the timbre of a target speaker using a short reference clip. |
|
|
|
|
|
This model card documents the repository **amirhossein-yousefi/speech-conversion**, which wraps the Hugging Face implementation of **SpeechT5 (voice conversion)** and the matching **HiFiGAN** vocoder, with a lightweight training loop and optional AWS SageMaker entry points. |
|
|
|
|
|
## Model Details |
|
|
|
|
|
### Model Description |
|
|
- **Developed by:** Amirhossein Yousefiramandi (repo author) |
|
|
- **Shared by :** Amirhossein Yousefiramandi |
|
|
- **Model type:** Speech-to-speech **voice conversion** using SpeechT5 (encoder–decoder) + HiFiGAN vocoder with **speaker x‑vectors** (ECAPA) for target voice conditioning |
|
|
- **Language(s):** English (training examples use CMU ARCTIC) |
|
|
- **License:** Repository does not include a license file (treat as “other”). Underlying base models on Hugging Face list **MIT** (see sources). |
|
|
- **Finetuned from model :** `microsoft/speecht5_vc` (with `microsoft/speecht5_hifigan` as vocoder); speaker embeddings via `speechbrain/spkrec-ecapa-voxceleb` |
|
|
|
|
|
### Model Sources |
|
|
- **Repository:** https://github.com/amirhossein-yousefi/speech-conversion |
|
|
- **Base model card:** https://huggingface.co/microsoft/speecht5_vc |
|
|
- **Paper (SpeechT5):** https://arxiv.org/abs/2110.07205 |
|
|
|
|
|
## Training Hardware & Environment |
|
|
|
|
|
- **Device:** Laptop (Windows, WDDM driver model) |
|
|
- **GPU:** NVIDIA GeForce **RTX 3080 Ti Laptop GPU** (16 GB VRAM) |
|
|
- **Driver:** 576.52 |
|
|
- **CUDA (driver):** 12.9 |
|
|
- **PyTorch:** 2.8.0+cu129 |
|
|
- **CUDA available:** ✅ |
|
|
|
|
|
## Training Logs & Metrics |
|
|
|
|
|
- **Total FLOPs (training):** `1,571,716,275,216,842,800` |
|
|
- **Training runtime:** `1,688.2899` seconds |
|
|
- **Hardware Type:** Single NVIDIA GeForce RTX 3080 Ti Laptop GPU |
|
|
## Uses |
|
|
|
|
|
### Direct Use |
|
|
- **Voice conversion (VC):** Given a **source** speech clip (content) and a short **reference** clip of the **target** speaker, synthesize the source content in the target’s voice. Intended for prototyping, research demos, and educational exploration. |
|
|
|
|
|
### Downstream Use |
|
|
- **Fine‑tuning VC** on paired speakers (e.g., CMU ARCTIC pairs) for improved similarity. |
|
|
- Building **hosted inference** services (e.g., AWS SageMaker real‑time endpoint) using the provided handlers. |
|
|
|
|
|
### Out‑of‑Scope Use |
|
|
- Any application that **impersonates** individuals without explicit consent (e.g., fraud, deepfakes). |
|
|
- Safety‑critical domains where audio identity must not be spoofed (e.g., voice‑based authentication). |
|
|
|
|
|
## Bias, Risks, and Limitations |
|
|
- **Identity & consent:** Converting into a target speaker’s timbre can be misused for impersonation. Always obtain **informed consent** from target speakers. |
|
|
- **Dataset coverage:** CMU ARCTIC is studio‑quality North American English; performance may degrade on other accents, languages, or noisy conditions. |
|
|
- **Artifacts & intelligibility:** Conversion quality depends on model checkpoint quality and speaker embedding robustness; artifacts may appear for out‑of‑domain inputs or poor reference audio. |
|
|
|
|
|
### Recommendations |
|
|
- Keep reference audio **mono 16 kHz**, clean, and at least a few seconds long. |
|
|
- Use **GPU** for real‑time or faster‑than‑real‑time conversion. |
|
|
- Consider adding **post‑filters** (denoising, loudness normalization) for production use. |
|
|
- Obtain explicit **consent** and disclose synthesized audio where appropriate. |
|
|
|
|
|
## How to Get Started with the Model |
|
|
|
|
|
### CLI (local inference) |
|
|
```bash |
|
|
# Requirements (Python 3.10+) |
|
|
pip install "transformers>=4.42" "datasets>=2.20" "torch>=2.1" \ |
|
|
"numpy>=1.24" "sentencepiece>=0.1.99" "protobuf>=4.23" \ |
|
|
"speechbrain>=1.0.0" soundfile |
|
|
|
|
|
# One‑shot conversion (mono 16 kHz WAVs) |
|
|
python scripts/convert_once.py \ |
|
|
--checkpoint microsoft/speecht5_vc \ |
|
|
--src path/to/src.wav \ |
|
|
--ref path/to/ref.wav \ |
|
|
--out converted.wav |
|
|
|
