---
library_name: transformers
pipeline_tag: audio-to-audio
tags:
- speech
- audio
- voice-conversion
- speecht5
- hifigan
- speechbrain
- xvector
- sagemaker
datasets:
- cmu-arctic
base_model:
- microsoft/speecht5_vc
- microsoft/speecht5_hifigan
- speechbrain/spkrec-ecapa-voxceleb
license: other
language: en
---

# Model Card for `speech-conversion`

> Any‑to‑any **voice conversion** (speech‑to‑speech) powered by Microsoft’s SpeechT5 voice‑conversion model. Convert a source utterance into the timbre of a target speaker using a short reference clip.

This model card documents the repository **amirhossein-yousefi/speech-conversion**, which wraps the Hugging Face implementation of **SpeechT5 (voice conversion)** and the matching **HiFiGAN** vocoder, with a lightweight training loop and optional AWS SageMaker entry points.

## Model Details

### Model Description
- **Developed by:** Amirhossein Yousefiramandi (repo author)
- **Shared by :** Amirhossein Yousefiramandi
- **Model type:** Speech-to-speech **voice conversion** using SpeechT5 (encoder–decoder) + HiFiGAN vocoder with **speaker x‑vectors** (ECAPA) for target voice conditioning
- **Language(s):** English (training examples use CMU ARCTIC)
- **License:** Repository does not include a license file (treat as “other”). Underlying base models on Hugging Face list **MIT** (see sources).
- **Finetuned from model :** `microsoft/speecht5_vc` (with `microsoft/speecht5_hifigan` as vocoder); speaker embeddings via `speechbrain/spkrec-ecapa-voxceleb`

### Model Sources
- **Repository:** https://github.com/amirhossein-yousefi/speech-conversion
- **Base model card:** https://huggingface.co/microsoft/speecht5_vc
- **Paper (SpeechT5):** https://arxiv.org/abs/2110.07205

## Training Hardware & Environment

- **Device:** Laptop (Windows, WDDM driver model)  
- **GPU:** NVIDIA GeForce **RTX 3080 Ti Laptop GPU** (16 GB VRAM)  
- **Driver:** 576.52  
- **CUDA (driver):** 12.9  
- **PyTorch:** 2.8.0+cu129  
- **CUDA available:** ✅

## Training Logs & Metrics

- **Total FLOPs (training):** `1,571,716,275,216,842,800`  
- **Training runtime:** `1,688.2899` seconds
- **Hardware Type:** Single NVIDIA GeForce RTX 3080 Ti Laptop GPU
## Uses

### Direct Use
- **Voice conversion (VC):** Given a **source** speech clip (content) and a short **reference** clip of the **target** speaker, synthesize the source content in the target’s voice. Intended for prototyping, research demos, and educational exploration.

### Downstream Use 
- **Fine‑tuning VC** on paired speakers (e.g., CMU ARCTIC pairs) for improved similarity.
- Building **hosted inference** services (e.g., AWS SageMaker real‑time endpoint) using the provided handlers.

### Out‑of‑Scope Use
- Any application that **impersonates** individuals without explicit consent (e.g., fraud, deepfakes).
- Safety‑critical domains where audio identity must not be spoofed (e.g., voice‑based authentication).

## Bias, Risks, and Limitations
- **Identity & consent:** Converting into a target speaker’s timbre can be misused for impersonation. Always obtain **informed consent** from target speakers.
- **Dataset coverage:** CMU ARCTIC is studio‑quality North American English; performance may degrade on other accents, languages, or noisy conditions.
- **Artifacts & intelligibility:** Conversion quality depends on model checkpoint quality and speaker embedding robustness; artifacts may appear for out‑of‑domain inputs or poor reference audio.

### Recommendations
- Keep reference audio **mono 16 kHz**, clean, and at least a few seconds long.
- Use **GPU** for real‑time or faster‑than‑real‑time conversion.
- Consider adding **post‑filters** (denoising, loudness normalization) for production use.
- Obtain explicit **consent** and disclose synthesized audio where appropriate.

## How to Get Started with the Model

### CLI (local inference)
```bash
# Requirements (Python 3.10+)
pip install "transformers>=4.42" "datasets>=2.20" "torch>=2.1" \
            "numpy>=1.24" "sentencepiece>=0.1.99" "protobuf>=4.23" \
            "speechbrain>=1.0.0" soundfile

# One‑shot conversion (mono 16 kHz WAVs)
python scripts/convert_once.py \
  --checkpoint microsoft/speecht5_vc \
  --src path/to/src.wav \
  --ref path/to/ref.wav \
  --out converted.wav