|
|
--- |
|
|
license: apache-2.0 |
|
|
--- |
|
|
# Emotion2Vec-S |
|
|
|
|
|
C<sup>2</sup>SER: [Paper](https://arxiv.org/abs/2502.18186) | [Code](https://github.com/zxzhao0/C2SER) | [HuggingFace](https://huggingface.co/collections/ASLP-lab/c2ser-67bc735d820403e7969fe8a0) |
|
|
|
|
|
## Introduction |
|
|
|
|
|
This repository contains the implementation of Emotion2Vec-S, a self-supervised learning (SSL) model for speech emotion recognition, as presented in our paper "Steering Language Model to Stable Speech Emotion Recognition via Contextual Perception and Chain of Thought". |
|
|
|
|
|
## Requirements and Installation |
|
|
|
|
|
This project follows the fairseq installation process. |
|
|
|
|
|
### Requirements |
|
|
|
|
|
- PyTorch version >= 1.10.0 |
|
|
- Python version >= 3.8 |
|
|
|
|
|
### Installation |
|
|
|
|
|
To install fairseq and develop locally: |
|
|
|
|
|
```bash |
|
|
git clone https://github.com/pytorch/fairseq |
|
|
cd fairseq |
|
|
pip install --editable ./ |
|
|
``` |
|
|
|
|
|
### Feature Extraction |
|
|
|
|
|
You can download the pre-trained [Emotion2vec-S model](https://drive.google.com/drive/folders/1LWWi6bahzn7fJP4fCgPleOyQ30sD_BWO?usp=drive_link) and put it in the `./Emotion2Vec-S/ckpt` folder. |
|
|
Meanwhile，we have provided the pretrained checkpoints in the huggingface model hub. You can also download ckpt file from [here](https://huggingface.co/ASLP-lab/Emotion2Vec-S). We also provide [here](https://drive.google.com/drive/folders/12AOVJT7I9GSLJnjHa-Elc-UKgog-mZR2) the feature files for the Emo-Emilia dataset extracted using Emotion2vec-S. |
|
|
|
|
|
If you want to extract features using Emotion2Vec-S，you will also need to provide a `wav.scp` file and place it in the `./Emotion2Vec-S` directory. Here is an example of the `wav.scp` file:： |
|
|
```pgsql |
|
|
audio_name1 /path/to/audio_name1.wav |
|
|
audio_name2 /path/to/audio_name2.wav |
|
|
audio_name3 /path/to/audio_name3.wav |
|
|
``` |
|
|
|
|
|
Next, you can directly run the following code to extract features： |
|
|
```python |
|
|
import torch |
|
|
import os |
|
|
import sys |
|
|
import json |
|
|
import numpy as np |
|
|
import argparse |
|
|
from tqdm import tqdm |
|
|
import torchaudio |
|
|
import torch.nn.functional as F |
|
|
import fairseq |
|
|
from dataclasses import dataclass |
|
|
|
|
|
SAMPLING_RATE=16000 |
|
|
|
|
|
@dataclass |
|
|
class UserDirModule: |
|
|
user_dir: str |
|
|
|
|
|
def extract_fairseq_feature(wav_path, model, device): |
|
|
try: |
|
|
wav, sr = torchaudio.load(wav_path) |
|
|
# 合并多声道为单声道（取平均） |
|
|
if wav.size(0) > 1: |
|
|
wav = torch.mean(wav, dim=0, keepdim=True) |
|
|
if sr != SAMPLING_RATE: |
|
|
wav = torchaudio.functional.resample(wav, sr, SAMPLING_RATE) |
|
|
wav = wav[0, :].view(1, -1) |
|
|
wav = wav.to(device) |
|
|
out = model.extract_features(wav) |
|
|
return out |
|
|
except Exception as e: |
|
|
print(f"Error processing audio file {wav_path}: {e}") |
|
|
return None |
|
|
|
|
|
if __name__ == '__main__': |
|
|
|
|
|
parser = argparse.ArgumentParser() |
|
|
parser.add_argument('--model_path', type=str, default="./Emotion2Vec-S/ckpt/checkpoint.pt") |
|
|
parser.add_argument('--model_dir', type=str, default="./Emotion2Vec-S/examples/data2vec/") |
|
|
parser.add_argument('--dump_dir', type=str, default="./Emotion2Vec-S/features_frm") |
|
|
parser.add_argument('--device', type=str, default='cuda') |
|
|
parser.add_argument('--data', type=str, default="./Emotion2Vec-S/wav.scp") |
|
|
parser.add_argument('--level', type=str, default="frame", help="frame or utterance") |
|
|
args = parser.parse_args() |
|
|
|
|
|
data = {} |
|
|
with open(args.data, 'r') as f: |
|
|
for line in f: |
|
|
seg_id, wav_path = line.strip().split(maxsplit=1) |
|
|
data[seg_id] = wav_path |
|
|
|
|
|
os.makedirs(args.dump_dir, exist_ok=True) |
|
|
|
|
|
seg_ids = data.keys() |
|
|
print(f'Loaded {len(seg_ids)} audio entries') |
|
|
# load models |
|
|
my_model_path = UserDirModule(args.model_dir) |
|
|
fairseq.utils.import_user_module(my_model_path) |
|
|
model, cfg, task = fairseq.checkpoint_utils.load_model_ensemble_and_task([args.model_path]) |
|
|
model = model[0].to(args.device) |
|
|
|
|
|
for seg_id in tqdm(seg_ids): |
|
|
|
|
|
wav_path = data[seg_id] |
|
|
if not os.path.exists(wav_path): |
|
|
print(f"WARNING: {wav_path} does not exist") |
|
|
continue |
|
|
try: |
|
|
torchaudio.load(wav_path) |
|
|
except: |
|
|
print(f'ERROR: Failed to load {wav_path}') |
|
|
continue |
|
|
|
|
|
feat = extract_fairseq_feature(wav_path, model, args.device) |
|
|
|
|
|
if feat is not None: |
|
|
if args.level == 'frame': |
|
|
feat = feat['x'].cpu().detach().numpy()[0] |
|
|
elif args.level == 'utterance': |
|
|
feat = feat['utt_x'].cpu().detach().numpy()[0] |
|
|
else: |
|
|
raise ValueError("Unknown level: {}".format(args.level)) |
|
|
|
|
|
save_path = os.path.join(args.dump_dir, f"{seg_id}.npy") |
|
|
os.makedirs(os.path.dirname(save_path), exist_ok=True) |
|
|
np.save(save_path, feat) |
|
|
print(f"Processed: {seg_id} | Shape: {feat.shape} | Saved to: {save_path}") |
|
|
else: |
|
|
print(f"Skipped problematic file: {seg_id}") |
|
|
|
|
|
``` |
|
|
Alternatively, you can adjust the code according to your needs. The code path is `./Emotion2Vec-S/speech_feature_extraction.py`. You can also use the `./Emotion2Vec-S/extract_feature.sh` script to batch process features for multiple datasets. The script supports parallel processing and offers the following parameters: |
|
|
|
|
|
- `--model_path`: Path to the checkpoint file |
|
|
- `--model_dir`: Path to the model |
|
|
- `--dump_dir`: Directory to save extracted features |
|
|
- `--device`: Device to run the model on (e.g., 'cuda:0') |
|
|
- `--data`: Path to the dataset scp file |
|
|
- `--level`: Level of feature (frame level or utterance level) |
|
|
|
|
|
## 2. Training and testing on EmoBox using extracted features |
|
|
|
|
|
If you want to test our model on other datasets using [EmoBox](https://github.com/emo-box/EmoBox/tree/main). There is also an example provided below, which you can modify to suit your needs: |
|
|
|
|
|
Use k-fold cross-validation with learning rates (1e-3, 1e-4) and hidden sizes (128, 256): |
|
|
|
|
|
```bash |
|
|
cd examples/sb |
|
|
data=/path/to/your/data_files |
|
|
lrs=(1e-3 1e-4) # Learning rate list |
|
|
hidden_sizes=(128 256) # Hidden size list |
|
|
gpus=(0 1 2 3) # GPU list |
|
|
task_id=0 |
|
|
declare -A dataset_folds=( |
|
|
["mesd"]=1 |
|
|
) |
|
|
declare -A dataset_classes=( |
|
|
["mesd"]=6 |
|
|
) |
|
|
datasets=("mesd") |
|
|
|
|
|
for dataset in "${datasets[@]}"; do |
|
|
folds=${dataset_folds[$dataset]} |
|
|
n_classes=${dataset_classes[$dataset]} |
|
|
|
|
|
for lr in "${lrs[@]}"; do |
|
|
for hidden_size in "${hidden_sizes[@]}"; do |
|
|
gpu=${gpus[$task_id % ${#gpus[@]}]} |
|
|
export CUDA_VISIBLE_DEVICES=$gpu |
|
|
task_number=$((task_id + 1)) |
|
|
for fold in $(seq 1 $folds); do |
|
|
echo "Training fold $fold with lr=$lr, hidden_size=$hidden_size on GPU $gpu, task_number=$task_number, dataset=$dataset..." |
|
|
python3 train.py \ |
|
|
hparams/data2vec2-large_freeze.yaml \ |
|
|
--output_folder /path/to/your/${dataset}-S/fold${fold}_lr${lr}_hidden${hidden_size} \ |
|
|
--seed 1234 \ |
|
|
--batch_size 32 \ |
|
|
--lr $lr \ |
|
|
--train_annotation ${data}/${dataset}/fold_${fold}/${dataset}_train_fold_${fold}.json \ |
|
|
--test_annotation ${data}/${dataset}/fold_${fold}/${dataset}_test_fold_${fold}.json \ |
|
|
--number_of_epochs 100 \ |
|
|
--feat_dir /path/to/your/dump_${dataset}-S \ |
|
|
--label_map ${data}/${dataset}/label_map.json \ |
|
|
--device cuda \ |
|
|
--out_n_neurons ${n_classes} \ |
|
|
--hidden_size $hidden_size & |
|
|
done |
|
|
task_id=$((task_id + 1)) |
|
|
done |
|
|
done |
|
|
done |
|
|
|
|
|
wait |
|
|
echo "All training tasks completed." |
|
|
``` |
