Qwen3-8B-Translator-LoRA

This model is a fine-tuned version of Qwen/Qwen3-8B using LoRA for English to Chinese translation, specifically tailored for audio product terminology.

Fine-tuning Details

• Fine-tuning Method: LoRA (Low-Rank Adaptation)
• Dataset: Custom parallel corpus for audio products (English-Chinese)
• Framework: PyTorch, Hugging Face Transformers, TRL, PEFT, Optimum TPU
• Hardware: Google Cloud TPU v3-8

Training Procedure

The model was trained using the SFTTrainer from the TRL library.

Training Hyperparameters

• max_seq_length: 1024
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 32
• num_train_epochs: 10
• eval_strategy: "steps"
• eval_steps: 10
• learning_rate: 5e-5
• lr_scheduler_type: "cosine"
• warmup_ratio: 0.05
• weight_decay: 0.005
• optim: "adamw_torch_xla"

LoRA Configuration

• r: 128
• lora_alpha: 256
• lora_dropout: 0.05
• bias: "none"
• target_modules: ["q_proj", "v_proj","o_proj", "gate_proj","up_proj","down_proj"]
• modules_to_save: ["lm_head"]

Training Results

Step	Training Loss	Validation Loss
10	1.070300	0.859889
20	0.710900	0.649465
30	0.632800	0.522307
40	0.492200	0.463456
50	0.386700	0.421875
60	0.328100	0.405993
70	0.316400	0.392578
80	0.252000	0.386410
90	0.245100	0.372893
100	0.205100	0.372327
110	0.125000	0.373869
120	0.144500	0.430407
130	0.127000	0.403783

Intended Use

This model is intended for translating English text related to audio products into Chinese. It can be used by professionals in the audio industry, technical writers, or anyone needing to translate such content.

Limitations and Bias

• The model's performance is best on text similar to the data it was trained on (audio product domain).
• It may not generalize well to other domains or highly colloquial language.
• As with any language model, there's a potential for biases present in the training data to be reflected in the output.