Qwen3-8B-Translator-LoRA

This model is a fine-tuned version of Qwen/Qwen3-8B using LoRA for English to Chinese translation, specifically tailored for audio product terminology.

Fine-tuning Details

• Fine-tuning Method: LoRA (Low-Rank Adaptation)
• Dataset: Custom parallel corpus for audio products (English-Chinese)
• Framework: PyTorch, Hugging Face Transformers, TRL, PEFT, Optimum TPU
• Hardware: Google Cloud TPU v3-8

Training Procedure

The model was trained using the SFTTrainer from the TRL library.

Training Hyperparameters

• max_seq_length: 768
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 32
• num_train_epochs: 10
• eval_strategy: "steps"
• eval_steps: 10
• learning_rate: 2e-5
• lr_scheduler_type: "cosine"
• warmup_ratio: 0.1
• weight_decay: 0.01
• optim: "adamw_torch_xla"

LoRA Configuration

• r: 128
• lora_alpha: 256
• lora_dropout: 0.05
• bias: "none"
• target_modules: ["q_proj", "v_proj", "gate_proj", "down_proj"]
• modules_to_save: ["lm_head", "embed_tokens"]

Training Results

Step	Training Loss	Validation Loss
10	0.844400	0.635387
20	0.486000	0.407656
30	0.439900	0.381002
40	0.391100	0.365226
50	0.370300	0.352978
60	0.307100	0.345395
70	0.368900	0.340513
80	0.306000	0.335354
90	0.273900	0.333215
100	0.272400	0.334439
110	0.256300	0.331390
120	0.226100	0.334290
130	0.246800	0.338176
140	0.230500	0.339353

Intended Use

This model is intended for translating English text related to audio products into Chinese. It can be used by professionals in the audio industry, technical writers, or anyone needing to translate such content.

Limitations and Bias

• The model's performance is best on text similar to the data it was trained on (audio product domain).
• It may not generalize well to other domains or highly colloquial language.
• As with any language model, there's a potential for biases present in the training data to be reflected in the output.