Sang-Buster/atc-llama-adapters

ATC Communication Expert Model (LoRA Adapters)

A specialized set of LoRA adapters fine-tuned for improving and analyzing Air Traffic Control (ATC) communications, extracting relevant information from raw transcripts.

Model Details

Model Description

These adapters fine-tune the meta-llama/Llama-3.2-3B-Instruct model to specialize in processing Air Traffic Control communications. When applied to the base model, it can:

• Improve raw ATC transcripts with proper punctuation and formatting
• Identify communication intentions (pilot requests, ATC instructions, etc.)
• Extract key information such as flight numbers, altitudes, headings, and other numerical data
• Analyze speaker roles and communication patterns

The adapters were created using LoRA (Low-Rank Adaptation) with PEFT (Parameter-Efficient Fine-Tuning) techniques to efficiently adapt the Llama 3B model to this specialized domain.

• Developed by: Sang-Buster
• Model type: LoRA adapters for meta-llama/Llama-3.2-3B-Instruct
• Language(s): English, specialized for ATC terminology
• License: Same as the base model
• Finetuned from model: meta-llama/Llama-3.2-3B-Instruct

Uses

Direct Use

These adapters are intended for:

• Transcribing and formatting raw ATC communications
• Training ATC communication skills
• Analyzing ATC communication patterns
• Extracting structured data from ATC communications
• Educational purposes for those learning ATC communication protocols

Downstream Use

The model can be integrated into:

• Air traffic management training systems
• Communication analysis tools
• ATC transcript post-processing pipelines
• Aviation safety monitoring systems
• Radio communication enhancement systems

Out-of-Scope Use

This model is not suitable for:

• Real-time ATC operations or safety-critical decision-making
• Full language translation (it's specialized for ATC terminology only)
• General language processing outside the ATC domain
• Any application where model errors could impact flight safety

Bias, Risks, and Limitations

• The model is specialized for ATC communications and may not perform well on general text
• It may have limitations with accents or non-standard ATC phraseology
• Performance depends on audio transcription quality for real-world applications
• Not intended for safety-critical applications without human verification
• May have biases based on the training data distribution

Recommendations

• Always have human verification for safety-critical applications
• Use in conjunction with standard ATC protocols, not as a replacement
• Provide clear domain context for optimal performance
• Test thoroughly with diverse ATC communications before deployment
• Consider fine-tuning further on your specific ATC subdomain if needed

How to Get Started with the Model

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

# Load the base model
base_model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.2-3B-Instruct",
    torch_dtype="auto",
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.2-3B-Instruct")

# Load the adapter
model = PeftModel.from_pretrained(base_model, "atc_llama")

# Process an ATC message
instruction = "As an ATC communication expert, improve this transcript and analyze its intentions and data."
message = "southwest five niner two turn left heading three four zero descend and maintain flight level two five zero"

prompt = f"<|begin_of_text|><|header_start|>user<|header_end|>\n\n{instruction}\n\nOriginal: {message}<|eot|><|header_start|>assistant<|header_end|>\n\n"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

# Generate improved transcript and analysis
outputs = model.generate(**inputs, max_new_tokens=512, do_sample=False)
response = tokenizer.decode(outputs[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True)
print(response)

Training Details

Training Data

The model was trained on a dataset of ATC communications with:

• Original raw transcripts
• Properly punctuated and formatted versions
• Annotated intentions (PSC, PSR, PRP, PRQ, PRB, PAC, ASC, AGI, ACR, END)
• Extracted numerical data (altitudes, headings, flight numbers, etc.)
• Speaker and listener information

Training Procedure

The model was fine-tuned using LoRA with the following approach:

• Parameter-efficient fine-tuning using PEFT
• LoRA applied to key attention layers (q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj)
• Optimized with Unsloth for efficiency

Training Hyperparameters

• Base model: meta-llama/Llama-3.2-3B-Instruct
• LoRA rank: 16
• LoRA alpha: 16
• Learning rate: 0.0002
• Batch size: 4
• Gradient accumulation steps: 4
• Epochs: 3
• Warmup ratio: 0.03
• Max sequence length: 2048
• Training regime: BF16 mixed precision where available, FP16 otherwise
• Optimizer: AdamW 8-bit

Evaluation

Testing

The adapters should be tested on diverse ATC communications, including:

• Clearances and instructions
• Pilot requests and reports
• Emergency communications
• Different accents and speaking patterns

Technical Specifications

Model Architecture and Objective

• Base architecture: meta-llama/Llama-3.2-3B-Instruct
• Fine-tuning method: LoRA with PEFT
• Optimization library: Unsloth
• Training objective: Improving and analyzing ATC communications

Compute Infrastructure

• Framework versions:
  • PEFT (compatible with the base model)
  • Unsloth (for efficient LoRA fine-tuning)
  • Transformers (compatible with the base model)
  • PyTorch (with BF16 support where available)