README.md

---
# For reference on model card metadata, see the spec: https://github.com/huggingface/hub-docs/blob/main/modelcard.md?plain=1
# Doc / guide: https://huggingface.co/docs/hub/model-cards
datasets:
  - "pietrolesci/amazoncat-13k"
language:
  - en
library_name: transformers
license: other  # Base model (Meta Llama 3.2) is under the Llama 3.2 Community License
pipeline_tag: text-classification
tags:
  - multi-label
  - LoRA
  - QLoRA
  - bitsandbytes
  - decoder-only
  - llama-3.2-1b
  - peft
  - text-classification
base_model: meta-llama/Llama-3.2-1B
---

# Model Card for Amirhossein75/LLM-Decoder-Tuning-Text-Classification

> **One‑line summary:** Decoder‑only LLMs (e.g., Llama‑3.2‑1B) fine‑tuned for **multi‑label text classification** using **LoRA** adapters, with optional **4‑bit QLoRA** quantization for memory‑efficient training and inference. A clean CLI and YAML config make it easy to reproduce results and swap backbones.

This model card accompanies the repository **LLM‑Decoder‑Tuning‑Text‑Classification** and documents a practical recipe for using decoder‑only LLMs as strong multi‑label classifiers with parameter‑efficient fine‑tuning (PEFT).

> **Note:** This card describes a *training pipeline + example checkpoints*. If you push a specific checkpoint to the Hub, please fill in exact dataset splits, metrics, and license at upload time.

---

## Model Details

### Model Description

This project provides a **modular training & inference stack** for multi‑label text classification built on top of **Hugging Face Transformers** and **PEFT**. It adapts **decoder‑only** LLMs (tested with `meta-llama/Llama-3.2-1B`) using **LoRA** adapters, and optionally enables **4‑bit quantization** (QLoRA‑style) for reduced memory footprint during training and inference. The repository exposes a **single CLI** for train/eval/predict and a **YAML configuration** to control data paths, model choice, and hyperparameters.

- **Developed by:** Amirhossein Yousefi (GitHub: `amirhossein-yousefi`; Hugging Face: `Amirhossein75`)
- **Model type:** Decoder‑only causal LM with PEFT (LoRA) for multi‑label classification
- **Language(s):** English (evaluated on AmazonCat‑13K subset)
- **License:** The **base model** (`meta-llama/Llama-3.2-1B`) is under the **Llama 3.2 Community License**. The LoRA adapter you publish should declare its own license and acknowledge base‑model terms.
- **Finetuned from:** `meta-llama/Llama-3.2-1B` (foundation)

### Model Sources

- **Repository:** https://github.com/amirhossein-yousefi/LLM-Decoder-Tuning-Text-Classification
- **Model (Hub placeholder):** https://huggingface.co/Amirhossein75/LLM-Decoder-Tuning-Text-Classification
- **Background reading:**
  - LoRA: Low‑Rank Adaptation of Large Language Models (Hu et al., 2021)
  - QLoRA: Efficient Finetuning of Quantized LLMs (Dettmers et al., 2023)
  - PEFT documentation (Hugging Face)

---

## Uses

### Direct Use

- **Multi‑label text classification** on English corpora (e.g., product tagging, topic tagging, content routing).
- Inference via:
  - Provided **CLI** (`python -m llm_cls.cli predict --config ...`) producing JSONL predictions.
  - Hugging Face pipelines with base model + LoRA adapter loaded (see “How to Get Started”).

### Downstream Use 

- **Domain transfer:** Re‑train on your domain labels by pointing the YAML to your CSVs.
- **Backbone swap:** Replace `model.model_name` in the config to try other decoders or encoders (set `use_4bit=false` for encoders).

### Out‑of‑Scope Use

- Safety‑critical decisions without human oversight.
- Tasks requiring **extreme multilabel** scaling (e.g., hundreds of thousands of labels) without additional adaptation.
- Non‑English or code‑mixed data without validation.
- Any use that conflicts with the base model’s license and acceptable‑use policies.

---

## Bias, Risks, and Limitations

- **Dataset bias:** AmazonCat‑13K originates from product data; labels and text reflect marketplace distributions and may encode demographic or topical biases.
- **Multi‑label long tail:** Minority classes are harder; macro‑F1 often trails micro‑F1. Consider class weighting, augmentation, or threshold tuning.
- **Decoder framing:** Treating classification as generation can be sensitive to prompt/format and decoding thresholds.
- **License & usage constraints:** Ensure compliance with the Llama 3.2 Community License for derivatives and deployment.

### Recommendations

- Track **micro‑ and macro‑F1** and per‑class metrics.
- Use **threshold tuning** on validation to balance precision/recall per class.
- For memory‑constrained environments, prefer **4‑bit + LoRA**; otherwise disable 4‑bit on platforms without `bitsandbytes` support.

---

## How to Get Started with the Model

Below is an example of loading a base Llama model with a LoRA adapter for classification‑style inference. Replace `BASE_MODEL` and `ADAPTER_REPO` with your IDs.

```python
from transformers import AutoTokenizer, AutoModelForCausalLM, TextGenerationPipeline
from peft import PeftModel
import torch

BASE_MODEL = "meta-llama/Llama-3.2-1B"
ADAPTER_REPO = "Amirhossein75/LLM-Decoder-Tuning-Text-Classification"  # or your own adapter

tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL, use_fast=True)
base = AutoModelForCausalLM.from_pretrained(
    BASE_MODEL,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model = PeftModel.from_pretrained(base, ADAPTER_REPO)
model.eval()

# Simple prompt format for multi-label classification (adjust to your training format).
labels = ["books","movies_tv","music","pop","literature_fiction","movies","education_reference","rock","used_rental_textbooks","new"]
text = "A thrilling space opera with deep character arcs and rich world-building."

prompt = (
    "You are a classifier. Given the text, return a JSON list of applicable labels from this set: "
    + ", ".join(labels) + ".\n"
    + f"Text: {text}\nLabels: "
)

pipe = TextGenerationPipeline(model=model, tokenizer=tokenizer, device=0 if torch.cuda.is_available() else -1)
out = pipe(prompt, max_new_tokens=64, do_sample=False)
print(out[0]["generated_text"])
```

For **CLI usage**:

```bash
# Train
python -m llm_cls.cli train --config configs/default.yaml

# Predict
python -m llm_cls.cli predict   --config configs/default.yaml   --input_csv data/test.csv   --output_jsonl preds.jsonl
```

---

## Training Details

### Training Data

- **Dataset:** AmazonCat‑13K (example subset; top‑10 categories for illustration). If you use the full dataset, update CSV paths and label columns accordingly.
- **Format:** CSV with at least a text column and one or more label columns (multi‑label). Configure names in `configs/default.yaml`.
- **Splits:** Train / Validation / (Optional) Test; sample scripts are provided to create CSV splits.

### Training Procedure

#### Preprocessing

- Tokenization with the base model’s tokenizer.
- Optional script to prepare AmazonCat‑13K CSVs (see `split_amazon_13k_data.py` in the repo).

#### Training Hyperparameters (illustrative config)

- **Base model:** `meta-llama/Llama-3.2-1B`
- **Problem type:** `multi_label_classification`
- **Precision / quantization:** `use_4bit: true` (QLoRA‑style); `torch_dtype: bfloat16` for computation
- **LoRA:** `r=2`, `alpha=2`, `dropout=0.05`
- **LoRA target modules:** `["q_proj","k_proj","v_proj","o_proj","gate_proj","down_proj","up_proj"]`
- **Batch size:** `4` (with `gradient_accumulation_steps=8`)
- **Max length:** `1024`
- **Optimizer:** 8‑bit optimizer when quantized (`optim_8bit_when_4bit: true`)
- **Epochs:** up to `20` with early stopping (`patience=2`)
- **Metric for best model:** `f1_micro`

#### Speeds, Sizes, Times (example run)

- **Device:** NVIDIA GeForce RTX 3080 Ti Laptop GPU (16 GB VRAM)
- **Runtime:** ~1,310 seconds for the best run
- **Throughput:** ≈0.784 steps/s (≈24.9 samples/s) during training
- **Artifacts:** Reproducible outputs under `outputs/<model_name>/<dataset_name>/run_<i>/`

---

## Evaluation

### Testing Data, Factors & Metrics

- **Testing data:** Held‑out split from AmazonCat‑13K (example subset).
- **Factors:** Evaluate both **micro‑F1** (overall) and **macro‑F1** (per‑class average) to reflect long‑tail performance.
- **Metrics:** `f1_micro`, `f1_macro`, eval loss, throughput (steps/s, samples/s).
### Metrics

- **Best overall (micro-F1):** **0.830** at **5 epochs**  
- **Best minority‑class sensitivity (macro-F1):** **0.752** at **6 epochs**  
- **Average across 4 runs:** micro‑F1 **0.824**, macro‑F1 **0.741**, eval loss **0.161**  
- **Throughput:** train ≈ **0.784 steps/s** (**24.9 samples/s**) ; eval time ≈ **34.0s** per run.

> Interpretation: going from **4 → 5 epochs** gives the best **micro‑F1**; **6 epochs** squeezes out the top **macro‑F1**, hinting at slightly better coverage of minority classes with a tiny trade‑off in micro‑F1.

---
### 📈 Per‑run metrics
| Run | Epochs | Train Loss | Eval Loss | F1 (micro) | F1 (macro) | Train Time (s) | Train steps/s | Train samples/s | Eval Time (s) |
|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
| 1 | 4 | 1.400 | 0.157 | 0.824 | 0.738 | 1309.6 | 0.962 | 30.543 | 33.6 |
| 2 | 5 | 1.220 | 0.159 | 0.830 | 0.743 | 1640.3 | 0.768 | 24.385 | 34.0 |
| 3 | 6 | 1.063 | 0.162 | 0.826 | 0.752 | 1984.2 | 0.635 | 20.159 | 34.4 |
| 4 | 5 | 1.265 | 0.165 | 0.816 | 0.729 | 1639.3 | 0.769 | 24.401 | 34.0 |

<sub>*F1(micro)* aggregates decisions over all samples; *F1(macro)* averages per‑class F1 equally, highlighting minority‑class performance.</sub>

### Results (example)

- **Best micro‑F1:** `0.830` at 5 epochs
- **Best macro‑F1:** `0.752` at 6 epochs
- **Average across 4 runs:** micro‑F1 `0.824`, macro‑F1 `0.741`, eval loss `0.161`

#### Summary

Decoder‑only LLMs with **LoRA** adapters provide competitive multi‑label performance with small memory/compute budgets. Slightly longer training (5–6 epochs) can improve macro‑F1, capturing more minority labels with minimal micro‑F1 trade‑off.

---

## Model Examination 

- Inspect confidence/threshold curves per label to tune decision thresholds.
- Use error analysis on false negatives for long‑tail labels; consider reweighting or augmentation.

---

## Environmental Impact

- **Hardware Type:** Single laptop GPU (RTX 3080 Ti Laptop, 16 GB)
- **Hours used (example run):** ~0.36 hours
---

## Technical Specifications 

### Model Architecture and Objective

- **Architecture:** Decoder‑only Transformer (Llama 3.2 class), adapted via **LoRA**.
- **Objective:** Multi‑label classification formulated as conditional generation with sigmoid/thresholding for label decisions.

### Compute Infrastructure

#### Hardware

- Laptop with NVIDIA GeForce RTX 3080 Ti (laptop) GPU, 16 GB VRAM.

#### Software

- Python, PyTorch, Hugging Face Transformers, PEFT, (optional) bitsandbytes for 4‑bit.

---

## Citation 

If you use this work, please consider citing the following:

**BibTeX:**

```bibtex
@article{Hu2021LoRA,
  title={LoRA: Low-Rank Adaptation of Large Language Models},
  author={Edward J. Hu and Yelong Shen and Phillip Wallis and Zeyuan Allen-Zhu and Yuanzhi Li and Shean Wang and Lu Wang and Weizhu Chen},
  journal={arXiv preprint arXiv:2106.09685},
  year={2021}
}

@article{Dettmers2023QLoRA,
  title={QLoRA: Efficient Finetuning of Quantized LLMs},
  author={Tim Dettmers and Artidoro Pagnoni and Ari Holtzman and Luke Zettlemoyer},
  journal={arXiv preprint arXiv:2305.14314},
  year={2023}
}
```

**APA:**

- Hu, E. J., Shen, Y., Wallis, P., Allen‑Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021). *LoRA: Low‑Rank Adaptation of Large Language Models*. arXiv:2106.09685.
- Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023). *QLoRA: Efficient Finetuning of Quantized LLMs*. arXiv:2305.14314.

---

## Glossary 

- **LoRA:** Low‑Rank Adaptation; injects small trainable matrices into a frozen backbone to adapt it efficiently.
- **QLoRA (4‑bit):** Finetuning with the backbone quantized to 4‑bit precision, training only LoRA adapters.
- **Micro‑/Macro‑F1:** Micro aggregates over all instances; Macro averages over classes equally (sensitive to minority classes).

---

## More Information 

- The repo ships a minimal CLI (`llm_cls/cli.py`) and example YAML config (`configs/default.yaml`) to reproduce results.
- For non‑Linux environments or if `bitsandbytes` is unavailable, disable 4‑bit and train in standard precision.

---

## Model Card Authors 

- **Author/Maintainer:** Amirhossein Yousefi (`amirhossein-yousefi` / `Amirhossein75`)

## Model Card Contact

- Open an issue in the GitHub repository or contact the Hugging Face user `Amirhossein75`.