Update README.md

README.md

@@ -1,189 +1,279 @@
 ---
 base_model: meta-llama/Meta-Llama-3-8B-Instruct
 library_name: peft
+tags:
+- llama-3
+- lora
+- qlora
+- domain-adaptation
+- text-generation
+- turkish
+- tr
+- epdk
+- mevzuat
+- ogulcanakca
+language: tr
+license: llama3
 ---
 
-# Model Card for Model ID
+# Model Card for ogulcanakca/llama3-8b-epdk-domain-adapter-v1
 
-<!-- Provide a quick summary of what the model is/does. -->
+## Model Description
 
+This repository contains a PEFT LoRA adapter fine-tuned on the `meta-llama/Meta-Llama-3-8B-Instruct` base model for **Domain Adaptation**. The adapter was trained on a Turkish dataset comprising various documents related to the Turkish Energy Market Regulatory Authority (EPDK), such as legislation, laws, communiqués, and board decisions.
 
+The goal is to make the base Llama 3 8B Instruct model more familiar with the language, terminology, style, and concepts specific to the Turkish energy market regulation domain. When used with the base model, this adapter potentially offers improved understanding of texts from this domain and can generate more consistent outputs related to it.
 
-## Model Details
+**Note:** This repository does not contain a full model, only the LoRA weights that need to be applied to the base model. This adapter was trained using 4-bit quantization (QLoRA) via the PEFT library.
 
-### Model Description
+* **Developed by:** ogulcanakca
+* **Model type:** Transformer-based Causal Language Model (Llama 3) + PEFT LoRA adapter
+* **Language(s) (NLP):** Turkish (tr)
+* **License:** Llama 3 Community License ([https://llama.meta.com/llama3/license/](https://llama.meta.com/llama3/license/))
+* **Finetuned from model:** `meta-llama/Meta-Llama-3-8B-Instruct`
 
-<!-- Provide a longer summary of what this model is. -->
+### Model Sources
 
-
-
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
-
-### Model Sources [optional]
-
-<!-- Provide the basic links for the model. -->
-
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
+* **Repository:** [https://huggingface.co/ogulcanakca/llama3-8b-epdk-domain-adapter-v1](https://huggingface.co/ogulcanakca/llama3-8b-epdk-domain-adapter-v1)
+* **Base Model:** [https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct)
+* **Dataset Used (Raw Cleaned):** [https://huggingface.co/datasets/ogulcanakca/epdk_elektrik_piyasasi_mevzuat](https://huggingface.co/datasets/ogulcanakca/epdk_elektrik_piyasasi_mevzuat)
 
 ## Uses
 
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-
 ### Direct Use
 
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+This LoRA adapter can be used by loading it onto the `meta-llama/Meta-Llama-3-8B-Instruct` base model. It is expected that the base model will then better understand, summarize, or provide more consistent answers to prompts related to EPDK legislation. It can be used with the code example provided in the "How to Get Started with the Model" section below.
 
-[More Information Needed]
-
-### Downstream Use [optional]
+**Note:** As this adapter was trained for a very short trial period (200 steps), the domain adaptation effect is limited. More extensive training or evaluation is recommended before using it for critical tasks.
 
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+### Downstream Use
 
-[More Information Needed]
+This adapter could serve as a better starting point for further fine-tuning on more specific tasks related to EPDK regulations (e.g., a specialized Q&A model, text classification, summarization).
 
 ### Out-of-Scope Use
 
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-
-[More Information Needed]
+* Not suitable for use as a general-purpose chatbot (the base Instruct model is better for that).
+* Not expected to show significant improvement over the base model on topics outside Turkish energy market regulations.
+* May generate incorrect, incomplete, or outdated information. Should absolutely not be used for legal or financial advice.
+* Should not be used to generate harmful, unethical, discriminatory, or biased content.
 
 ## Bias, Risks, and Limitations
 
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-
-[More Information Needed]
+* This adapter inherits potential biases and risks present in the base `meta-llama/Meta-Llama-3-8B-Instruct` model.
+* The training data (EPDK documents) covers a specific time range and may contain outdated information. Information generated by the model may not be current.
+* The model may generate incorrect information ("hallucinations"), especially on topics not well-represented or contradictory in the training data.
+* Potential OCR errors or imperfections during the cleaning phase in the training data might affect the model's performance.
+* **Limited Training:** This adapter was trained for only **200 steps**, which is very short for domain adaptation. Therefore, the adaptation effect is minimal, and the model's knowledge or stylistic alignment within this domain may not be significantly improved. This version should be considered more of a "pipeline test".
 
 ### Recommendations
 
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+* The accuracy of information generated by the model should always be verified against reliable sources.
+* Users should be aware of the model's limitations and avoid using it for critical applications (e.g., legal interpretation).
+* For more reliable results, further training with significantly more steps/epochs is recommended.
 
 ## How to Get Started with the Model
 
-Use the code below to get started with the model.
-
-[More Information Needed]
-
+You need the `transformers`, `peft`, `accelerate`, and `bitsandbytes` libraries to use this LoRA adapter. The code below loads the base model with 4-bit quantization and applies this adapter:
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
+from peft import PeftModel
+import os
+
+# Ensure you are logged in to Hugging Face if the repo is private or requires Llama access
+# from huggingface_hub import login
+# login(token="YOUR_HF_TOKEN") # Or use huggingface-cli login
+
+# Base model ID
+base_model_id = "meta-llama/Meta-Llama-3-8B-Instruct"
+# This adapter's Hub ID
+adapter_id = "ogulcanakca/llama3-8b-epdk-domain-adapter-v1"
+
+# 4-bit Quantization Config
+bnb_config = BitsAndBytesConfig(
+    load_in_4bit=True,
+    bnb_4bit_quant_type="nf4",
+    bnb_4bit_compute_dtype=torch.bfloat16 # Use bfloat16 if supported
+)
+
+# Load the base model quantized
+print(f"Loading base model: {base_model_id}")
+base_model = AutoModelForCausalLM.from_pretrained(
+    base_model_id,
+    quantization_config=bnb_config,
+    device_map="auto",
+    torch_dtype=torch.bfloat16,
+    trust_remote_code=True # May be needed
+)
+
+# Load the tokenizer
+print("Loading tokenizer...")
+tokenizer = AutoTokenizer.from_pretrained(base_model_id)
+if tokenizer.pad_token is None:
+    tokenizer.pad_token = tokenizer.eos_token
+
+# Load the LoRA adapter onto the base model
+print(f"Loading adapter: {adapter_id}")
+model = PeftModel.from_pretrained(base_model, adapter_id)
+print("Adapter loaded successfully.")
+model.eval() # Set model to evaluation mode
+
+# Inference Example
+prompt = "EPDK'nın elektrik piyasasındaki temel görevleri nelerdir?" # Example prompt in Turkish
+messages = [
+    {"role": "system", "content": "You are a helpful assistant knowledgeable about Turkish energy market regulations."},
+    {"role": "user", "content": prompt}
+]
+
+# Llama 3 Instruct prompt format
+print("Generating response...")
+input_ids = tokenizer.apply_chat_template(
+    messages,
+    add_generation_prompt=True,
+    return_tensors="pt"
+).to(model.device)
+
+# Terminator IDs to stop generation
+terminators = [
+    tokenizer.eos_token_id,
+    tokenizer.convert_tokens_to_ids("<|eot_id|>")
+]
+
+# Generate text
+with torch.no_grad():
+    outputs = model.generate(
+        input_ids,
+        max_new_tokens=512, # Max new tokens to generate
+        eos_token_id=terminators,
+        do_sample=True, # For more creative outputs
+        temperature=0.6,
+        top_p=0.9,
+    )
+
+# Decode the generated output
+response_ids = outputs[0][input_ids.shape[-1]:]
+response_text = tokenizer.decode(response_ids, skip_special_tokens=True)
+
+print("\nModel Output:")
+print(response_text)
+```
 ## Training Details
 
 ### Training Data
 
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+The model was fine-tuned on cleaned text from the `ogulcanakca/epdk_elektrik_piyasasi_mevzuat` Hugging Face dataset. This dataset was derived from ~3300 various Turkish documents (PDF, DOCX, etc.) related to the Turkish Energy Market Regulatory Authority (EPDK), obtained through text extraction, OCR, and basic NLP cleaning processes.
 
-[More Information Needed]
+Dataset Link (Raw Cleaned): [https://huggingface.co/datasets/ogulcanakca/epdk_elektrik_piyasasi_mevzuat](https://huggingface.co/datasets/ogulcanakca/epdk_elektrik_piyasasi_mevzuat)
+
+Before training, the texts in this dataset were chunked using the `meta-llama/Meta-Llama-3-8B-Instruct` tokenizer into segments of **2048 tokens** with an overlap of **200 tokens**. The resulting `domain_adaptation_data.jsonl` file (or equivalent `Dataset` object) containing 31,271 chunks was used for training.
 
 ### Training Procedure
 
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+* **Fine-tuning Type:** Domain Adaptation (Causal LM objective).
+* **Technique:** QLoRA (4-bit NormalFloat Quantization + Low-Rank Adaptation) using the PEFT library.
+* **Libraries:** `transformers`, `peft`, `accelerate`, `bitsandbytes`, `datasets`.
 
 #### Preprocessing [optional]
 
-[More Information Needed]
-
+Cleaning steps mentioned above (whitespace, header/footer removal etc.) and tokenizer-based chunking were applied. `DataCollatorForLanguageModeling` was used during training.
 
 #### Training Hyperparameters
 
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+* **base_model:** `meta-llama/Meta-Llama-3-8B-Instruct`
+* **quantization:** `4-bit` (NF4, compute_dtype=bfloat16)
+* **lora_r:** 16
+* **lora_alpha:** 32
+* **lora_dropout:** 0.05
+* **lora_target_modules:** `["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]`
+* **learning_rate:** 2e-4 (0.0002)
+* **batch_size:** 1
+* **gradient_accumulation_steps:** 8 (Effective batch size: 8)
+* **optimizer:** Paged AdamW (8-bit)
+* **lr_scheduler_type:** cosine
+* **warmup_ratio:** 0.05
+* **max_steps:** 200 (Short trial run)
+* **seq_length:** 2048
+* **precision:** bf16 (mixed precision)
+* **gradient_checkpointing:** True
 
 #### Speeds, Sizes, Times [optional]
 
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-
-[More Information Needed]
+* Training was performed on a single GPU in Kaggle's free tier (likely T4 or P100 - exact type not logged).
+* The 200-step training run took approximately **8.5 hours**. Flash Attention 2 could not be used.
+* Final training loss: ~2.151
+* The saved LoRA adapter size is relatively small (typically tens or hundreds of MBs).
 
 ## Evaluation
 
-<!-- This section describes the evaluation protocols and provides the results. -->
+No formal evaluation metrics were calculated for this initial trial run (200 steps). The model's performance was only observed via the training loss. After more extensive training, standard language modeling metrics (like Perplexity) or performance on domain-specific downstream tasks could be evaluated.
 
 ### Testing Data, Factors & Metrics
 
 #### Testing Data
 
-<!-- This should link to a Dataset Card if possible. -->
-
-[More Information Needed]
+N/A
 
 #### Factors
 
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-
-[More Information Needed]
+N/A
 
 #### Metrics
 
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-
-[More Information Needed]
+N/A
 
 ### Results
 
-[More Information Needed]
+N/A
 
 #### Summary
 
-
+The short 200-step training demonstrated that the fine-tuning pipeline works, but was insufficient for significant domain adaptation. A slight decrease in training loss was observed.
 
 ## Model Examination [optional]
 
-<!-- Relevant interpretability work for the model goes here -->
-
 [More Information Needed]
 
 ## Environmental Impact
 
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
+* **Hardware Type:** Kaggle GPU (Likely T4 or P100 tier)
+* **Hours used:** ~8.5 hours (for 200 steps)
+* **Cloud Provider:** Kaggle (using Google Cloud infrastructure)
+* **Compute Region:** Unknown (Managed by Kaggle)
+* **Carbon Emitted:** Can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute), but estimating accurately requires specific GPU power consumption data, which is difficult to obtain for Kaggle free tiers.
 
 ## Technical Specifications [optional]
 
 ### Model Architecture and Objective
 
-[More Information Needed]
+The base model relies on the Llama 3 architecture (Transformer decoder-only). This adapter adds LoRA layers without changing the base model weights, trained with a Causal Language Modeling objective for domain adaptation.
 
 ### Compute Infrastructure
 
-[More Information Needed]
-
 #### Hardware
 
-[More Information Needed]
+Kaggle Notebook environment with a single NVIDIA GPU (likely T4 or P100).
 
 #### Software
 
-[More Information Needed]
+* PyTorch (`torch==2.5.1` recommended based on Kaggle env)
+* Transformers (`transformers`)
+* PEFT (`peft==0.15.2`)
+* Accelerate (`accelerate`)
+* BitsAndBytes (`bitsandbytes`)
+* Datasets (`datasets`)
+* Python 3.10/3.11 (Kaggle default)
 
 ## Citation [optional]
 
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-
-**BibTeX:**
-
-[More Information Needed]
-
-**APA:**
+Please cite the base Llama 3 model and the relevant PEFT/LoRA techniques:
 
-[More Information Needed]
+* **Llama 3:** [https://ai.meta.com/blog/meta-llama-3/](https://ai.meta.com/blog/meta-llama-3/)
+* **PEFT:** [https://github.com/huggingface/peft](https://github.com/huggingface/peft)
+* **LoRA:** Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., ... & Chen, W. (2021). LoRA: Low-Rank Adaptation of Large Language Models. arXiv preprint arXiv:2106.09685.
 
 ## Glossary [optional]
 
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-
 [More Information Needed]
 
 ## More Information [optional]
@@ -192,11 +282,16 @@ Carbon emissions can be estimated using the [Machine Learning Impact calculator]
 
 ## Model Card Authors [optional]
 
-[More Information Needed]
+ogulcanakca
 
 ## Model Card Contact
 
-[More Information Needed]
+ogulcanakca (Hugging Face user)
+
 ### Framework versions
 
-- PEFT 0.15.2
+* PEFT 0.15.2
+* Transformers [More Information Needed - Check with `!pip show transformers`]
+* PyTorch [More Information Needed - Check with `!pip show torch`]
+* Datasets [More Information Needed - Check with `!pip show datasets`]
+* Tokenizers [More Information Needed - Check with `!pip show tokenizers`]