ogulcanakca
/

llama3-8b-epdk-domain-adapter-v1

Text Generation

domain-adaptation

Model card Files Files and versions Community

ogulcanakca commited on Apr 25

Commit

4bb4a4a

·

verified ·

1 Parent(s): 70bd2ae

Update README.md

Files changed (1) hide show

README.md +3 -7

README.md CHANGED Viewed

@@ -174,7 +174,7 @@ Before training, the texts in this dataset were chunked using the `meta-llama/Me
 * **Technique:** QLoRA (4-bit NormalFloat Quantization + Low-Rank Adaptation) using the PEFT library.
 * **Libraries:** `transformers`, `peft`, `accelerate`, `bitsandbytes`, `datasets`.
-#### Preprocessing [optional]
 Cleaning steps mentioned above (whitespace, header/footer removal etc.) and tokenizer-based chunking were applied. `DataCollatorForLanguageModeling` was used during training.
@@ -197,7 +197,7 @@ Cleaning steps mentioned above (whitespace, header/footer removal etc.) and toke
 * **precision:** bf16 (mixed precision)
 * **gradient_checkpointing:** True
-#### Speeds, Sizes, Times [optional]
 * Training was performed on a single GPU in Kaggle's free tier (likely T4 or P100 - exact type not logged).
 * The 200-step training run took approximately **8.5 hours**. Flash Attention 2 could not be used.
@@ -230,10 +230,6 @@ N/A
 The short 200-step training demonstrated that the fine-tuning pipeline works, but was insufficient for significant domain adaptation. A slight decrease in training loss was observed.
-## Model Examination [optional]
-[More Information Needed]
 ## Environmental Impact
 * **Hardware Type:** Kaggle GPU (Likely T4 or P100 tier)
@@ -242,7 +238,7 @@ The short 200-step training demonstrated that the fine-tuning pipeline works, bu
 * **Compute Region:** Unknown (Managed by Kaggle)
 * **Carbon Emitted:** Can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute), but estimating accurately requires specific GPU power consumption data, which is difficult to obtain for Kaggle free tiers.
-## Technical Specifications [optional]
 ### Model Architecture and Objective

 * **Technique:** QLoRA (4-bit NormalFloat Quantization + Low-Rank Adaptation) using the PEFT library.
 * **Libraries:** `transformers`, `peft`, `accelerate`, `bitsandbytes`, `datasets`.
+#### Preprocessing
 Cleaning steps mentioned above (whitespace, header/footer removal etc.) and tokenizer-based chunking were applied. `DataCollatorForLanguageModeling` was used during training.
 * **precision:** bf16 (mixed precision)
 * **gradient_checkpointing:** True
+#### Speeds, Sizes, Times
 * Training was performed on a single GPU in Kaggle's free tier (likely T4 or P100 - exact type not logged).
 * The 200-step training run took approximately **8.5 hours**. Flash Attention 2 could not be used.
 The short 200-step training demonstrated that the fine-tuning pipeline works, but was insufficient for significant domain adaptation. A slight decrease in training loss was observed.
 ## Environmental Impact
 * **Hardware Type:** Kaggle GPU (Likely T4 or P100 tier)
 * **Compute Region:** Unknown (Managed by Kaggle)
 * **Carbon Emitted:** Can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute), but estimating accurately requires specific GPU power consumption data, which is difficult to obtain for Kaggle free tiers.
+## Technical Specifications
 ### Model Architecture and Objective