Initial GPTQ model commit
Browse files
README.md
CHANGED
|
@@ -230,7 +230,7 @@ Thank you to all my generous patrons and donaters!
|
|
| 230 |
|
| 231 |
# OpenOrca x OpenChat - Preview2 - 13B
|
| 232 |
|
| 233 |
-
We have used our own [OpenOrca dataset](https://huggingface.co/datasets/Open-Orca/OpenOrca) to fine-tune Llama2-13B using [OpenChat](https://huggingface.co/openchat) packing
|
| 234 |
This dataset is our attempt to reproduce the dataset generated for Microsoft Research's [Orca Paper](https://arxiv.org/abs/2306.02707).
|
| 235 |
|
| 236 |
This second preview release is trained on a curated filtered subset of most of our GPT-4 augmented data.
|
|
@@ -242,7 +242,7 @@ As well, this is done with <1/10th the compute requirement and using <20% of the
|
|
| 242 |
We have run extensive evaluations internally and expect this model to **place number 1** on both the HuggingFaceH4 Open LLM Leaderboard and the GPT4ALL Leaderboard for 13B models.
|
| 243 |
|
| 244 |
"One" of [OpenChat](https://huggingface.co/openchat) has joined our team, and we'd like to provide special thanks for their training of this model!
|
| 245 |
-
We have utilized OpenChat
|
| 246 |
This has significantly reduced training time, with efficiency improvement of 3-10X over traditional methods.
|
| 247 |
|
| 248 |
|
|
|
|
| 230 |
|
| 231 |
# OpenOrca x OpenChat - Preview2 - 13B
|
| 232 |
|
| 233 |
+
We have used our own [OpenOrca dataset](https://huggingface.co/datasets/Open-Orca/OpenOrca) to fine-tune Llama2-13B using [OpenChat](https://huggingface.co/openchat) packing.
|
| 234 |
This dataset is our attempt to reproduce the dataset generated for Microsoft Research's [Orca Paper](https://arxiv.org/abs/2306.02707).
|
| 235 |
|
| 236 |
This second preview release is trained on a curated filtered subset of most of our GPT-4 augmented data.
|
|
|
|
| 242 |
We have run extensive evaluations internally and expect this model to **place number 1** on both the HuggingFaceH4 Open LLM Leaderboard and the GPT4ALL Leaderboard for 13B models.
|
| 243 |
|
| 244 |
"One" of [OpenChat](https://huggingface.co/openchat) has joined our team, and we'd like to provide special thanks for their training of this model!
|
| 245 |
+
We have utilized OpenChat [MultiPack algorithm](https://github.com/imoneoi/multipack_sampler) which achieves 99.85% bin-packing efficiency on our dataset.
|
| 246 |
This has significantly reduced training time, with efficiency improvement of 3-10X over traditional methods.
|
| 247 |
|
| 248 |
|