| [English](./non-lore-README.md) | [简体中文](./non-lore-README-cn.md) | |
| # Bigger Body 70b | |
| A roleplay-focused ~~pseudo full-finetune~~ qlora finetune of Llama 3.3 70b. | |
| The successor to the Ink series. | |
| ## Dataset | |
| The Bigger Body (referred to as Ink v2.1, because that's still the internal name) mix is absolutely disgusting. It's even more cursed than the original Ink mix. | |
| <details> | |
| <summary>(Public) Original Datasets</summary> | |
| <ul> | |
| <li><a href="https://huggingface.co/datasets/Fizzarolli/limarp-processed">Fizzarolli/limarp-processed</a></li> | |
| <li><a href="https://huggingface.co/datasets/Norquinal/OpenCAI">Norquinal/OpenCAI</a> - <code>two_users</code> split</li> | |
| <li><a href="https://huggingface.co/datasets/allura-org/Celeste1.x-data-mixture">allura-org/Celeste1.x-data-mixture</a></li> | |
| <li><a href="https://huggingface.co/datasets/mapsila/PIPPA-ShareGPT-formatted-named">mapsila/PIPPA-ShareGPT-formatted-named</a></li> | |
| <li><a href="https://huggingface.co/datasets/allenai/tulu-3-sft-personas-instruction-following">allenai/tulu-3-sft-personas-instruction-following</a></li> | |
| <li><a href="https://huggingface.co/datasets/readmehay/medical-01-reasoning-SFT-json">readmehay/medical-01-reasoning-SFT-json</a></li> | |
| <li><a href="https://huggingface.co/datasets/LooksJuicy/ruozhiba">LooksJuicy/ruozhiba</a></li> | |
| <li><a href="https://huggingface.co/datasets/shibing624/roleplay-zh-sharegpt-gpt4-data">shibing624/roleplay-zh-sharegpt-gpt4-data</a></li> | |
| <li><a href="https://huggingface.co/datasets/CausalLM/Retrieval-SFT-Chat">CausalLM/Retrieval-SFT-Chat</a></li> | |
| <li><a href="https://huggingface.co/datasets/ToastyPigeon/fujin-filtered-instruct">ToastyPigeon/fujin-filtered-instruct</a></li> | |
| </ul> | |
| </details> | |
| ## Quants | |
| - [bartowski's imatrix ggufs](https://huggingface.co/bartowski/allura-org_Bigger-Body-70b-GGUF) | |
| - [readyart's exl2 quants](https://huggingface.co/collections/ReadyArt/bigger-body-70b-exl2-67d56b31546412c770930887) | |
| ## Recommended Settings | |
| Chat template: Llama 3 Instruct | |
| Recommended samplers (not the be-all-end-all, try some on your own!): | |
| - I have literally no idea. you're on your own. | |
| ## Hyperparams | |
| ### General | |
| - Epochs = 2 | |
| - LR = 1e-5 | |
| - LR Scheduler = [REX](https://github.com/IvanVassi/REX_LR) | |
| - Optimizer = [CAME](https://github.com/yangluo7/CAME) | |
| - Effective batch size = 16 | |
| - Weight Decay = 0.01 | |
| - Warmup steps = 0 | |
| - Total steps = 920 | |
| - Quantization = 4bit | |
| ## LoRA | |
| - LoRA rank = 16 | |
| - LoRA alpha = 32 | |
| - LoRA dropout = 0.25 | |
| ## Credits | |
| Humongous thanks to the people who created the data. | |
| Big thanks to all Allura members for testing and emotional support ilya /platonic |