metadata
base_model: https://huggingface.co/Phind/Phind-CodeLlama-34B-v2
inference: false
license: llama2
model_creator: https://huggingface.co/Phind
model_name: Phind-Codellama-34B-v2
model_type: llama
quantized_by: latimar
Phind-CodeLlama-34B-v2 EXL2
Weights of Phind-CodeLlama-34B-v2 converted to EXL2 format.
Each separate quant is in a different branch, like in The Bloke's GPTQ repos.
export BRANCH=5_0-bpw-h8
git clone --single-branch --branch ${BRANCH} https://huggingface.co/latimar/Phind-Codellama-34B-v2-exl2
There are the following branches:
5_0-bpw-h8
5_0-bpw-h8-evol-ins
4_625-bpw-h6
4_4-bpw-h8
4_125-bpw-h6
3_8-bpw-h6
2_75-bpw-h6
2_55-bpw-h6
- Calibration dataset used for conversion: wikitext-v2
- Evaluation dataset used to calculate perplexity: wikitext-v2
- Calibration dataset used for conversion of
5_0-bpw-h8-evol-ins: wizardLM-evol-instruct_70k - Evaluation dataset used to calculate ppl for
Evol-Ins: : nikrosh-evol-instruct - When converting
4_4-bpw-h8quant, additional-mr 32arg was used.
PPL was measured with the test_inference.py exllamav2 script:
python test_inference.py -m /storage/models/LLaMA/EXL2/Phind-Codellama-34B-v2 -ed /storage/datasets/text/evol-instruct/nickrosh-evol-instruct-code-80k.parquet
| BPW | PPL on Wiki | PPL on Evol-Ins | File Size (Gb) |
|---|---|---|---|
| 2.55-h6 | 11.0310 | 2.4542 | 10.56 |
| 2.75-h6 | 9.7902 | 2.2888 | 11.33 |
| 3.8-h6 | 6.7293 | 2.0724 | 15.37 |
| 4.125-h6 | 6.6713 | 2.0617 | 16.65 |
| 4.4-h8 | 6.6487 | 2.0509 | 17.76 |
| 4.625-h6 | 6.6576 | 2.0459 | 18.58 |
| 5.0-h8 | 6.6379 | 2.0419 | 20.09 |
| 5.0-h8-ev | 6.7785 | 2.0445 | 20.09 |