Configuration Parsing
Warning:
In config.json: "quantization_config.bits" must be an integer
Configuration Parsing
Warning:
In config.json: "quantization_config.bits" must be greater than or equal to 2
Note
This was made to test extremely low quants. 70B is almost usable at 1.45 bpw. A custom strategy for quanting was used, for details check EXL3 repo. This is borderline usable, the goal was to run 70b model under 16gb of vram. If you have 16GB VRAM use 4k context and it will fit, and sort of work. very borderline, but can be used.
- Downloads last month
- 4
Inference Providers
NEW
This model isn't deployed by any Inference Provider.
🙋
Ask for provider support
Model tree for SicariusSicariiStuff/Negative_LLAMA_70B_EXL3_d_1.45bpw
Base model
meta-llama/Llama-3.1-70B
Finetuned
meta-llama/Llama-3.3-70B-Instruct
Finetuned
SicariusSicariiStuff/Negative_LLAMA_70B