File size: 1,405 Bytes
1c6d4c2 e6075e0 1c6d4c2 e6075e0 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 |
---
license: mit
train: false
inference: false
pipeline_tag: text-generation
---
This is an <a href="https://github.com/mobiusml/hqq/">HQQ</a> all 4-bit (group-size=128) quantized <a href="https://huggingface.co/microsoft/Phi-4-mini-instruct">Phi-4-mini-instruct</a> model, via <a href="https://github.com/pytorch/ao/">TorchAO</a> and <a href="https://github.com/mobiusml/gemlite/">GemLite</a> as a backend.
## Usage
First, install the dependecies:
```
pip install torchao;
pip install git+https://github.com/mobiusml/gemlite.git;
```
Then you can use the sample code below:
``` Python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, HqqConfig
model_id = "mobiuslabsgmbh/Phi-4-mini-instruct_gemlite-ao_a16w4_gs_128_pack_32bit"
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype=torch.float16,
device_map='cuda',
)
tokenizer = AutoTokenizer.from_pretrained(model_id)
```
Use in <a href="https://github.com/vllm-project/vllm/">vLLM</a>:
```Python
from vllm import LLM
from vllm.sampling_params import SamplingParams
model_id = "mobiuslabsgmbh/Phi-4-mini-instruct_gemlite-ao_a16w4_gs_128_pack_32bit"
llm = LLM(model=model_id, max_model_len=4096)
sampling_params = SamplingParams(temperature=0.8, top_p=0.95, max_tokens=1024)
outputs = llm.generate(["What is the capital of Germany?"], sampling_params)
print(outputs[0].outputs[0].text)
``` |