|
|
--- |
|
|
base_model: aihub-app/zyte-1B |
|
|
inference: false |
|
|
model_type: llama |
|
|
prompt_template: | |
|
|
<|system|> You are a helpful AI assistant.</s> |
|
|
<|user|>{prompt}</s> |
|
|
<|assistant|> |
|
|
quantized_by: mwitiderrick |
|
|
tags: |
|
|
- deepsparse |
|
|
--- |
|
|
## zyte-1B - DeepSparse |
|
|
This repo contains model files for [zyte-1B ](https://huggingface.co/aihub-app/zyte-1B) optimized for [DeepSparse](https://github.com/neuralmagic/deepsparse), a CPU inference runtime for sparse models. |
|
|
|
|
|
This model was quantized and pruned with [SparseGPT](https://arxiv.org/abs/2301.00774), using [SparseML](https://github.com/neuralmagic/sparseml). |
|
|
|
|
|
## Inference |
|
|
Install [DeepSparse LLM](https://github.com/neuralmagic/deepsparse) for fast inference on CPUs: |
|
|
```bash |
|
|
pip install deepsparse-nightly[llm] |
|
|
``` |
|
|
Run in a [Python pipeline](https://github.com/neuralmagic/deepsparse/blob/main/docs/llms/text-generation-pipeline.md): |
|
|
```python |
|
|
from deepsparse import TextGeneration |
|
|
|
|
|
prompt = "How to make banana bread?" |
|
|
formatted_prompt = f"<|system|> You are a helpful AI assistant.</s><|user|>{prompt}</s><|assistant|>" |
|
|
model = TextGeneration(model_path="hf:nm-testing/zyte-1B-pruned50-quant-dsnm-testing/zyte-1B-pruned50-quant-ds") |
|
|
print(model(formatted_prompt, max_new_tokens=200, repetition_penalty=1.1, do_sample=True).generations[0].text) |
|
|
|
|
|
""" |
|
|
Absolutely! Here's a recipe for banana bread, which can be made using several fresh bananas: |
|
|
Banana bread-making recipe 1 cup of oatmeal |
|
|
1 riped banan (approx 10-12 ripe unripened harun); |
|
|
35 grams of unsalted soaked soy flour; Into 8 cups (with two layers) oats (oats as little as one teaspoon/497grts. ), and sugar. The next ingredience is: Soaked rice grains or any kind flat, allso in granulates soled and leaching dried grains, together with shelling grains allso in granulates on the top level are needed to help softening, stirring and preparing the pancakers at 20 degrees Celronic/56° |
|
|
|
|
|
""" |
|
|
``` |
|
|
## Prompt template |
|
|
|
|
|
``` |
|
|
<|system|> You are a helpful AI assistant.</s> |
|
|
<|user|>{prompt}</s> |
|
|
<|assistant|> |
|
|
|
|
|
``` |
|
|
## Sparsification |
|
|
For details on how this model was sparsified, see the `recipe.yaml` in this repo and follow the instructions below. |
|
|
|
|
|
```bash |
|
|
git clone https://github.com/neuralmagic/sparseml |
|
|
pip install -e "sparseml[transformers]" |
|
|
python sparseml/src/sparseml/transformers/sparsification/obcq/obcq.py aihub-app/zyte-1B open_platypus --recipe recipe.yaml --save True |
|
|
python sparseml/src/sparseml/transformers/sparsification/obcq/export.py --task text-generation --model_path obcq_deployment |
|
|
cp deployment/model.onnx deployment/model-orig.onnx |
|
|
``` |
|
|
Run this kv-cache injection to speed up the model at inference by caching the Key and Value states: |
|
|
```python |
|
|
import os |
|
|
import onnx |
|
|
from sparseml.exporters.kv_cache_injector import KeyValueCacheInjector |
|
|
input_file = "deployment/model-orig.onnx" |
|
|
output_file = "deployment/model.onnx" |
|
|
model = onnx.load(input_file, load_external_data=False) |
|
|
model = KeyValueCacheInjector(model_path=os.path.dirname(input_file)).apply(model) |
|
|
onnx.save(model, output_file) |
|
|
print(f"Modified model saved to: {output_file}") |
|
|
``` |
|
|
Follow the instructions on our [One Shot With SparseML](https://github.com/neuralmagic/sparseml/tree/main/src/sparseml/transformers/sparsification/obcq) page for a step-by-step guide for performing one-shot quantization of large language models. |
|
|
## Slack |
|
|
|
|
|
For further support, and discussions on these models and AI in general, join [Neural Magic's Slack Community](https://join.slack.com/t/discuss-neuralmagic/shared_invite/zt-q1a1cnvo-YBoICSIw3L1dmQpjBeDurQ) |
