File size: 1,681 Bytes
de87165 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 |
---
license: apache-2.0
base_model:
- nvidia/Mistral-NeMo-Minitron-8B-Instruct
tags:
- OpenArc
- OpenVINO
- Intel
---
My Project [OpenArc](https://github.com/SearchSavior/OpenArc), an inference engine for OpenVINO, now supports this model and serves inference over OpenAI compatible endpoints for text to text *and* text with vision!
We have a growing Discord community of others interested in using Intel for AI/ML.
[](https://discord.gg/maMY7QjG)
- Find documentation on the Optimum-CLI export process [here](https://huggingface.co/docs/optimum/main/en/intel/openvino/export)
- Use my HF space [Echo9Zulu/Optimum-CLI-Tool_tool](https://huggingface.co/spaces/Echo9Zulu/Optimum-CLI-Tool_tool) to build commands and execute locally
## This repo contains OpenVINO quantizations of nvidia/Mistral-NeMo-Minitron-8B-Instruct.
I reccomend starting with **Mistral-NeMo-Minitron-8B-Instruct-int4_asym-awq-se-ov**
To download individual models from this repo use the provided snippet:
```
from huggingface_hub import snapshot_download
repo_id = "Echo9Zulu/Mistral-NeMo-Minitron-8B-Instruct-OpenVINO"
# Choose the weights you want
repo_directory = "Mistral-NeMo-Minitron-8B-Instruct-int4_asym-awq-se-ov"
# Where you want to save it
local_dir = "./Echo9Zulu_Mistral-NeMo-Minitron-8B-Instruct/Mistral-NeMo-Minitron-8B-Instruct-int4_asym-awq-se-ov"
snapshot_download(
repo_id=repo_id,
allow_patterns=[f"{repo_directory}/*"],
local_dir=local_dir,
local_dir_use_symlinks=True
)
print("Download complete!")
``` |