---
base_model: Intelligent-Internet/II-Medical-8B-1706
library_name: transformers
---
## About

Static quants of https://huggingface.co/Intelligent-Internet/II-Medical-8B-1706

## Provided Quants

(sorted by size, not necessarily quality)

| Link | Type | Size/GB | Notes |
|:-----|:-----|--------:|:------|
| [GGUF](https://huggingface.co/Intelligent-Internet/II-Medical-8B-1706-GGUF/resolve/main/II-Medical-8B-1706.Q2_K.gguf) | Q2_K | 3.4 |  |
| [GGUF](https://huggingface.co/Intelligent-Internet/II-Medical-8B-1706-GGUF/resolve/main/II-Medical-8B-1706.Q3_K_S.gguf) | Q3_K_S | 3.9 |  |
| [GGUF](https://huggingface.co/Intelligent-Internet/II-Medical-8B-1706-GGUF/resolve/main/II-Medical-8B-1706.Q3_K_M.gguf) | Q3_K_M | 4.2 |  |
| [GGUF](https://huggingface.co/Intelligent-Internet/II-Medical-8B-1706-GGUF/resolve/main/II-Medical-8B-1706.Q3_K_L.gguf) | Q3_K_L | 4.5 |  |
| [GGUF](https://huggingface.co/Intelligent-Internet/II-Medical-8B-1706-GGUF/resolve/main/II-Medical-8B-1706.Q4_K_S.gguf) | Q4_K_S | 4.9 |  |
| [GGUF](https://huggingface.co/Intelligent-Internet/II-Medical-8B-1706-GGUF/resolve/main/II-Medical-8B-1706.Q4_K_M.gguf) | Q4_K_M | 5.1 |  |
| [GGUF](https://huggingface.co/Intelligent-Internet/II-Medical-8B-1706-GGUF/resolve/main/II-Medical-8B-1706.Q5_K_S.gguf) | Q5_K_S | 5.8 |  |
| [GGUF](https://huggingface.co/Intelligent-Internet/II-Medical-8B-1706-GGUF/resolve/main/II-Medical-8B-1706.Q5_K_M.gguf) | Q5_K_M | 6.0 |  |
| [GGUF](https://huggingface.co/Intelligent-Internet/II-Medical-8B-1706-GGUF/resolve/main/II-Medical-8B-1706.Q6_K.gguf) | Q6_K | 6.8 |  |
| [GGUF](https://huggingface.co/Intelligent-Internet/II-Medical-8B-1706-GGUF/resolve/main/II-Medical-8B-1706.Q8_0.gguf) | Q8_0 | 8.8 |  |
| [GGUF](https://huggingface.co/Intelligent-Internet/II-Medical-8B-1706-GGUF/resolve/main/II-Medical-8B-1706.f16.gguf) | F16 | 16.5 |  |


## Running II-Medical-8B-1706
```bash
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DLLAMA_CURL=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-gguf-split
cp llama.cpp/build/bin/llama-* llama.cpp
```

```bash
./llama.cpp/llama-cli \
    --model path/to/II-Medical-8B-1706.Q8_0.gguf \
    --threads 32 \
    --ctx-size 16384 \
    --n-gpu-layers 99 \
    -ot ".ffn_.*_exps.=CPU" \
    --seed 3407 \
    --prio 3 \
    --temp 0.6 \
    --min-p 0.01 \
    --top-p 0.9 \
    -no-cnv \
    --prompt "<|im_start|>user\nI'm feeling unwell. Please help. <|im_end|>\n<|im_start|>assistant\n
```