File size: 1,841 Bytes
83d534f
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
---
language:
- en
library_name: gguf
base_model: ChatDOC/OCRFlux-3B
tags:
- gguf
- quantized
- llama.cpp
license: apache-2.0
---

# ChatDOC/OCRFlux-3B - GGUF

This repository contains GGUF quantizations of [ChatDOC/OCRFlux-3B](https://huggingface.co/ChatDOC/OCRFlux-3B).

## About GGUF

GGUF is a quantization method that allows you to run large language models on consumer hardware by reducing the precision of the model weights.

## Files

| Filename | Quant type | File Size | Description |
| -------- | ---------- | --------- | ----------- |
| model-f16.gguf | f16 | Large | Original precision |
| model-q4_0.gguf | Q4_0 | Small | 4-bit quantization |
| model-q4_1.gguf | Q4_1 | Small | 4-bit quantization (higher quality) |
| model-q5_0.gguf | Q5_0 | Medium | 5-bit quantization |
| model-q5_1.gguf | Q5_1 | Medium | 5-bit quantization (higher quality) |
| model-q8_0.gguf | Q8_0 | Large | 8-bit quantization |

## Usage

You can use these models with llama.cpp or any other GGUF-compatible inference engine.

### llama.cpp

```bash
./llama-cli -m model-q4_0.gguf -p "Your prompt here"
```

### Python (using llama-cpp-python)

```python
from llama_cpp import Llama

llm = Llama(model_path="model-q4_0.gguf")
output = llm("Your prompt here", max_tokens=512)
print(output['choices'][0]['text'])
```

## Original Model

This is a quantized version of [ChatDOC/OCRFlux-3B](https://huggingface.co/ChatDOC/OCRFlux-3B). Please refer to the original model card for more information about the model's capabilities, training data, and usage guidelines.

## Conversion Details

- Converted using llama.cpp
- Original model downloaded from Hugging Face
- Multiple quantization levels provided for different use cases

## License

This model inherits the license from the original model. Please check the original model's license for usage terms.