tardellirs/colibri-embed-ptbr-GGUF

🐦 Colibri — GGUF (llama.cpp)

GGUF builds of tardellirs/colibri-embed-ptbr, a ~157M-parameter Brazilian-Portuguese embedding model, for llama.cpp (llama-server, llama-embedding) and the runtimes built on it.

The two Dense projection layers of the SentenceTransformer pipeline are inside the file, and the tokenizer is the model's own 64k BPE vocabulary, so the token ids are exactly those of the HF model. Every file gives the same 768-dim embeddings as the original, up to quantization.

File	Size	Peak RAM	Latency (p50)	Throughput	Cosine vs HF fp32 (mean / min)
colibri-embed-ptbr-f16.gguf	315 MB	367 MB	36 ms	32 sent/s	1.0000 / 1.0000
colibri-embed-ptbr-Q8_0.gguf	169 MB	229 MB	35 ms	31 sent/s	0.9996 / 0.9995
colibri-embed-ptbr-Q6_K.gguf	137 MB	199 MB	42 ms	33 sent/s	0.9978 / 0.9958
colibri-embed-ptbr-Q5_K_M.gguf	122 MB	185 MB	39 ms	31 sent/s	0.9950 / 0.9930
colibri-embed-ptbr-Q4_K_M.gguf	111 MB	217 MB	31 ms	38 sent/s	0.9857 / 0.9802

Q8_0 is the one to use: indistinguishable from the original at about a quarter of the PyTorch RAM. Q6_K and Q5_K_M are still close; Q4_K_M is the smallest file but loses the most, and uses more RAM than Q5_K_M.

Measured on a Verda CPU.4V.16G (4 vCPU AMD EPYC, 16 GB), llama.cpp a4d4f2f (2026-11-09), llama-server with 4 threads. Peak RAM is the server process's peak resident memory. Latency is one sentence per request; throughput is 512 sentences sent 32 per request. Cosine is against the HF model in fp32 on 30 Brazilian-Portuguese sentences.

Usage

llama-server -m colibri-embed-ptbr-Q8_0.gguf --embeddings --pooling mean -c 2048 -ub 2048 -b 2048

import requests

def embed(texts):
    r = requests.post("http://127.0.0.1:8080/v1/embeddings", json={"input": texts})
    return [d["embedding"] for d in r.json()["data"]]

# llama.cpp doesn't add the EmbeddingGemma prompts: put them in yourself
q = embed(["task: search result | query: Como declarar imposto de renda de aluguel?"])
d = embed(["title: none | text: Rendimentos de aluguel devem ser informados na ficha de Rendimentos Tributáveis..."])

Other prompts (classification, clustering, similarity, …) are listed in the original model's config_sentence_transformers.json. Embeddings come back L2-normalized, so the dot product is the cosine.

How these were made

From the original model with two fixes to its tokenizer (now also in the original repo), then convert_hf_to_gguf.py --sentence-transformers-dense-modules and llama-quantize:

1. The trim left 6,246 unused added tokens (<unused0…>, <image_soft_token>, …) with ids past the 64,000-row embedding matrix. They were removed; no text tokenizes differently.
2. The trimmed vocabulary changes the pre-tokenizer hash llama.cpp uses to recognize the tokenizer, so the hash was registered as Gemma's pre-tokenizer, which is unchanged (embedding-vocab-trimmer, gguf_bpe_patch.py).