Instructions to use farbodtavakkoli/OTel-2.0-Decision-26B-A4B-4bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use farbodtavakkoli/OTel-2.0-Decision-26B-A4B-4bit with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf farbodtavakkoli/OTel-2.0-Decision-26B-A4B-4bit:Q4_K_M # Run inference directly in the terminal: llama cli -hf farbodtavakkoli/OTel-2.0-Decision-26B-A4B-4bit:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf farbodtavakkoli/OTel-2.0-Decision-26B-A4B-4bit:Q4_K_M # Run inference directly in the terminal: llama cli -hf farbodtavakkoli/OTel-2.0-Decision-26B-A4B-4bit:Q4_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf farbodtavakkoli/OTel-2.0-Decision-26B-A4B-4bit:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf farbodtavakkoli/OTel-2.0-Decision-26B-A4B-4bit:Q4_K_M
Use Docker
docker model run hf.co/farbodtavakkoli/OTel-2.0-Decision-26B-A4B-4bit:Q4_K_M
- Llama new
- LM Studio
- Jan
- vLLM
How to use farbodtavakkoli/OTel-2.0-Decision-26B-A4B-4bit with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "farbodtavakkoli/OTel-2.0-Decision-26B-A4B-4bit" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "farbodtavakkoli/OTel-2.0-Decision-26B-A4B-4bit", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/farbodtavakkoli/OTel-2.0-Decision-26B-A4B-4bit:Q4_K_M
- Ollama
How to use farbodtavakkoli/OTel-2.0-Decision-26B-A4B-4bit with Ollama:
ollama run hf.co/farbodtavakkoli/OTel-2.0-Decision-26B-A4B-4bit:Q4_K_M
- Unsloth Desktop
- Pi
How to use farbodtavakkoli/OTel-2.0-Decision-26B-A4B-4bit with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf farbodtavakkoli/OTel-2.0-Decision-26B-A4B-4bit:Q4_K_M
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "farbodtavakkoli/OTel-2.0-Decision-26B-A4B-4bit:Q4_K_M" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use farbodtavakkoli/OTel-2.0-Decision-26B-A4B-4bit with Docker Model Runner:
docker model run hf.co/farbodtavakkoli/OTel-2.0-Decision-26B-A4B-4bit:Q4_K_M
- Lemonade
How to use farbodtavakkoli/OTel-2.0-Decision-26B-A4B-4bit with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull farbodtavakkoli/OTel-2.0-Decision-26B-A4B-4bit:Q4_K_M
Run and chat with the model
lemonade run user.OTel-2.0-Decision-26B-A4B-4bit-Q4_K_M
List all available models
lemonade list
- Hermes Agent
How to use farbodtavakkoli/OTel-2.0-Decision-26B-A4B-4bit with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf farbodtavakkoli/OTel-2.0-Decision-26B-A4B-4bit:Q4_K_M
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default farbodtavakkoli/OTel-2.0-Decision-26B-A4B-4bit:Q4_K_M
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use farbodtavakkoli/OTel-2.0-Decision-26B-A4B-4bit with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf farbodtavakkoli/OTel-2.0-Decision-26B-A4B-4bit:Q4_K_M
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "farbodtavakkoli/OTel-2.0-Decision-26B-A4B-4bit:Q4_K_M" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
OTel-2.0-Decision-26B-A4B-4bit
Versioning notice: This card describes the current OTel-2.0-Decision-26B-A4B-4bit checkpoint. The model weights may be updated over time. For reproducible evaluation or production deployment, pin a specific model revision, checkpoint hash, or release tag rather than relying on the floating latest version.
Details on the training data, training method and configuration, data-processing pipeline, and training infrastructure will be added soon.
OTel-2.0-Decision-26B-A4B-4bit is a 4-bit W4A16 build of Google's Gemma 4 26B-A4B-it for typed decisions: classification, Yes/No questions, and ordered levels. Questions are supplied at request time, and decisions are read from answer-label probabilities rather than generated prose.
Intended Use
Classify messages, call transcripts, tickets, and documents, or ask multiple questions about the same text. Use label probabilities for confidence thresholds and routing. This release is not intended for open-ended chat, retrieval, or reranking.
Evaluation
Benchmarks use the typed-decision readout (level=L0, no prior, shared-prefix execution, adaptive Yes/No ordering with margin 4.0). All runs reported zero errors and answer_mass of 1.00.
| Benchmark | AMD MI350P | AMD Ryzen AI MAX+ PRO 395 (Strix Halo) |
|---|---|---|
| JevBench public (231 tasks) | 86.58% to 87.45% (200 to 202 of 231) | 87.45% (202/231) |
JevBench by tier: 100% on easy, 94.44% to 95.83% on original, and 74.77% to 76.58% on hard tasks. The MoE kernel is not bit-deterministic, so results vary on a few low-margin tasks across runs. The Strix Halo run used vLLM 0.26.0 on ROCm 7.13 (48/48 easy, 71/72 original, 83/111 hard).
Use a system message that asks for a single label (see Prompting and Label Probabilities). Without one, the model sometimes answers hard tasks in prose instead of a label, which lowers hard-tier accuracy by about 6 points.
Inference Recipe
This repository holds the same model in two formats. Pick the recipe that matches your engine:
| Weights | Engine | API | Section |
|---|---|---|---|
| Safetensors, W4A16 (repository root) | vLLM 0.26 or 0.30 on ROCm or CUDA | /v1/chat/completions with logprobs |
Options 1 and 2, NVIDIA H100, Strix Halo |
| Safetensors, W4A16 (repository root) | vLLM nightly | /v1/systemone |
System One API |
GGUF Q8_0 or Q4_K_M (gguf/) |
llama.cpp llama-server (Vulkan, CUDA and other backends) |
/v1/chat/completions with logprobs |
Inference with llama.cpp |
GGUF Q8_0_systemone or Q4_K_M_systemone (gguf/) |
llama.cpp llama-server |
/v1/systemone |
/v1/systemone with llama.cpp |
Serve on one AMD MI350P with vLLM 0.30.0. Both options use the same weights and API. For NVIDIA GPUs, see NVIDIA H100: vLLM on CUDA below.
| Option | Image | Throughput |
|---|---|---|
| 1. Official vLLM | vllm/vllm-openai-rocm:v0.30.0 |
22.7k tok/s |
| 2. Prebuilt image | farbodatdocker/otel-decision-vllm-rocm:v0.30.0 |
32.3k tok/s (42% more) |
Throughput figures are peak aggregate output tokens per second. The highest throughput is reached only when the GPU runs near maximum utilization, with many concurrent requests in flight. A single request runs much slower, so batch or parallelize requests to reach these numbers.
Download
pip install -U huggingface_hub
hf download farbodtavakkoli/OTel-2.0-Decision-26B-A4B-4bit \
--exclude "gguf/*" --local-dir ./OTel-2.0-Decision-26B-A4B-4bit
--exclude "gguf/*" skips the four GGUF files (87.3 GB), which only llama-server needs.
Option 1: Official vLLM
docker run --rm --name otel_decision \
--device /dev/kfd --device /dev/dri --group-add video \
--ipc=host --security-opt seccomp=unconfined -p 8000:8000 \
-v "$PWD/OTel-2.0-Decision-26B-A4B-4bit:/model:ro" \
vllm/vllm-openai-rocm:v0.30.0 \
/model \
--served-model-name OTel-2.0-Decision-26B-A4B-4bit \
--max-model-len 16384 \
--enable-prefix-caching \
--max-logprobs 64 \
--port 8000
With vLLM 0.30.0 already installed, the equivalent is:
vllm serve ./OTel-2.0-Decision-26B-A4B-4bit \
--served-model-name OTel-2.0-Decision-26B-A4B-4bit \
--max-model-len 16384 --enable-prefix-caching --max-logprobs 64 --port 8000
Option 2: Prebuilt Image
The official image with a corrected Gemma 4 MoE kernel and MI350P tile configuration.
docker run --rm --name otel_decision \
--device /dev/kfd --device /dev/dri --group-add video \
--ipc=host --security-opt seccomp=unconfined -p 8000:8000 \
-v "$PWD/OTel-2.0-Decision-26B-A4B-4bit:/model:ro" \
farbodatdocker/otel-decision-vllm-rocm:v0.30.0 \
/model \
--served-model-name OTel-2.0-Decision-26B-A4B-4bit \
--max-model-len 16384 \
--enable-prefix-caching \
--max-logprobs 64 \
--moe-backend flydsl \
--port 8000
Use --moe-backend flydsl only with Option 2. The first start takes 4 to 10 minutes.
System One API (/v1/systemone)
vLLM serves this model on its System One endpoint with no conversion. One request carries a state and any number of named questions (choice, Yes/No, score), and the reply gives a probability for every option. The endpoint is on vLLM main and in nightly builds, not yet in a release. Tested on one NVIDIA H100 80GB with 0.33.0rc1.dev613+g44965a109, the vLLM main build of commit 44965a1; the command below installs exactly that build. A newer nightly or release that includes /v1/systemone should also work: replace the pinned install line with uv pip install -U vllm --torch-backend=auto --extra-index-url https://wheels.vllm.ai/nightly.
uv venv --python 3.12 --seed && source .venv/bin/activate
uv pip install "vllm==0.33.0rc1.dev613+g44965a109" --torch-backend=auto \
--extra-index-url https://wheels.vllm.ai/44965a10974ee417f6f937eee207409462b2a0ab
vllm serve farbodtavakkoli/OTel-2.0-Decision-26B-A4B-4bit \
--served-model-name OTel-2.0-Decision-26B-A4B-4bit \
--max-model-len 16384 --enable-prefix-caching \
--logprobs-mode raw_logprobs --port 8000
--logprobs-mode raw_logprobs is required. With prefix caching, the state is prefilled once and each question only adds its own tokens.
curl -s localhost:8000/v1/systemone -H 'content-type: application/json' -d '{
"model": "OTel-2.0-Decision-26B-A4B-4bit",
"instructions": "You are a decision function. You will be given a state and one question. Reply with the answer label only: no words, no punctuation, no explanation.",
"state": "Caller: There is a 14 dollar charge on my bill I do not recognise. I want it refunded. I already rebooted the router twice.",
"questions": {
"refund_requested": {"type": "noul", "instructions": "Did the caller explicitly ask for a refund or credit?"},
"route": {"type": "choice", "instructions": "Which team should handle this call?",
"criteria": {"billing": "Charges, refunds and payments", "technical": "Internet, TV and equipment faults", "sales": "New plans and upgrades"}},
"urgency": {"type": "score", "instructions": "How urgent is this call?", "criteria": ["Routine", "Soon", "Today"]}
}
}'
Each answer has probabilities for every option and confidence; a Yes/No answer's noul is the probability of yes. diagnostics.<id>.label_mass is the probability the model put on the answer labels and should be close to 1.0. Pass the system prompt in instructions as above: without it, label_mass falls close to zero on some hard multiple choice questions.
Measured on /v1/systemone (one request at a time) |
Accuracy | Latency mean / median / p95 |
|---|---|---|
| JevBench public, 231 tasks, one question per request | 85.28% (197/231) | 0.030 / 0.019 / 0.064 s |
| Synthetic call transcripts, 150 transcripts with 27 Yes/No questions each, one request per transcript | 99.80% | 0.114 / 0.112 / 0.148 s per transcript |
JevBench is lower than in Evaluation above because the endpoint writes its own prompt and reads each question once, without the adaptive Yes/No ordering.
Inference with llama.cpp (GGUF)
The gguf/ folder holds two GGUF files for llama-server chat completions, made from the BF16 weights with llama.cpp's own converter. Use llama.cpp release v0.9.0 or newer.
| File | Size | When to use |
|---|---|---|
OTel-2.0-Decision-26B-A4B-Q8_0.gguf |
26.9 GB | Recommended: about the same accuracy as the safetensors |
OTel-2.0-Decision-26B-A4B-Q4_K_M.gguf |
16.8 GB | Less memory, about 2.6 points lower on JevBench |
llama-server -hf farbodtavakkoli/OTel-2.0-Decision-26B-A4B-4bit:Q8_0 \
-ngl 999 -c 16384 --jinja --host 0.0.0.0 --port 8080
-hf downloads only the chosen GGUF, not the safetensors; use :Q4_K_M for the smaller file. Get llama-server for Vulkan from the llama.cpp releases page, or build it with cmake -B build -DGGML_VULKAN=ON && cmake --build build -j (needs the Vulkan SDK, or libvulkan-dev and glslc); the binary is build/bin/llama-server. To use a downloaded file, replace -hf ... with -m OTel-2.0-Decision-26B-A4B-Q8_0.gguf.
Read the label probabilities from /v1/chat/completions, as in Prompting and Label Probabilities below, with max_tokens 1, temperature 0, top_logprobs 64 and "chat_template_kwargs": {"enable_thinking": false}. For /v1/systemone, use the _systemone files below.
curl -s localhost:8080/v1/chat/completions -H 'content-type: application/json' -d '{
"messages": [
{"role": "system", "content": "You are a decision function. You will be given a state and one question. Reply with the answer label only: no words, no punctuation, no explanation."},
{"role": "user", "content": "State:\nCaller: There is a 14 dollar charge on my bill I do not recognise. I want it refunded.\n\nQuestion: Did the caller explicitly ask for a refund or credit?\nAnswer Yes or No."}
],
"max_tokens": 1, "temperature": 0, "logprobs": true, "top_logprobs": 64,
"chat_template_kwargs": {"enable_thinking": false}
}'
The reply's top_logprobs should put almost all probability on Yes.
llama-server on 1x H100 (CUDA), one request at a time |
JevBench, 231 tasks (median per task) | Synthetic transcripts, 27 Yes/No questions each (mean per transcript) |
|---|---|---|
| Q8_0 | 87.01% (201/231), 0.072 s | 99.85%, 2.753 s |
| Q4_K_M | 84.42% (195/231), 0.072 s | 99.75%, 2.195 s |
On 70 JevBench Yes/No tasks whose criteria are labelled true/false, the model answers true or false, so Yes/No carry little probability and the decision is read from those small values, as in the published evaluation. Each transcript is 27 separate requests here, which is why it is slower than on vLLM.
On AMD Ryzen AI MAX+ PRO 395 (Radeon 8060S, Vulkan, llama.cpp v0.9.0), both files gave the expected answer to the request above on 20 of 20 runs, with a median of 0.077 s (Q8_0, 27.1 GiB of GPU memory) and 0.058 s (Q4_K_M, 17.6 GiB) per request after the first. Validate on your own labelled data.
/v1/systemone with llama.cpp
The gguf/ folder also holds OTel-2.0-Decision-26B-A4B-Q8_0_systemone.gguf and OTel-2.0-Decision-26B-A4B-Q4_K_M_systemone.gguf: the same weights with metadata that turns on llama-server's own /v1/systemone. They take the same request as the vLLM System One API above and answer every question of a request in one call. These files answer decisions only (chat returns 501), and their system prompt is built in, so a request's instructions field has no effect.
llama-server -hf farbodtavakkoli/OTel-2.0-Decision-26B-A4B-4bit:Q8_0_systemone \
-ngl 999 -c 65536 -np 8 -fa on --swa-full --host 0.0.0.0 --port 8081
Use :Q4_K_M_systemone for the smaller file. --swa-full is required: without it the questions of one request do not share the state, and answers can change between identical requests.
curl -s localhost:8081/v1/systemone -H 'content-type: application/json' -d '{
"state": "Caller: There is a 14 dollar charge on my bill I do not recognise. I want it refunded. I already rebooted the router twice.",
"questions": {
"refund_requested": {"type": "noul", "instructions": "Did the caller explicitly ask for a refund or credit?"},
"route": {"type": "choice", "instructions": "Which team should handle this call?",
"criteria": {"billing": "Charges, refunds and payments", "technical": "Internet, TV and equipment faults", "sales": "New plans and upgrades"}},
"urgency": {"type": "score", "instructions": "How urgent is this call?", "criteria": ["Routine", "Soon", "Today"]}
}
}'
A Yes/No answer is {"type": "noul", "noul": <probability of yes>}; choice and score answers have probabilities for every option, as on vLLM. The reply has no diagnostics field.
/v1/systemone with llama-server, one request at a time |
JevBench, 231 tasks (median per task) | Synthetic transcripts, 27 Yes/No questions each (mean per transcript) |
|---|---|---|
| Q8_0_systemone, 1x H100 (CUDA) | 87.88% (203/231), 0.031 s | 99.75%, 0.588 s |
| Q4_K_M_systemone, 1x H100 (CUDA) | 86.58% (200/231), 0.032 s | 99.78%, 0.589 s |
On AMD Ryzen AI MAX+ PRO 395 (Radeon 8060S, Vulkan, llama.cpp v0.9.0), both files gave the expected answers to a five question request on 20 of 20 runs, with a median of 0.515 s (Q8_0, 39.7 GiB of GPU memory) and 0.472 s (Q4_K_M, 30.1 GiB) per request.
NVIDIA H100: vLLM on CUDA
Serve on one NVIDIA H100 80GB (SXM, HBM3) with the official CUDA image of vLLM 0.26.0. vLLM selects the Marlin W4A16 kernels for both the dense and MoE layers automatically, and no config shim or extra flags are needed.
docker run --rm --name otel_decision \
--gpus '"device=0"' \
--ipc=host -p 8000:8000 \
-v "$PWD/OTel-2.0-Decision-26B-A4B-4bit:/model:ro" \
vllm/vllm-openai:v0.26.0 \
/model \
--served-model-name OTel-2.0-Decision-26B-A4B-4bit \
--max-model-len 16384 \
--enable-prefix-caching \
--max-logprobs 64 \
--port 8000
With vLLM 0.26.0 already installed (pip install vllm==0.26.0), the equivalent is:
CUDA_VISIBLE_DEVICES=0 vllm serve ./OTel-2.0-Decision-26B-A4B-4bit \
--served-model-name OTel-2.0-Decision-26B-A4B-4bit \
--max-model-len 16384 --enable-prefix-caching --max-logprobs 64 --port 8000
| Measured (1x H100 80GB, vLLM 0.26.0, 512 output tokens per request) | Value |
|---|---|
| Peak aggregate output, 256 concurrent requests | 10.6k tok/s |
| 1,024 concurrent requests | 10.1k tok/s |
| Single request | 223 tok/s |
The weights use about 17 GB, which leaves about 51 GB for KV cache (about 242k tokens) at the default --gpu-memory-utilization 0.92. The first start takes about 4 minutes because of torch.compile and CUDA graph capture. Later starts reuse the compile cache. To serve several replicas on one multi-GPU node, add --data-parallel-size N.
AMD Ryzen AI MAX+ PRO 395 (Strix Halo): vLLM on ROCm
Serve on one AMD Ryzen AI MAX+ PRO 395 (Strix Halo: Radeon 8060S GPU, 128 GB unified memory, configured as 96 GB GPU and 32 GB NPU) with vLLM 0.26.0 and ROCm 7.13 for gfx1151.
# 1. ROCm 7.13 + vLLM for gfx1151 (managed by the ROCm CLI)
rocm engines install vllm
PY=~/.rocm/runtimes/wheel/release-wheel-gfx1151-7-13-0/bin/python
# 2. Config shim: transformers 5.19 drops Gemma 4's global head sizes that vLLM 0.26 reads
mkdir -p shim && cat > shim/sitecustomize.py <<'EOF'
from transformers import PretrainedConfig
from transformers.models.gemma4 import configuration_gemma4 as g4
PretrainedConfig.allow_global_per_layer_attribute_access = True
_init = g4.Gemma4TextConfig.__init__
def init(self, *a, **k):
_init(self, *a, **k)
lt, pl = list(getattr(self, "layer_types", []) or []), getattr(self, "per_layer_config", None)
if pl is not None and "full_attention" in lt:
full = pl[lt.index("full_attention")]
for dst, src in (("global_head_dim", "head_dim"), ("num_global_key_value_heads", "num_key_value_heads")):
if dst not in self.__dict__ and getattr(full, src, None) is not None:
object.__setattr__(self, dst, getattr(full, src))
g4.Gemma4TextConfig.__init__ = init
EOF
# 3. Serve (the torch wheel links OpenMPI 4's libmpi_cxx.so.40; point at its lib dir)
LD_LIBRARY_PATH=/path/to/openmpi4/lib PYTHONPATH=$PWD/shim \
$PY -m vllm.entrypoints.cli.main serve ./OTel-2.0-Decision-26B-A4B-4bit \
--served-model-name OTel-2.0-Decision-26B-A4B-4bit \
--max-model-len 16384 --enable-prefix-caching --max-logprobs 64 \
--gpu-memory-utilization 0.55 --max-num-seqs 256 --port 8000
| Measured (AC power, performance profile) | Value |
|---|---|
| Peak aggregate output, 256 concurrent requests | 377 tok/s |
| 64 concurrent requests | 257 tok/s |
| Single request | 30 tok/s |
| JevBench public, 8 concurrent | 1.27 tasks/s |
Peak throughput again needs near-maximum utilization: aggregate output keeps rising up to 256 concurrent requests. Keep HIP graphs on (the default). --enforce-eager cuts single-request speed to about 19 tok/s. The shim is not needed with vLLM releases that read Gemma 4's per-layer config directly. torch fails to import without an OpenMPI 4.x runtime. Distributions that ship only OpenMPI 5 (for example Ubuntu 26.04) lack libmpi_cxx.so.40, so extract an OpenMPI 4 package into a local directory and use that directory; no sudo is needed.
Prompting and Label Probabilities
Put the text first, then the question and labels. Use letters (A, B, …) for options or Yes/No, and numbers for ordered levels. Add a system message that asks for exactly one label, and request only the label through the chat endpoint, which applies the repository's template with thinking disabled.
import math
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")
question = """State:
Order #1182. Status: shipped on 3 September. Carrier: DHL.
Question: Has the order been shipped? Answer strictly from the facts stated in the text.
Options:
A. Yes
B. No
Answer with the letter only."""
labels = ["A", "B"]
r = client.chat.completions.create(
model="OTel-2.0-Decision-26B-A4B-4bit",
temperature=0,
max_tokens=1,
logprobs=True,
top_logprobs=64,
messages=[
{"role": "system", "content": "You are a typed decision engine. Reply with exactly one answer label "
"(a single letter or number) and nothing else. Do not explain."},
{"role": "user", "content": question},
],
)
top = {t.token: t.logprob for t in r.choices[0].logprobs.content[0].top_logprobs}
probs = {label: math.exp(top[label]) if label in top else 0.0 for label in labels}
print("probabilities:", probs)
print("decision:", max(probs, key=probs.get))
print("answer_mass:", sum(probs.values()))
# Expected: decision A, answer_mass approximately 1.0
Choose the label with the highest probability. answer_mass is the summed probability of the answer labels and should be close to 1.0; a low value indicates an invalid decision readout. For multiple questions, keep the shared text byte-identical so prefix caching can reuse it.
Verify the Deployment
Check that http://localhost:8000/v1/models lists the branded model name, then run the example above and confirm decision A with answer_mass near 1.0. For llama-server, use port 8080 and the request in the llama.cpp section instead; /v1/models lists the GGUF path given to -m (or a name set with --alias), and the answer should be Yes. Validate representative prompts against a reference run before deployment.
Limitations and Responsible Use
Evaluation covers English text only. JevBench is small and does not guarantee production accuracy. Validate on your own human-labeled data, and do not assume label probabilities are calibrated for your workload. Keep human review for consequential decisions.
The recipe starts an unauthenticated server. Add authentication before exposing it beyond your machine.
Resources
- Project Page
- OTel code
- OTel Selected Coverage
- OTel Paper
- OTel 1.0 Coverage
- OTel 2.0 Coverage
- OTel Docker
Citation
@inproceedings{tavakkoli2026otel2,
title = {OTel 2.0: Open Telco AI Datasets, Benchmarks, and Models},
author = {Tavakkoli, Farbod and others},
booktitle = {TBD},
year = {2026},
url = {https://huggingface.co/farbodtavakkoli}
}
Contact
If you have any technical questions, please feel free to reach out to mark.austin@att.com or farbod.tavakkoli@att.com or farbodtavakoli@gmail.com
- Downloads last month
- 4,014,149