Zenyx V3 Base (1.5B Mixture-of-Experts)
Zenyx V3 is an efficient 1.5B-parameter Mixture-of-Experts (MoE) foundation model built for low-latency inference and high throughput. It is written from scratch in JAX/Flax and trained on TPU v5e-8.
This is a BASE model — it is not instruction-tuned. It completes text; it does not follow instructions or hold a conversation. Prompt it with a prefix to continue (
"The capital of France is"), not with a request ("Explain gravity"). Pretraining is still in progress; SFT/chat variants will follow.
Current checkpoint: step 70,400 · 42.4B tokens seen
Model Architecture
- Sparse Mixture-of-Experts: 12 routed experts + 1 shared expert, exactly 2 active per token, with a Sinkhorn transport-based gate.
- Multi-head Latent Attention (MLA): compresses the KV cache into a low-rank latent subspace, cutting HBM bandwidth and memory footprint.
- Hyper-Connections: Sinkhorn-normalised residual routing for gradient stability at scale.
- Multi-Token Prediction (MTP): one auxiliary prediction head during training.
- Context: pretrained at up to 4,096 tokens (progressive 2,048 → 4,096). YaRN and RoPE scaling factors are precomputed so context can be extended at inference time beyond the trained length.
| Total parameters | ~1.5B |
| Active parameters / token | ~0.4B |
| Layers | 16 (2 dense + 14 MoE) |
| Hidden size | 1,536 |
| Attention heads | 12 (head dim 128) |
| Vocabulary | 129,280 |
| Precision | bfloat16 |
Benchmarks — checkpoint step 70,400 (42.4B tokens)
All tasks are evaluated with the standard base-model protocol: the model scores the log-likelihood of every candidate continuation and the highest-scoring one is taken as the answer. Nothing is generated and no output parsing is involved, so the numbers do not depend on instruction-following ability. 0-shot, full evaluation sets, no subsampling.
acc_norm normalises each continuation's log-likelihood by its length in characters,
which removes the bias toward short answers; it is the headline metric wherever the
task has candidates of differing lengths.
| Benchmark | acc |
acc_norm |
Random | Δ | n | Description |
|---|---|---|---|---|---|---|
| HellaSwag | 29.67% ± 0.46 | 32.51% ± 0.47 | 25.0% | +7.5 | 10,042 | Commonsense sentence completion |
| ARC-Easy | 49.58% ± 1.03 | 45.58% ± 1.02 | 25.0% | +20.6 | 2,376 | Grade-school science questions |
| ARC-Challenge | 19.28% ± 1.15 | 25.00% ± 1.26 | 25.0% | +0.0 | 1,172 | Hard grade-school science questions |
| PIQA | 60.61% ± 1.14 | 59.96% ± 1.14 | 50.0% | +10.0 | 1,838 | Physical commonsense reasoning |
| WinoGrande | 49.57% ± 1.40 | — | 50.0% | -0.4 | 1,267 | Pronoun resolution / coreference |
| OpenBookQA | 16.80% ± 1.67 | 29.00% ± 2.03 | 25.0% | +4.0 | 500 | Elementary science with open book |
| BoolQ | 62.14% ± 0.85 | 62.17% ± 0.85 | 62.2% | -0.1 | 3,270 | Yes/no reading comprehension |
| SciQ | 76.30% ± 1.34 | 68.20% ± 1.47 | 25.0% | +51.3 | 1,000 | Science exam questions with support |
| LAMBADA (OpenAI) | 26.96% ± 0.62 | — | 0.0% | +27.0 | 5,153 | Long-range last-word prediction |
| MMLU (5-shot) | 26.01% ± 0.37 | — | 25.0% | +1.0 | 14,042 | 57 subjects of academic knowledge |
| RACE | 29.94% ± 0.65 | 32.85% ± 0.67 | 25.0% | +7.9 | 4,934 | Exam reading comprehension |
| CommonsenseQA | 28.26% ± 1.29 | 30.55% ± 1.32 | 20.0% | +10.5 | 1,221 | 5-choice commonsense (random = 20%) |
| COPA | 59.00% ± 4.92 | 59.00% ± 4.92 | 50.0% | +9.0 | 100 | Causal reasoning |
| LogiQA | 19.51% ± 1.55 | 25.81% ± 1.71 | 25.0% | +0.8 | 651 | Logical deduction |
| WSC273 | 51.28% ± 3.03 | — | 50.0% | +1.3 | 273 | Winograd coreference |
| TruthfulQA MC1 | 19.83% ± 1.39 | — | 22.8% | -3.0 | 817 | Resistance to common misconceptions |
| Arithmetic | 1.04% ± 0.09 | — | 0.0% | +1.0 | 14,000 | 2-5 digit add/sub/mul, generated in-harness |
Bold marks the metric that is conventional for that task — acc_norm for
HellaSwag, ARC, PIQA and OpenBookQA; acc for WinoGrande, BoolQ, SciQ and
LAMBADA. The convention is applied per task, not chosen per result: it lowers
the reported figure for ARC-Easy (44.53 rather than 49.54) and PIQA (59.79
rather than 60.83). Δ compares the bolded metric to the random baseline.
Both metrics
| Benchmark | acc |
acc_norm |
n |
|---|---|---|---|
| HellaSwag | 29.67% ± 0.46 | 32.51% ± 0.47 | 10,042 |
| ARC-Easy | 49.58% ± 1.03 | 45.58% ± 1.02 | 2,376 |
| ARC-Challenge | 19.28% ± 1.15 | 25.00% ± 1.26 | 1,172 |
| PIQA | 60.61% ± 1.14 | 59.96% ± 1.14 | 1,838 |
| WinoGrande | 49.57% ± 1.40 | — | 1,267 |
| OpenBookQA | 16.80% ± 1.67 | 29.00% ± 2.03 | 500 |
| BoolQ | 62.14% ± 0.85 | 62.17% ± 0.85 | 3,270 |
| SciQ | 76.30% ± 1.34 | 68.20% ± 1.47 | 1,000 |
| LAMBADA (OpenAI) | 26.96% ± 0.62 | — | 5,153 |
| MMLU (5-shot) | 26.01% ± 0.37 | — | 14,042 |
| RACE | 29.94% ± 0.65 | 32.85% ± 0.67 | 4,934 |
| CommonsenseQA | 28.26% ± 1.29 | 30.55% ± 1.32 | 1,221 |
| COPA | 59.00% ± 4.92 | 59.00% ± 4.92 | 100 |
| LogiQA | 19.51% ± 1.55 | 25.81% ± 1.71 | 651 |
| WSC273 | 51.28% ± 3.03 | — | 273 |
| TruthfulQA MC1 | 19.83% ± 1.39 | — | 817 |
| Arithmetic | 1.04% ± 0.09 | — | 14,000 |
Language modelling
| Corpus | Value | Metric |
|---|---|---|
| WikiText-2 (raw) | 25.79 | token-level perplexity |
| WikiText-2 (raw) | 48.18 | word-level perplexity |
| WikiText-2 (raw) | 1.0425 | bits per byte |
| LAMBADA | 34.15 | perplexity of the target word |
WikiText-2 is scored with a rolling 1024-token window at stride 512, so every counted token is predicted with at least 512 tokens of left context and each token is counted exactly once. (Scoring disjoint windows instead inflates these figures by ~15% because the leading tokens of each window are predicted from nothing.)
Trajectory across all benchmarked checkpoints
Tokens seen: 34.8B | 36.5B | 39.4B | 42.4B. The pretraining data mixture was changed partway through this sequence (code weight raised, several synthetic sources cut), so these columns do not represent a tokens-only progression.
Accuracy benchmarks (higher is better)
| Benchmark | 63,200 | 64,800 | 67,600 | 70,400 | net | |
|---|---|---|---|---|---|---|
| HellaSwag | 32.22% | 32.66% | 32.84% | 32.51% | +0.29 | up |
| ARC-Easy | 44.53% | 45.08% | 44.91% | 45.58% | +1.05 | up |
| ARC-Challenge | 25.77% | 25.51% | 26.02% | 25.00% | -0.77 | down |
| PIQA | 59.79% | 59.85% | 61.53% | 59.96% | +0.16 | up |
| WinoGrande | 49.49% | 50.51% | 51.07% | 49.57% | +0.08 | up |
| OpenBookQA | 30.00% | 28.00% | 29.80% | 29.00% | -1.00 | down |
| BoolQ | 60.55% | 60.83% | 61.80% | 62.14% | +1.59 | up |
| SciQ | 75.10% | 76.10% | 75.70% | 76.30% | +1.20 | up |
| LAMBADA (OpenAI) | 25.50% | 25.42% | 24.94% | 26.96% | +1.46 | up |
| MMLU (5-shot) | 26.07% | 26.71% | 25.77% | 26.01% | -0.06 | down |
| RACE | 32.79% | 32.77% | 32.96% | 32.85% | +0.06 | up |
| CommonsenseQA | 29.57% | 29.57% | 29.40% | 30.55% | +0.98 | up |
| COPA | 58.00% | 58.00% | 61.00% | 59.00% | +1.00 | up |
| LogiQA | 25.81% | 25.19% | 25.65% | 25.81% | +0.00 | flat |
| WSC273 | 54.95% | 52.38% | 52.75% | 51.28% | -3.66 | down |
| TruthfulQA MC1 | 19.22% | 20.20% | 19.58% | 19.83% | +0.61 | up |
| Arithmetic | 0.14% | 0.55% | 0.58% | 1.04% | +0.91 | up |
Language modelling (LOWER is better)
| Metric | 63,200 | 64,800 | 67,600 | 70,400 | net | |
|---|---|---|---|---|---|---|
| WikiText-2 perplexity | 26.47 | 25.79 | 25.98 | 25.79 | -0.6806 | BETTER |
| WikiText-2 bits/byte | 1.051 | 1.043 | 1.045 | 1.043 | -0.008356 | BETTER |
| LAMBADA perplexity | 35.79 | 36.19 | 36.84 | 34.15 | -1.647 | BETTER |
The Pile, by content type (bits/byte, LOWER is better)
| Category | 63,200 | 64,800 | 67,600 | 70,400 | net | |
|---|---|---|---|---|---|---|
| Code / technical | 0.9558 | 0.9518 | 0.9438 | 0.9341 | -0.0217 | BETTER |
| Science / legal | 0.8987 | 0.8950 | 0.8930 | 0.8892 | -0.0095 | BETTER |
| Web / reference | 1.1586 | 1.1561 | 1.1557 | 1.1543 | -0.0043 | BETTER |
| Prose / spoken | 1.5606 | 1.5466 | 1.5650 | 1.5383 | -0.0223 | BETTER |
| Every technical category improved monotonically at every checkpoint. Prose/spoken dipped for exactly one interval after the mixture changed, then recovered and overshot to new bests -- a one-off transition cost, not a permanent trade. |
Does few-shot prompting help? (MMLU by shot count)
| Shots | step 67600 | step 70,400 | Shot source |
|---|---|---|---|
| 5 | 25.77% | 26.01% ± 0.37 | dev split, the published convention |
No. More demonstrations do not help and the 5-shot result is the best of the three at both checkpoints, with 10-shot dropping to the 25% chance line (-1.47 points vs 5-shot at step 70,400, ~2.8 sigma). The same ordering appears independently at both checkpoints, so it is not a fluke of one run.
This is what a model without in-context learning looks like: using examples to infer a task is an ability that emerges later in training, and before it does, extra shots are just tokens competing for attention with the actual question. Practical consequence: prompt this model with a short direct prefix, not a long few-shot preamble.
Arithmetic
Exact-match on the answer, greedy decoding, GPT-3 prompt format
(Question: What is 47 plus 21? / Answer: 68).
| Operation | step 67600 | step 70,400 | n |
|---|---|---|---|
| 2-digit addition | 0.30% | 4.25% | 2,000 |
| 2-digit subtraction | 3.35% | 2.50% | 2,000 |
| 3-digit addition | 0.00% | 0.00% | 2,000 |
| 3-digit subtraction | 0.40% | 0.40% | 2,000 |
| 4-digit addition | 0.00% | 0.00% | 2,000 |
| 5-digit addition | 0.00% | 0.00% | 2,000 |
| 2-digit multiplication | 0.00% | 0.15% | 2,000 |
| overall | 0.579% | 1.043% | 14,000 |
The model essentially cannot do arithmetic — but two-digit subtraction moved from 0.65% to 3.30% between these two checkpoints (5.1x, ~6 sigma on identical problems), which is the signature of a capability just beginning to emerge. Note that 15.5% of the pretraining mix is mathematics, yet that has bought fluency in mathematical language rather than the ability to compute.
Items are generated in-harness from a fixed seed using this prompt format, because
EleutherAI/arithmeticis a loading script with no parquet branch and cannot be fetched underdatasets>=3. Both checkpoints see byte-identical problems, so the comparison is exact — but these numbers are not interchangeable with published EleutherAI/arithmetic results.
Language modelling by genre (The Pile)
Bits-per-byte on each Pile domain, lower is better, scored with the same rolling 1024-token window as WikiText-2 so the numbers are directly comparable to it. This is the clearest picture of what the model is actually good at, because it measures raw prediction rather than multiple-choice ability.
| Domain | bits/byte | perplexity | tokens | Δ vs prev |
|---|---|---|---|---|
| Github | 0.617 | 3.91 | 479,656 | -0.0083 |
| PubMed Central | 0.773 | 15.81 | 292,334 | -0.0037 |
| USPTO Backgrounds | 0.805 | 16.94 | 296,162 | -0.0027 |
| NIH ExPorter | 0.883 | 25.86 | 41,533 | -0.0021 |
| ArXiv | 0.900 | 8.14 | 447,478 | -0.0138 |
| PubMed Abstracts | 0.906 | 21.57 | 306,636 | -0.0034 |
| StackExchange | 0.968 | 13.57 | 387,038 | -0.0064 |
| FreeLaw | 0.995 | 21.33 | 339,260 | -0.0063 |
| Wikipedia (en) | 1.016 | 22.21 | 341,511 | +0.0015 |
| Pile-CC | 1.123 | 36.59 | 326,706 | -0.0003 |
| OpenWebText2 | 1.160 | 34.10 | 348,909 | -0.0057 |
| BookCorpus2 | 1.162 | 34.27 | 139,043 | -0.0003 |
| Enron Emails | 1.265 | 22.27 | 16,119 | -0.0046 |
| Gutenberg (PG-19) | 1.294 | 35.52 | 133,371 | +0.0006 |
| HackerNews | 1.319 | 43.87 | 57,843 | -0.0012 |
| Books3 | 1.360 | 35.74 | 396,623 | -0.0002 |
| OpenSubtitles | 1.366 | 32.12 | 234,514 | -0.0012 |
| DM Mathematics | 1.380 | 8.10 | 370,912 | -0.0176 |
| PhilPapers | 1.384 | 53.75 | 38,263 | -0.0856 |
| Ubuntu IRC | 1.791 | 44.25 | 14,407 | -0.0364 |
| YoutubeSubtitles | 1.908 | 138.15 | 51,428 | -0.0500 |
| EuroParl | 2.042 | 139.50 | 19,523 | -0.0401 |
The ordering here is a direct readout of the pretraining mix: code, papers and mathematics sit at the top because they are what the model has been fed most of.
Progress since the previous checkpoint
Same suite, same code, same full evaluation sets — only the checkpoint differs. Step 67,600 → 70,400 is +2.94B tokens.
| Benchmark | step 67,600 | step 70,400 | Δ | ±2σ needs |
|---|---|---|---|---|
HellaSwag (acc_norm) |
32.84% | 32.51% | -0.33 | ±0.66 |
ARC-Easy (acc_norm) |
44.91% | 45.58% | +0.67 | ±1.44 |
ARC-Challenge (acc_norm) |
26.02% | 25.00% | -1.02 | ±1.80 |
PIQA (acc_norm) |
61.53% | 59.96% | -1.58 | ±1.61 |
WinoGrande (acc) |
51.07% | 49.57% | -1.50 | ±1.99 |
OpenBookQA (acc_norm) |
29.80% | 29.00% | -0.80 | ±2.88 |
BoolQ (acc) |
61.80% | 62.14% | +0.34 | ±1.20 |
SciQ (acc) |
75.70% | 76.30% | +0.60 | ±1.91 |
LAMBADA (OpenAI) (acc) |
24.94% | 26.96% | +2.02 | ±0.86 |
MMLU (5-shot) (acc) |
25.77% | 26.01% | +0.25 | ±0.52 |
RACE (acc_norm) |
32.96% | 32.85% | -0.10 | ±0.95 |
CommonsenseQA (acc_norm) |
29.40% | 30.55% | +1.15 | ±1.85 |
COPA (acc) |
61.00% | 59.00% | -2.00 | ±6.93 |
LogiQA (acc_norm) |
25.65% | 25.81% | +0.15 | ±2.42 |
WSC273 (acc) |
52.75% | 51.28% | -1.47 | ±4.28 |
TruthfulQA MC1 (acc) |
19.58% | 19.83% | +0.24 | ±1.97 |
Arithmetic (acc) |
0.58% | 1.04% | +0.46 | ±0.11 |
| WikiText-2 perplexity | 25.98 | 25.79 | -0.1949 | — |
| WikiText-2 bits/byte | 1.045 | 1.043 | -0.002416 | — |
| LAMBADA perplexity | 36.84 | 34.15 | -2.688 | — |
Δ is on the conventional metric for each task. Bold marks a change larger than two standard errors of the difference; anything unbolded is inside the noise floor and should not be read as movement. The quoted error treats the two runs as independent, which is conservative here — they score identical items, so the true paired error is smaller.
What actually changed. The multiple-choice suite is flat as usual (9/17 improved, sign test p = 0.50) -- a 7.5% token increase cannot move benchmarks whose error bars are +/-0.5 to 5 points. Two metrics cleared 2 sigma, and the language-modelling picture improved on every axis:
- Arithmetic nearly doubled, 81 -> 146 / 14,000 (4.3 sigma), driven by 2-digit addition emerging: 6 -> 85 / 2000, 8.5 sigma. Addition had been stuck at zero while subtraction emerged in an earlier interval; it has now overtaken it. 2-digit multiplication produced its first non-zero result (3/2000). Identical fixed-seed problems at every checkpoint.
- LAMBADA +2.02 points (2.34 sigma) and its perplexity -7.3% -- the first accuracy benchmark in this programme to clear 2 sigma, on narrative fiction.
- The Pile: 20 of 22 domains improved (sign test p = 0.0001), token-weighted mean 1.0554 -> 1.0488, the largest single-interval gain recorded here.
- WikiText-2 perplexity recovered to a new best, 25.983 -> 25.788.
On the earlier data-mixture change. The previous interval showed every technical domain improving and every prose domain regressing, which read as a permanent trade. It was not: prose was paying a one-interval transition cost. This interval it recovered and overshot, with the largest gains in exactly the domains that had suffered most (PhilPapers -0.086, YoutubeSubtitles -0.050, EuroParl -0.040). Technical improved monotonically throughout and never paid anything.
Reading these numbers. This is a partially-trained 1.5B base model, so knowledge-heavy multiple-choice tasks sit close to their random baselines — that is expected at this scale and token count. The signal to watch is the language-modelling side: LAMBADA accuracy and WikiText perplexity measure whether the model has actually learned to predict text, and those improve steadily long before multiple-choice benchmarks move. Note also that BoolQ's majority-class baseline is 62.2%, so a score near that is not evidence of comprehension.
Hardware Serving Benchmarks (NVIDIA L4, 24 GB)
Measured with the JAX/Flax serving loop: static shape pre-allocation, bucketed prefill and GPU-native sampling.
| Metric | Value | Notes |
|---|---|---|
| Decode speed | 68.5 tok/s | steady-state autoregressive decode |
| Warm prefill | ~20 ms | short prompt, shape already compiled |
| Checkpoint load | ~26 s | params → GPU, from local cache |
| Active VRAM | ~5.0 GB | of 24 GB |
Cold shapes pay a one-off JIT compile (tens of seconds) the first time a new (prompt length, max tokens) pair is seen; warm requests are the numbers above.
Inference Example
from zenyx_v3_inference import ZenyxGenerator
generator = ZenyxGenerator(step=70400)
# Base model: give it a prefix to CONTINUE, not an instruction to follow.
print(generator.generate(
"The capital of France is",
max_new_tokens=80,
temperature=0.7,
repetition_penalty=1.15,
))
Evaluation Reproducibility
Benchmarks were produced by modal_base_evals.py on a single NVIDIA L4, scoring
continuations in batches with length-bucketed padding. Task formats follow the
lm-evaluation-harness conventions (prompt templates, acc / acc_norm definitions
and answer-key handling), so the numbers are broadly comparable to published
base-model results, though this is an independent implementation rather than a
harness run.
Limitations
- Pretraining is incomplete — the model will change substantially with more tokens.
- Not instruction-tuned, not RLHF'd, and not safety-filtered. Outputs may be factually wrong, biased, or nonsensical.
- Trained predominantly on English text, code, mathematics and synthetic reasoning data; other languages are not supported.