DJLougen/LFM2.5-VL-3B-DFlash-drafter

LFM2.5-VL-3B DFlash Drafter

A DFlash block-diffusion drafter for LiquidAI/LFM2.5-VL-3B, intended for lossless speculative decoding with llama.cpp.

The drafter is a 4-layer Qwen3-style backbone that drafts a block of tokens in one forward pass. llama.cpp loads it with --model-draft (arch dflash) and verifies every drafted token against the target model, so output is token-for-token identical to the target model while (in principle) reducing per-token latency.

Status: this is a research artifact trained under a hard budget. The drafter loads, is recognized as draft-dflash, and generates through the llama.cpp runtime, but its measured draft acceptance is low (~0.11 greedy top-1 prefix acceptance at n_max=3) on the held-out general-text split. It is published for reproducibility and as a starting point, not as a production speedup.

Architecture

Field	Value
Arch	dflash (Qwen3-style 4-layer decoder)
Hidden size	2048
FFN intermediate	6144
Layers	4
Attention heads	16 Q / 8 KV, head-dim 128
Vocab	128000 (mask id 125017 = [PAD125017])
Block size	7
Anchor-first	sample_from_anchor = true
Target layers (trained id space)	[1, 10, 18, 27]
Fused feature dim	4 x 2048 = 8192 (fc fusion + RMS norm)

At each draft step the runtime feeds [anchor, <mask> x (block_size-1)]; the drafter denoises the masked positions into block_size-1 draft tokens. A single linear fc layer fuses the target model's hidden states from the configured target_layers into a 8192-dim context feature that is injected as the draft-side K/V source; the draft carries its own token embedding and a tied output head initialized from the LFM2.5 embeddings so token ids align 1:1 with the target vocabulary.

Training

• Base model: LiquidAI/LFM2.5-VL-3B (text LM; Lfm2 arch, 2048 hidden, 30 layers).
• Backbone: 4-layer Qwen3-style DFlash drafter, block_size 7, anchor-first.
• Loss: block-diffusion masked cross-entropy over the 6 non-anchor slots of each block.
• Dataset: fineweb-edu (streaming, sample-10BT split) tokenized with the LFM2.5 tokenizer; 1.29 GiB tokens.npy (~322M tokens), held-out = the last 0.5%.
• Precision: bf16 (AMP), H100 NVL, batch 8 x seq 2048, ~16.5k tok/s.
• Optimizer: AdamW (beta 0.9/0.95, wd 0.01), body LR 5e-4, embedding LR 1e-4, cosine decay + 300-step warmup, grad clip 1.0.
• Tokens trained: ~91M (loss ~5.9 at 91M; final log 98.1M tokens, loss 5.76, min 5.68 at step 5600). Loss curve: 8.31 -> 7.0 -> 6.3 -> 5.9 (40-step chunk means).
• Checkpoint: step005600. The run crashed at step 6000 while writing the next checkpoint because the pod container hit its disk quota; step005600 is the last complete checkpoint. No retraining was performed.
• Acceptance on held-out text (greedy top-1, n_max=3, 60 x 511-token sequences): acceptance_rate 0.113, top-1 slot agreement 0.171, mean accepted length 0.34. This is below a 0.4 target and is reported honestly.

Supported engines

• llama.cpp (verified): arch=dflash, commit c7bda030e (and later dflash-capable builds). Use llama-server --model-draft (see below). Verified on an H100 NVL with a pinned CUDA build.
• sglang / vLLM: not included. Those runtimes need their own DFlash adapter; the dflash package in this repo is the transformer-side reference only.
• The GGUF in this repo (LFM2.5-VL-3B-DFlash-drafter.gguf) is F16 with F32 norm/fc weights, vocab 128000, tokenizer.ggml.mask_token_id = 125017.

Usage (llama.cpp)

Build llama.cpp at commit c7bda030e (or later) with CUDA, then:

llama-server \
  --model ~/models/lfm25vl/LFM2.5-VL-3B-Q4_K_M.gguf \
  --model-draft ~/models/lfm25vl/LFM2.5-VL-3B-DFlash-drafter.gguf \
  -c 131072 -ngl 99999 -fa on -ctk f16 -ctv f16 \
  --rope-scaling yarn --yarn-orig-ctx 32768 \
  --override-kv lfm2.context_length=int:131072 \
  --spec-draft-n-max 3 \
  --host 0.0.0.0 --port 8081

Full-vocab draft with its own embeddings/head; no reduced-vocab d2t mapping is used.

Verification

The llama.cpp runtime auto-detects the draft arch and registers the implementation:

spec common_specu: auto-detected speculative type 'draft-dflash' from the draft model metadata
common_speculative_impl_draft_dflash: adding speculative implementation 'draft-dflash'
common_speculative_impl_draft_dflash: - n_max=3, n_min=1, p_min=0.00
common_speculative_impl_draft_dflash: - block_size=7, mask_token_id=125017, n_extract=4, sample_from_anchor=true
llama_server: listening on http://127.0.0.1:18090

Generation succeeds end-to-end (speculative round ran with 3 draft tokens; 0 accepted on this short prompt, consistent with the measured low acceptance):

prompt: "What is 2+2? Answer in one word."
response: "Four"
timings: {"draft_n":3,"draft_n_accepted":0}

Files

• model.safetensors — 4-layer drafter weights (F32 norms/fc, bf16->fp32 stored).
• config.json — Qwen3-style config incl. dflash_config.
• tokenizer.json, tokenizer_config.json, generation_config.json, processor_config.json, chat_template.jinja — copied from LiquidAI/LFM2.5-VL-3B.
• LFM2.5-VL-3B-DFlash-drafter.gguf — llama.cpp dflash GGUF (F16, block_size 7, sample_from_anchor true, mask_token_id 125017, target_layers [2, 11, 19, 28] hidden-state indices).

License

Mirrors the upstream license where applicable. Trained weights are released for research.