IsValorum

/

Cyber-Tiel-Coder-35B-A3B-APEX-I-MiniPlus-V2.1-GGUF

Like 35

I don't know why, but sometimes the score falls short of 1.2 points, consistantly at the given task coding-05 scoring 85/100. It is affected by reasoning level, appearing from medium to minimal. It might also be affected by k/v-cache quantization, context size and compaction.

Thanks for the follow-up. I compared the compacted run (2d46ce8b, 98.3) with the 100/100 one (32804f02). Everything else was 100 except coding-05 and systems-02. In coding-05, all the retry and vendor checks passed; the only missing verifier is string_id (15 pts), which drops it to 85/100. systems-02 is missing mixed_schema (5 pts), so it lands at 95/100. That means the retry logic is fine. It looks like the failure is type/schema fidelity: the model normalizes a string ID and collapses a mixed schema when reasoning is low or history is compacted. The 1.2 points you saw is exactly this: 15/12 = 1.25.

Your q4_0 + minimal run at 100 makes the KV cache an unlikely cause. Both 100/100 runs were at high effort with different context sizes (96K and 160K), so context size is unlikely too. And since the same build passes at high effort, it's not the weight quantization either. It points to compaction plus medium/minimal effort. The template's low/minimal guidance only says to keep reasoning brief, so it can take the shortcut of normalizing types.

For schema-sensitive tasks, stay on high effort and keep compaction off. If you need compaction, raise the threshold and keep this outside the compacted region: "Preserve exact field types and schema. String IDs must remain strings even if numeric-looking; never coerce identifiers. Preserve mixed/heterogeneous schemas; do not normalize or unify field types. Verify these invariants before finishing."

I'll add this to the model card as a known limitation. The compacted run page doesn't record the reasoning effort, so if you have the run where coding-05 hit 85 with medium/minimal, share it and I'll confirm it's the same verifier. Thanks for testing it so thoroughly.

I am currently trying to generate as much empirical data on that matter as I can. There is something going on that seems to be non-deterministic from my perspective. I also question whether my system actually is fully stable. It does not produce WHEA, but there might be uncaught AVX2 miscalculation due to my Intel 14700K being undervolted. It also might be IMC related, or busy pci-e bus? I currently not have physical access to that machine to rule that out.

I have attached the raw data and I will continue to do so while I progress through the test setup.

each test:

• change in context + quantization -> different moe offloading param*
• fresh llama.cpp instance
• fresh pi instance from empty folder
• test temp folder swipe**
• no changes except for quantization, context size params and reasoning level
• one single change per run
• one run per configuration, no benchmaxing

what's changed to prior runs/ published results:
Instead of the integrated compaction of Pi I am currently using npm:@pi-unipi/compactor.
Before that I was using Billion-Context, which lead to truncated tool calls during compaction with some Qwen-based models, especially when utilizing an jinga template text file.
Recently the creators of Billion-Context have introduced transient dependencies with high vulnerability ratings according to npm. So I had to drop them for good.
Also I've tried out pi-vcc, which had good compaction performance but lacked summary. Manually adding a package called pi-observational-memory
resulted in excessively prolonged compaction with measurable negative impact on coding quality.
So I tried pi-blackhole, which combines pi-vcc and pi-observational-memory in a single package. But it's slop and not worth risking code quality in production critical repositories.
The current package "pi-unipi/compactor" has been used for all of my current measurements and seems to be superior so far, also utilizing vcc.

*) I set it as low as it handles a standardized question with the fastest prompt processing "what's the weather forecast for [location] today?"
I am aware that this is not scientific at all. I believe that I activate the same experts that way. It is not going to paint the whole picture across an entire benchmark run, but it's what I do to at least have some kind of consistency and adaptive performance. In the end time is relevant to our benchmarks.

**) I still need to verify that Gitnexus' graphs on prior benchmark data influence future benchmark runs, but it seems so.

I also want to point out that q8, medium effort, context size 65536 completed just shy of 12 minutes (719.5 seconds) in this run: https://shortbench.vercel.app/runs/022562a6-4589-40c0-a839-8ab8768451b6
😉

*edit: before this gets out of hand I'll dial down the temperature und retest