Knowledge and calibration check vs the unpruned GSQ-RCO IQ3_XXS

#19
by apollo-mg - opened

Thanks for the release. The code claim holds up on my side: HumanEval+ 93.3% for the Coder vs 93.9% for your unpruned GSQ-RCO IQ3_XXS (164 problems, temp 0, thinking off).

I also ran a factual-knowledge and calibration check against that same unpruned parent (same pipeline, same server, 4x P100, llama.cpp/buun):

714 factual probes (4 popularity tiers), thinking off: 63.3% correct -> 32.9% (paired McNemar p ~ 1e-45).
Of the answers it committed to, the wrong share went from 14% -> 52%. Refusals stayed at 0% in both, so the model does not become unsure about what it lost; it answers anyway.
The loss is concentrated in long-tail entity facts: rarest tier 45.6% -> 9.4%, Wikidata entity probes 64% -> 20%, while general-knowledge probes barely moved.
On a separate 240-question set, first-level capitals went 0.90 -> 0.28, with confident answers like "the capital of Maranhao is Recife" (it is Sao Luis).

This matches what I measured for REAP pruning on the same base (-20 pts, wrong share 14% -> 50%), so it seems to be a property of removing experts that the calibration mix doesn't exercise, not of RCO specifically. A suggestion: pruned releases could report a calibration check against the parent (wrong-answer rate on factual and unanswerable questions) next to the capability scores, since the benchmarks the pruning targets pass by construction.

Caveats: one quant tier each, thinking off, no agentic/SWE-bench run.

Full numbers and method: https://github.com/apollo-mg/Project-Apollo/blob/main/data/receipts/coder-prune/RESULT_CODER_PRUNE.md

(Test designed and run with my agent, Claude; I reviewed it before posting.)

Sign up or log in to comment