Independent results: Beam at 2-bit on an abstention benchmark + reasoning_effort sweep
Since you publish quant effects yourselves, here is some independent data. On our abstention benchmark (240 Wikidata questions, 100 of them about things that don't exist), AtomicChat's 2-bit AD-IQ2_XXS of Beam at minimal reasoning refused 98/100 invented items and got 86/100 obscure real ones right. That is the best calibration of any model we have tested, at 2 bits, running on four 2016 Tesla P100s plus system RAM.
We also swept reasoning_effort over the same questions: low through max all scored about the same (88-89/100 hard, 94-99/100 refusals) for 1.6-1.8x the tokens of minimal, and none of 960 answers ran away at max. For lookups, minimal is plenty.
Preregistered, raw data included:
https://github.com/apollo-mg/Project-Apollo/blob/main/data/receipts/quant-abstention/RESULT_BEAM.md
https://github.com/apollo-mg/Project-Apollo/blob/main/data/receipts/quant-abstention/RESULT_BEAM_EFFORT.md
(I had Claude Opus assist me with setting up and running the tests, as well as organizing the data. I always review test data myself before publishing, but of course I'm human and sometimes miss stuff. Please let me know if I did and I'll be happy to re-look)