Kwaipilot

/

KAT-Coder-V2.5-Dev

like 444

Follow

Kwaipilot 652

1. Evaluation method. All metrics presented in the table are reproduced in-house: we download the public model checkpoints, deploy them via vLLM or SGLang, and evaluate under a unified standardized pipeline. No officially reported results of the respective models are directly adopted in this table. Each model is tested only once on each evaluation set; retests are conducted only if obvious errors are found.

Thanks for flagging this. The difference is not a transcription error; it mainly comes from differences in evaluation methodology and configuration.

All values in our table are reproduced in-house from public checkpoints deployed with vLLM or SGLang. We do not directly copy the officially reported numbers from the Ornith or Qwen reports. We use a unified evaluation pipeline, with pass@1 and a single run per benchmark, so our results are not strictly comparable to official graphs that may use different harness versions, agents, toolsets, prompts, or test-set optimizations.

Our main settings are:

• SWE-bench and KAT-Code-Bench: Claude Code 2.1.195, temperature 1.0, top-p 0.95, 256K context.
• Terminal-Bench 2.1: Terminus-2/Claude Code, temperature 0.7, top-p 1.0.
• PinchBench: OpenClaw 2026.3.13, temperature 0.7, top-p 1.0.
• SciCode: temperature 0.6, top-p 1.0.

We also welcome you and the community to download our model and evaluate it independently under your preferred setup. We would be happy to compare results and investigate any meaningful discrepancies together.

I'm intrigued, why choose Qwen3.5 27B instead of Qwen3.6 27B?

Thx for the infos.

Thanks for your great question!

1. When selecting baseline models for comparison, we referenced the evaluation setup adopted by the Qwen team. In their benchmark results for Qwen3.6-35BA3B, they chose Qwen3.5-27B as the dense baseline, so we followed the same baseline selection principle to maintain consistency and fairness.
2. We have also run internal tests on Qwen3.6-27B. The preliminary results show that overall performance of Qwen3.6-27B is still slightly lower than KAT-Coder-V2.5-Dev across our evaluated benchmarks.

Great job!Better than Qwen3.6-27B.

I think Gemma 4 31b typically out performs Qwen 3.6 27b in the SciCode benchmark, so I'm interested in what differences in internal testing show Qwen beating Gemma 4 in that benchmark and how those changes translate to KAT. If the benchmark is run in a way that has Gemma beating Qwen, where would KAT fall in that methodology?

@nothin-net Thank you for your question regarding the SciCode benchmark comparison between Gemma 4 31B and Qwen 3.6 27B.

Our SciCode evaluation setup adopts unified configurations for all tested models to guarantee fairness: pass@k=1, temperature=0.6, top_p=1.0, and a fixed 256k context window. These hyperparameters remain consistent across every model in our test suite, with no special tuning or biased settings tailored for any single model.

Up to now, our model has accumulated over 100,000 downloads (Including download volumes of derivative models), and we have not received any valid community feedback pointing out unfair bias in our benchmark pipeline. Deliberately skewing evaluation results brings no benefit to our side; biased testing will inevitably be verified and exposed by community reproductions, which would only damage credibility.

@liushiyang Can I assume that those are optimal parameters for testing KAT Coder V 2.5 Dev?

I just want to clarify that I'm not suggesting that there was any bias in your benchmarking. I'm actually really excited to test out the model. But, it is the case that on other independent third party benchmarks, Gemma 4 31B typically outperforms Qwen 3.6 27B in the SciCode benchmark. This makes the benchmark stand out to me because most benchmarks have things the other way around; but, in my particular use case, Gemma 4 31B performs better than Qwen 3.6 27B. So, I've been wondering if the SciCode benchmark is a good indicator or performance for my use case. I'll test out KAT Coder V2.5 Dev and see how it performs. If it beats Gemma 4 31B, then SciCode might be a good benchmark for me to look to, if not then maybe it was just a random anomaly. But I do think that people who pay attention to that benchmark will notice Qwen outperforming Gemma as an oddity in your numbers. It's not bias or anything unfair, but it is noticable.

@nothin-net First of all, the evaluation parameters for SciCode were not customized exclusively for KAT-Coder-V2.5-Dev, and we did not make any targeted parameter tweaks for this model. Frankly speaking, such targeted tuning would be pointless anyway.
Secondly, it is not rational to evaluate a model’s overall competence merely by one single benchmark like SciCode. Scores on a single test dataset only reflect the model’s performance within that narrow specific sub-domain, rather than its comprehensive coding capability. That is precisely why we run evaluations across seven different test benchmarks for code tasks instead of relying solely on SciCode alone.

@liushiyang I think something is getting lost in translation here because you are not responding to the questions I'm asking.

Assuming I have a task that I think is similar to what the SciCode benchmark evaluates, what temperature, top p, etc. would you recommend I use? I am currently actively testing the model and so far it does substantially out perform Qwen 3.6 35B a3b in my own testing scenarios, not the SciCode benchmark. However, if I use temperature=0.6, top_p=1.0 for KAT, it does very slightly under perform Gemma 4 31b (tested at optimal parameters for Gemma). If there are some optimal parameters that you recommend for KAT (your model card doesn't say anything about recommended parameters), then I'd like to know what those parameters are. Because, if I can get KAT 2.5 dev to out perform Gemma 4 31b then I will absolutely be switching to using it and telling all of my friends to use it as well. I may even make the switch as is simply because of how much faster KAT 2.5 dev is when compared to Gemma 4 31b; but I'd still like to know what direction I should tweak parameters to try to improve performace.

I think it is entirely rational for you to hold parameters fixed for your in house testing and am not taking issue with that. But for me in the real world, I don't care about how the models perform at a pre fixed set of parameters. I want to know, for my personal use case, if KAT at optimal parameters for KAT can out perform Gemma at optimal parameters for Gemma. Now I have some reason to think that for my personal use case SciCode might be a good indicator of model performance. So if you have optimal parameters for that benchmark, I'd like to know. But if you don't, I'd still like to know if at large you have reccomended parameters because your model card doesn't state any recommended settings (which most models typically do).