MCP-Bench: Benchmarking Tool-Using LLM Agents with Complex Real-World Tasks via MCP Servers
Paper
ā¢
2508.20453
ā¢
Published
ā¢
63
AI & ML interests
Natural Language Processing, Computer Vision, Speech, Reinforcement Learning, Graph Neural Networks
eugenesiowĀ
authored
a
paper
4 months ago
eugenesiowĀ
posted
an
update
9 months ago
Post
1912
GPT-4.1 dropped this week - and it puts OpenAI back in the race for coding & agentic leadership.
āļø API only - no ChatGPT toggle for this.
š» Coding performance is back on par with Claude 3.7 Sonnet & Gemini 2.5 Pro (though Gemini still leads).
šø Pricing:
ā¢ Full: $3.50 / 1M tokens
ā¢ Mini: $0.70 / 1M
ā¢ Nano: $0.17 / 1M
š Gemini 2.5 Pro = best price/perf ($3.44 / 1M)
šµ Claude 3.5 Sonnet = $6 / 1M (!)
š§ Not a "thinking" model.
š Mini shines on general reasoning tasks (e.g. GPQA), but only the full model holds up in SWE-bench-verified (GitHub issue solving).
āļø API only - no ChatGPT toggle for this.
š» Coding performance is back on par with Claude 3.7 Sonnet & Gemini 2.5 Pro (though Gemini still leads).
šø Pricing:
ā¢ Full: $3.50 / 1M tokens
ā¢ Mini: $0.70 / 1M
ā¢ Nano: $0.17 / 1M
š Gemini 2.5 Pro = best price/perf ($3.44 / 1M)
šµ Claude 3.5 Sonnet = $6 / 1M (!)
š§ Not a "thinking" model.
š Mini shines on general reasoning tasks (e.g. GPQA), but only the full model holds up in SWE-bench-verified (GitHub issue solving).