Alibaba's Qwen 3.8-Max arrived this week claiming superiority over GPT-5, Claude Opus 5, and other top models on agentic computer use tasks. Alibaba's own benchmarks showed it leading Claude Fable 5 on one of twelve coding-agent rows. Yet independent testing told a sharply different story. A benchmark run using the preview version placed Qwen 3.8-Max in mid-pack at best settings and dead last at default settings.
Both results are accurate. The discrepancy hinges on token and time budgets, the computational resources each model receives to solve problems.
Benchmark scores alone reveal little about real-world performance. Two models with identical accuracy ratings can deliver vastly different results depending on how many tokens they consume per query and how much time they spend reasoning through tasks. Qwen 3.8-Max likely performs well when given extended reasoning time and token allowances, matching or exceeding competitors under generous constraints. Default settings, which most users encounter, tell a different story.
This gap between laboratory claims and practical performance applies across the LLM landscape. Vendors optimize for headline-grabbing benchmarks using parameters that differ from production environments. A model trained for extended reasoning may shine on standardized tests but arrive slower and costlier in actual deployments.
The Claude Opus and Qwen comparison exposes a persistent problem in AI evaluation. Marketing departments point to favorable metrics. Independent researchers run identical models under different conditions and get opposite conclusions. Neither party lies. The methodology divergence makes direct comparison nearly impossible.
This matters for builders choosing models for production systems. A model ranking first in labs might rank fifth in real deployments if token budgets tighten or latency constraints kick in. Performance benchmarks require transparency about computational constraints, not just accuracy numbers.
Alibaba and other vendors must dis