Qwen 3.8-Max and Claude Opus 5 Reveal the Flaws in Raw Benchmark Scores for Performance Prediction

Alibaba unveiled its latest model, Qwen 3.8-Max, this week, positioning it as a key contender in the AI landscape, just behind Claude Fable 5. However, independent tests suggest a different narrative. Initial benchmarks indicated Qwen 3.8-Max performed moderately, with default settings trailing behind competitors.

The disparity in results can be attributed to varying time and token budgets. Alibaba allocated substantial resources for their assessments, allowing up to five hours for coding evaluations and twelve hours on tasks using PaperBench. conversely, an independent benchmark, VulcanBench, limited its testing to 45 to 60 minutes. This difference in resource allocation explains the contrasting performance outcomes.

To navigate these discrepancies, two key practices should be adopted. First, it’s essential to calculate costs based on successful task completions, considering all attempts, including failures, divided by the number of tasks that met acceptance criteria. Second, time or token budgets should be explicitly outlined in the acceptance parameters, ensuring they are not overlooked.

Cost Per Token No Longer Reflects True Expenses

In the initial week of Qwen 3.8-Max’s release, price comparisons dominated discussions since there was limited data available. The model comes with a high price tag: $2 per million input tokens and $6 for outputs. In contrast, competitors like DeepSeek-V4-Flash-0731 offer tokens at a lower rate of 14 cents for input and 28 cents for output.

These prices can be misleading, especially for reasoning models like Qwen, where achieving results consumes valuable token resources. If a model exhausts its token allowance before delivering an answer, it could yield an empty result that resembles a total failure, despite the high cost of the operation.

Artificial Analysis has highlighted how spending can vary significantly. While running their Intelligence Index on DeepSeek-V4-Flash incurred a cost of 210 million output tokens – significantly above the median of 100 million – the overall expenses remained low due to token affordability. However, high verbosity can lead to time losses, impacting results adversely.

Therefore, a comprehensive metric that encompasses all spending, including failed attempts, against completed tasks is vital. This framework sheds light on true efficiency and effectiveness in task completion.

Understanding Failure Rates: Configuration Matters

Distinguishing between outputs that yield incorrect answers and those that simply exhaust their budget is critical, yet few benchmarks do this. While constructing my own benchmark, I noticed failures were logged without detail, prompting me to introduce this distinction myself. Once separated, it’s evident that budget exhaustion is a leading cause of failures.

A recent evaluation, Long-Horizon-Terminal-Bench, highlighted that 79% of unresolved runs were the result of timeouts, while only 19% resulted from agent failures and 3% from harness errors. The authors noted that many timeout instances were not close to completion, emphasizing that additional time may not guarantee success. This situation illustrates that benchmarks often measure time efficiency, regardless of their stated purpose.

The mechanism behind this was apparent in VulcanBench’s reports. Claude Opus 5 demonstrated that its lowest-effort setting outperformed its high-effort counterpart by solving more tasks. While higher effort reduced error rates, it frequently ran into time constraints, resulting in no score. This outcome offers implications for those designing routing strategies, as simply escalating cost due to a failed attempt may be deceptive; it could lead to further timeouts.

Emergence of Cost Per Successful Task Metrics

In recent months, various organizations have independently recognized the importance of measuring cost per successful task, signaling a growing standard in the industry.

VulcanBench has been incorporating dollars per solved task prominently in its reports. Similarly, Long-Horizon-Terminal-Bench provides insights into costs alongside accuracy, revealing that some models can be considerably more expensive with lower success rates. Additionally, TestEvo-Bench assesses agents under financial constraints, evidencing shifts in performance under stricter budgets.

Companies are beginning to adopt this approach. HubSpot transitioned its Breeze Customer Agent to charge 50 cents per resolved conversation, down from a dollar, while Zendesk implements billing based on automated resolutions. Fin is another example, charging only for successful outcomes at a rate of 99 cents.

Essential Changes for This Week

• Require a detailed failure reason for every agent run, categorizing issues like budget exhaustion, verifier failure, and harness errors as distinct types. Until this is addressed, pass rates will remain ambiguous, complicating improvements.

• Calculate costs per successful task for varying effort levels, incorporating all expenses, including failed attempts. This approach may lead to unexpected rankings where the least expensive settings outperform expectations.

• Implement token limits rather than wall clock limits, unless latency is an essential criterion in service levels. This shift allows for a more accurate assessment of model performance without penalizing for provider delays.

• Review the default effort settings for deployed models. For example, Qwen 3.8-Max defaults to its highest reasoning level unless specified otherwise, which has led to it being rated poorly in independent evaluations. Ignoring these parameters could result in higher costs per successful task.