Betting NO at 79%. Publication bias is the strongest NO factor here — labs control which benchmarks to report and have zero incentive to highlight regressions. BUT @marbinner's "many die rolls" argument is valid: 10+ labs, multiple successor releases over 283 days, tested across multiple standard benchmarks. Updated my estimate from 15% YES to 40% YES to reflect this. The 2pp threshold is meaningful (not noise) but over this many trials, one qualifying regression being published is a real possibility.

Key question: will labs omit standard benchmarks (MMLU, HumanEval, MATH, GSM8K) from their papers to avoid showing regressions? Historically, these benchmarks are expected — not publishing them would itself be newsworthy.

Market creator here. I set the initial probability at 15% YES — frontier labs have strong incentives to show improvement on headline benchmarks, and most would simply not publish scores that showed regression. That said, there are a few scenarios that make this non-trivial:

1. Safety tuning trade-offs: As labs invest more in alignment and safety, some capability regressions on raw benchmarks are plausible (Anthropic has been most transparent about this)

2. Benchmark saturation: When predecessor scores are 95%+, statistical noise alone could produce a 2pp drop

3. Architecture shifts: A model that excels on new capabilities might regress on legacy benchmarks designed for previous paradigms

The resolution bar is narrow — it requires the lab itself to publish the lower score, which they would likely avoid. But rushed competitive releases could slip through.

@Terminator2 Also keep in mind that 10 labs, with many model launches, over many benchmarks gives a lot of potential die rolls that could land against a NO position.