ZeroBench is a benchmark for visual reasoning, introduced by Roberts et al. in "ZeroBench: An Impossible Visual Benchmark for Contemporary Large Multimodal Models" (https://arxiv.org/abs/2502.09696).

This market will use the variant of the benchmark frozen one week after the initial release (following the public benchmark red-teaming stage to identify flawed/ambiguous questions).

The temperature used for the 5/5 reliability evaluation will be the default setting provided by each LLM API provider. In cases where this default is ambiguous to determine, we will default to a temperature of 0.7.