MaxxBench tests adaptation to changed coding requirements

A small request can expose a large failure

The authors modify instructions or starting code in 89 DeepSWE tasks. Their examples include changing report filenames to lowercase. They report agents sometimes favoring imagined hidden tests over the revised request.

MaxxBench research introduction ↗

Inspect the revised test and solution

The repository separates instructions, environments, tests and reference patches. A separate container grades the submitted patch. Model-provider usage and container resources add costs; the published ranking was not rerun here.

MaxxBench implementation ↗

What you can do

1. Compare a modified task’s instructions with its reference patch to see exactly which changed requirement the tests enforce. Open reference ↗

Where the evidence stops

These are author-reported results on a bounded benchmark. Changes in prompts, runtime or model versions can affect comparisons.

Sources read. Public documentation read November 9, 2026 UTC. No installation, model call or complete workflow executed.

Sources & further reading