Real merged PRs with held-out tests become the benchmark that guides model and harness choices across a multi-million-line codebase.
An internal coding benchmark built from actual merged PRs across Databricks' codebase. Tasks span Scala, Go, Rust, Python, TypeScript, and Bazel. The benchmark evaluates models and harnesses on real engineering work to guide cheaper, more efficient coding-agent choices.
Unity Gateway captures logs of coding interactions. Candidate PRs are filtered for recency, human authorship, and high-quality tests. Each task is constructed by summarizing the PR's intent as a prompt, splitting out tests as the validation set, and running agents against the code. Tests grade correctness rather than an LLM judge. Results cluster into capability tiers, revealing that open models like GLM 5.2 can match Opus 4.8 quality at lower cost, and that harness choice dramatically affects efficiency.