Inside theSoftware Factory
On deck

Databricks logoDatabricks' AI coding benchmark built from real PRs

Real merged PRs with held-out tests become the benchmark that guides model and harness choices across a multi-million-line codebase.

What they built

An internal coding benchmark built from actual merged PRs across Databricks' codebase. Tasks span Scala, Go, Rust, Python, TypeScript, and Bazel. The benchmark evaluates models and harnesses on real engineering work to guide cheaper, more efficient coding-agent choices.

How it works

Unity Gateway captures logs of coding interactions. Candidate PRs are filtered for recency, human authorship, and high-quality tests. Each task is constructed by summarizing the PR's intent as a prompt, splitting out tests as the validation set, and running agents against the code. Tests grade correctness rather than an LLM judge. Results cluster into capability tiers, revealing that open models like GLM 5.2 can match Opus 4.8 quality at lower cost, and that harness choice dramatically affects efficiency.

Key numbers

$1.28GLM 5.2 cost per task
$1.94Opus 4.8 cost per task
~3×less context per turn with Pi harness

People

Vinay Gaba
Ankit Mathur
Rishabh Singh
Patrick Wendell
Matei Zaharia

Related patterns

More On deck

We're keeping an eye on Databricks. On deck companies aren't yet placed on a maturity level. With more detail on their whole setup, they'll get a full profile and a level.
Sources (1)
1
Benchmarking Coding Agents on Databricks' Multi-Million Line Codebase
Vinay Gaba, Ankit Mathur, Rishabh Singh, Patrick Wendell, Matei Zaharia · 2026-07-08