Inside theSoftware FactoryWhat is a software factory?
Software factory pattern

The quality gate

The checkpoint that decides whether an agent-written change is safe to merge, and what to look for when deciding on one.

What it is

The checkpoint that decides whether a change an agent wrote is safe to merge, sitting between "the agent opened a PR" and "the PR is live." Industry has already settled on a name for it, quality gate, and the same shape recurs everywhere it's implemented: layered automated checks that triage most diffs on their own, escalating only the ones that need a human, with the decision to block or merely advise made deliberately rather than by default.

Fig. 1
Agent-authored diff code, tests, or a PR Quality gate risk score · triage Auto-merge low-risk, deterministic Human reviewer higher risk, escalated
Every agent-authored diff passes through the same gate; the gate decides whether it merges on its own or escalates to a person, not the diff's author. Hover a box to trace its connections. Original diagram, not a reproduction. Click to enlarge.

Why you need it (in a large team)

  • Meta's own review paper opens with the number that makes this pattern non-optional. Lines of code per diff rose 105.9% year over year, attributed to AI-assisted authoring outrunning human review capacity. Every other company in this corpus describes that squeeze qualitatively; Meta measured it and had it peer-reviewed.
  • Once agents can author faster than humans can read, review capacity becomes the binding constraint on the whole pipeline. Adding more agents without touching review just moves the bottleneck downstream, it doesn't remove it.
  • GitHub's practitioner framing for what breaks without a gate is the sharpest in the corpus. "Hallucinated correctness," code that looks plausible and reads clean but is wrong in a way ordinary review misses, alongside CI checks quietly weakened, duplicate utilities reinvented, and prompt-injection risk in workflow-triggered agents. A quality gate exists to catch exactly the failure modes a human skimming a diff is least equipped to catch at agent volume.

How it should work

The pattern that recurs across every serious implementation is layering, cheap deterministic checks first, then a model, then a human, never a single LLM pass standing in for review. Meta's RADAR runs static exclusion rules, then risk-percentile gating from its Diff Risk Score, then an LLM reviewer that must clear at least 8 out of 10 confidence to approve, then deterministic validation; deterministic codemods skip per-diff review entirely, but AI-generated ones do not. Uber's uReview runs specialised comment generators, then confidence scoring and semantic deduplication, then inline delivery, then developer ratings feeding back into evaluation, all inside a median four-minute CI turnaround.

Whether the gate blocks or advises is a real design decision, not a default. Uber built uReview as a second reviewer that engineers keep trusting rather than a gatekeeper, and states plainly where it's weak: no access to past PRs, feature flags, schemas, or docs, and better at catching bugs than at judging design. Atlassian went further upstream and gates the input instead of the output, a classifier at 97% precision and 50% recall decides whether a ticket is even suitable for an agent before one runs, and rather than showing engineers a confidence score, it coaches them on writing a better ticket. GitHub took the governance question and made it explicit: agent PRs re-enter the same branch-protection and required-reviewer pipeline as human PRs, and agents are barred from approving their own work.

Publishing the failure rate, not just the success rate, is what separates a credible gate from a marketing claim. Meta reports a revert rate of 1 in 13.3 and a production-incident rate of 1 in 50 against non-RADAR diffs. Atlassian's HULA paper states that 82% of agent plans get approved and 59% of generated PRs merge, which means the paper is also stating that 41% didn't, and it says so plainly rather than only quoting the 59%.

Where it fits into a software factory

This is the last checkpoint before a change an agent produced touches production, sitting downstream of everything else in this series: the model router and MCP gateway give an agent somewhere to reach and something to reach it with, shared agent context tells it what "correct" looks like going in, and the quality gate is what checks whether it actually got there once the work exists. Nothing upstream of it can substitute for it, better context reduces how often the gate has to catch something, but it doesn't remove the need for the gate itself.

What you should look for when deciding on one

  • Blocks a merge or only advises, decided up front, as a trust decision, not a technical one. Uber chose advisory specifically to keep engineers believing the tool; a gate that blocks too aggressively or too often gets routed around instead of trusted.
  • The failure rate, not just the catch rate, with a denominator attached. "59% of PRs merged" and "41% didn't" are the same fact stated two different ways, and a vendor or internal team that only ever states the first one is telling you less than it sounds like.
  • Layered checks or a single LLM pass standing alone. Meta's static rules and deterministic validation exist specifically because an LLM reviewer alone isn't reliable enough to gate production changes by itself, the model handles judgment, deterministic checks handle everything that shouldn't need judgment in the first place.
  • What the gate admits it can't see. Uber's own post lists uReview's blind spots, past PRs, feature flags, schemas, docs, rather than letting the tool's coverage numbers speak for its limits. A gate that only publishes wins is harder to trust than one that publishes both.

Companies using a quality gate for AI-written code

← All 9 organisations