What breaks AI pilots isn't the model, it's the ATO
Teams build against the wrong environment, bolt explainability on after the fact, and treat the compliance review as a formality, then discover the pilot was a rebuild from day one.
TL;DR
Patrick Bryant, who runs an AI implementation firm working with small and mid-size government contractors, lays out the pattern: agencies have pilots running everywhere, but few systems make it through compliance review into an ATO package for daily use. The models work. What breaks is everything around them, the wrong cloud environment, explainability treated as documentation rather than design, human-in-the-loop reviews that collapse under volume, and vendor-risk paperwork nobody scoped time for. The fix isn't flashy: build the first version inside the actual authorization boundary and run the compliance workstream from day one.
Patrick Bryant, writing in Federal News Network, identifies a gap that will sound familiar to anyone who's shepherded a system through a federal ATO: the models are ready, but nothing around them is.
He names four failure patterns. First, pilots get stood up on public API keys in unmanaged cloud tenants with convenient test data, the demo sings, but the compliance reviewer asks where the data lives and whether the provider's terms of service match agency requirements. At that point the pilot isn't six weeks from production. It's a rebuild.
Second, explainability gets bolted on after the model is built. When an IG or program counsel asks why a recommendation was made and what happens when it's wrong, teams discover the architecture doesn't support the answer, the model is a black box by construction. Systems that survive review bake the question "how do we explain this to an auditor?" into design, not documentation.
Third, human-in-the-loop requirements wither under real-world volume. Bryant notes that reviewers' actual review time per item drops as throughput rises, turning a genuine safeguard into a theoretical one. Compliance officers are now asking about actual review throughput, not just whether a review step exists.
Fourth, vendor documentation (FedRAMP status, SOC 2, subprocessor lists, data-residency commitments) gets started too late. The code is ready, but the capability sits in limbo for months while paperwork catches up.
Bryant's prescription isn't complicated: build the first version inside the actual authorization boundary, even if that means a slower start, and treat the compliance package as a parallel workstream from day one rather than a post-construction gate. A prototype that has to be thrown away because it lived in the wrong environment costs more time than starting inside the constraints.
Published ·Deep Fathom