Consulting
Workable AI Safeguards for Agency Chief AI Officers
Independent analysis by Starro. Not client work.

Objective
Sizing the OMB Minimum Practices
OMB M-25-21 requires seven minimum risk management practices for high-impact federal AI, but it does not say how much testing, monitoring or appeals capacity is enough. Agencies and their contractors need numbers they can defend to an auditor.
How much testing, monitoring and appeals capacity does a high-impact AI system need to meet the minimum practices?
Findings
Test, Monitoring and Appeals Capacity in Numbers
Detecting an 8 % against 5 % error-rate gap between subgroups takes about 1,059 labeled test cases per group. With 100 audited decisions a week, a CUSUM chart detects a rise from 5 % to 8 % in 3.8 weeks on average, against 9.7 weeks for a three-sigma p-chart.
Meeting a two-day start target for appeals takes 2 reviewers when 10 % of adverse decisions are appealed, and 4 at twice that volume.

Approach
Practices Converted to Evidence and Sized on a Triage System
Each of the seven practices becomes an evidence artifact with an owner, cadence and pass criterion. The three quantitative practices, pre-deployment testing, ongoing monitoring and appeals, are sized for a synthetic benefits-claim triage system deciding 50,000 claims a year.

Validation
Simulation Checks and the Official Memo Text
How we checked the analysis:
Each sizing is checked by simulation.
Appeals staffing agrees with a discrete-event simulation of the queue.
The memo text, including Appendix 4(b) and the waiver terms, was checked against the official OMB copy in November 2026.
Recommendations
Produce Records an Auditor Can Check
Turn each practice into a record an inspector general can check: a test sized to the gap that matters, a monitoring chart sized to catch drift in weeks rather than months, and an appeals queue staffed for surges. Confirm the memo is still in force before relying on it.
Tools used:
Statistical sizing with simulation checks; discrete-event appeals queue


