Federal agencies lack critical information about some of their most risky AI systems

Federal agencies are acquiring dozens of proprietary AI algorithms for tasks that could affect people’s physical safety and civil rights without having access to detailed information about how the systems work or are trained, according to recently released data.

Customs and Border Protection and the Transportation Security Administration lack documentation of the quality of data used to build and evaluate algorithms that scan travelers’ bodies for threats, according to the agencies’ 2024 inventory. reports.

The Veterans Health Administration is in the process of acquiring an algorithm from a private company that is supposed to predict chronic diseases among veterans, but the agency said it is “not clear how the company obtained the data” on the veterans’ medical records it used for training the model.

And for more than 100 algorithms that could affect people’s safety and rights, the agency using the models did not have access to the source code that explains how they work.

Like the future Trump administration being prepared for scrap recently introduced rules on federal AI procurement and safety, the inventory data shows how heavily the government has come to rely on private companies for its riskiest AI systems.

“I’m really worried about proprietary systems that take the democratic power away from agencies to manage and deliver benefits and services to people,” said Varun Mathur, who until earlier this month was the White House’s senior adviser on AI, responsible for coordinating AI inventory process. “We have to work hand in hand with our own suppliers. A lot of the time it’s helpful, but a lot of the time we don’t know what they’re doing. And if we don’t have control over our data, how are we going to manage risk?”

Internal studies and external investigations have found serious problems with high-risk algorithms at some federal agencies, such as racially biased model The IRS uses the IRS to determine which taxpayers to audit suicide prevention algorithm that prioritize white men over other groups.

The 2024 inventories provide the most detailed look yet at how the federal government uses artificial intelligence and what it knows about these systems. For the first time since the inventory began in 2022, agencies had to answer multiple questions about whether they had access to model documentation or source code and whether they had assessed the risks associated with their AI systems.

Of the 1,757 AI systems agencies reported for use during the year, 227 were considered likely to affect civil rights or physical safety, and more than half of those riskiest systems were developed entirely by commercial vendors. (For 60 of the high-risk systems, the agencies did not provide information on who created them. Some agencies, including the Department of Justice, the Department of Education, and the Department of Transportation, have not yet released their AI inventories, and the military and intelligence agencies are not required to they do).

For at least 25 systems affecting safety or rights, the agencies reported that “no documentation exists regarding the maintenance, composition, quality, or purpose of training and evaluation data.” For at least 105 of them, the agencies said they did not have access to the source code. The agencies did not respond to the documentation question for 51 of the tools or the source code question for 60 of the tools. Some of the high-risk systems are still in the development or acquisition phase.

Under the Biden administration, the Office of Management and Budget (OMB) issued new directives to agencies requiring them to implement in-depth evaluations of risky AI systems and to ensure that contracts with AI providers provide access to the necessary information about the models, which may include training data documentation or the code itself.

The rules are stricter than anything AI vendors are likely to face when selling their products to other companies or to state and local governments (although many states will consider AI safety bills in 2025), and government software vendors pushed them back, arguing that agencies must decide what kind of assessment and transparency is needed on a case-by-case basis.

“Trust, but verify,” said Paul Lecas, head of global public policy at the Software and Information Industry Association. “We are careful with the heavy demands on AI developers. At the same time, we recognize that some consideration needs to be given to what degree of transparency is needed to develop the kind of trust that government needs to use these tools.

The US Chamber of Commerce, in comments sent to OMB on the new rules, said “the government should not require any specific training data or datasets for AI models that the government acquires from vendors.” Palantir, a major AI vendor, wrote that the federal government should “avoid overly prescribing strict documentation tools and instead give AI service providers and vendors the necessary leeway to characterize context-specific risk.”

Instead of access to training data or source code, AI vendors say that in most cases agencies should be comfortable with model scorecards — documents that characterize the data and machine learning techniques used by an AI model, but not include technical details that companies consider trade secrets.

Cary Miller, who helped develop international standards for purchasing algorithms and co-founded the nonprofit Artificial Intelligence Procurement Lab, describes the scorecards as a lobbyist’s solution, not a bad starting point, just a starting point for what which vendors of high-quality risk algorithms should be contractually obligated to disclose.

“Procurement is one of the most important governance mechanisms, it’s where the rubber meets the road, it’s the front door, it’s where you can decide whether or not to let bad things in,” she said. “You need to understand if the data in this model is representative, is it biased or is it unbiased? What did they do with this data and where did it come from? Does everything come from Reddit or Quora? Because if so, it might not be what you need.

Like OMB noted when it rolls out its AI rules, the federal government is the largest single purchaser in the US economy, responsible for more than $100 billion in IT purchases in 2023. The direction it takes on AI — what it requires vendors to disclose and how it tests products before deploying them — will likely set the standard for how transparent AI companies are about their products when selling to smaller government agencies or even others private companies.

President-elect Trump has strongly signaled his intention to roll back OMB rules. He campaigned on a party platform that calls for “repeal (of) Joe Biden’s dangerous executive order that impedes AI innovation and imposes radical left-wing ideas on the development of this technology.”

Mather, the former White House senior adviser on artificial intelligence, said he hoped the incoming administration would renege on that promise and pointed out that Trump had boosted efforts to build trust in federal artificial intelligence systems with his executive order in 2020

Simply getting agencies to inventory their AI systems and answer questions about the proprietary systems they use has been a monumental task, Mathur said, that has been “extremely helpful” but requires follow-up.

“If we don’t have the code, the data or the algorithm, we won’t be able to understand the impact we have,” he said.