NIST Artificial Intelligence Risk Management Framework
The National Institute of Standards and Technology, under the United States Department of Commerce, has published an Artificial Intelligence Risk Management Framework (AI RMF 1.0). The RMF is a good place to start for a comprehensive view of how to quantify risks, understand their manifestations, how to detect them, their impacts, and their management. It emphasizes the need to address AI trustworthiness from the beginning of AI system projects, not just as an “afterthought” considered near the end of system development.
Detecting a risk requires a clear definition, from which metrics that quantify the risk have to be defined, along with how and where to measure them. Impacts include the concept of tolerance; what are the consequences if a risk event occurs and can your organization and customers accept those consequences. Higher consequences mean more effort is required to prevent the corresponding risk events. Understanding consequences helps you prioritize the risks so you can work on those of most importance.
The framework argues that AI systems must have the following characteristics to be considered trustworthy:
- Valid and reliable: Are responses factually accurate? Is the AI system consistently able to perform as expected, including the ability to handle reasonable generalizations from its training data? These qualities are prerequisites for what follows.
- Safe: Used here in the narrow sense that the AI system doesn’t impose risks to health, life, property, etc.
- Secure and resilient: Resilient against cybersecurity threats and unexpected inputs, load, etc.
- Accountable and transparent: AI system behaviors are understandable to the user, which supports accountability for behaviors. Transparency answers the question, “what happened?”
- Explainable and interpretable: How the AI system encodes information and responds to inputs is knowable and the responses are meaningful in the context of the system’s purposes. Explainability answers the question “how” and interpretability answers the question “why”.
- Privacy enhanced: Safeguarding human autonomy, identity, and dignity, limiting observations and disclosures of sensitive information.
- Fair, with harmful biases managed: Free of bias, discrimination, etc. (The broad class of concerns that most people think of first when discussing AI fairness.)
To achieve these characteristics, four “functions” are used. (Use of all capital letters is their convention.)
- GOVERN: Infusing risk management in all aspects of a project’s evolution and culture. This function is supported by the other three functions.
- MAP: Understand the contexts of usage and their implications for risks. This exercise helps surface how risks form a complex network of dependencies, where unanticipated and undesirable interactions occur when smaller “modules” interact.
- MEASURE: Determine how to measure risks and perform those measurements.
- MANAGE: Respond to and recover from risk incidents.
The relationship between these functions are shown in Figure 1.
Figure 1: Functions organize AI risk management activities at their highest level to govern, map, measure, and manage AI risks. (Source: Artificial Intelligence Risk Management Framework (AI RMF 1.0))
These functions are somewhat continuously and iteratively applied over a system’s lifetime. It is reminiscent of the OODA loop - Observe, Orient, Decide, Act - that was developed by United States Air Force Colonel John Boyd for combat operations, where it is essential to constantly update situational awareness, make effective, well-informed decisions, and act upon them. See also this Pacific Northwest Laboratory presentation.
NIST is expanding its AI-related efforts. The Assessing Risks and Impacts of AI program was launched in May 2024. From their website:
The latest in a portfolio of evaluations managed by the NIST Information Technology Laboratory – ARIA will assess models and systems submitted by technology developers from around the world. ARIA is an evaluation environment which is sector and task agnostic.
ARIA will support three evaluation levels: model testing, red-teaming, and field testing. ARIA is unique in that it will move beyond an emphasis on system performance and accuracy and produce measurements on technical and societal robustness.
The program will result in guidelines, tools, methodologies, and metrics that organizations can use for evaluating the safety of their systems as part of their governance and decision-making processes to design, develop, release or use AI technology. ARIA will inform the work of the U.S. AI Safety Institute at NIST. – Source: Assessing Risks and Impacts of AI
We will update this section as more details become available.
For a complementary assessment of risks in AI, see this comprehensive guide to AI security risk analysis from BIML: An Architectural Risk Analysis of Large Language Models.
The next section explores Trust and Safety at Meta.