Here’s an AI Risk Management Framework We Really Like
Two-week sprints have become two-day sprints. Here’s the risk framework I trust to keep AI-built software safe without slowing teams down.
TL;DR
- Most commercial software QA revolves around validating functional use case scenarios: Does the software fulfill its intended purpose?
- In a regulated industry, there is another layer: Whether you can provide evidence that the product does what it is supposed to do.
- Risk management is yet another layer: What needs to be tested or monitored to ensure that the software doesn’t do harm given the nondeterministic nature of AI?
- This article builds a framework for understanding this risk management layer, which may be new to many QA professionals.
AI has the potential to transform society and people’s lives for the better. It also poses risks that can affect everything from individuals to humanity itself. Many of those risks are unique to AI systems and difficult to understand.
For executives responsible for building and testing software in healthcare and financial services, the NIST AI Risk Management Framework has become a standard that is difficult to ignore. In a regulatory landscape that is fragmented, fast-moving, and politically contested, the NIST AI RMF provides a durable, government-recognized foundation for managing AI risk. It has already persisted through a change in federal AI policy and remains a reference point in procurement, pending legislation, and agency guidance.
For those responsible for delivering and validating AI-enabled systems, the framework translates the abstract goal of “trustworthy AI” into a concrete, testable discipline, which makes it a practical foundation for an AI risk program. Sector-specific frameworks, guidance, and regulations can then build on it rather than replace it. In healthcare, examples include Joint Commission/CHAI guidance and HIPAA; in financial services, the Cyber Risk Institute’s Financial Services AI RMF and existing model-risk requirements add sector-specific requirements.
This article provides a high-level overview of the key components of the NIST AI RMF and shows how I think organizations can apply it to specific AI adoption use cases.
Flexible by Design
The NIST AI RMF is not a checklist of mandatory controls. It does not tell you which tests to run, what thresholds to hit, or which documents to produce. Instead, it defines outcomes that a trustworthy AI program should achieve and leaves implementation to the organization.
That flexibility is deliberate. A single set of requirements could not sensibly govern a hospital’s diagnostic model, a bank’s fraud detector, and a chatbot that answers benefits questions. Instead, the framework is adaptable to context. A system that can affect someone’s health, credit, or civil rights warrants far more scrutiny than a low-stakes internal tool, and organizations are expected to calibrate their use of the framework to each use case and their risk tolerance. The four functions below describe the outcomes to address; how far an organization takes each one depends on how much is at stake.
What “Trustworthy AI” Means in Practice
The framework has a single objective: AI systems worthy of trust. NIST defines that objective through seven characteristics that together make a system trustworthy
Validity and reliability form the foundation: an AI system that isn’t accurate and dependable in real-world conditions can’t be safe, fair, or anything else. The remaining characteristics build on that foundation. A trustworthy system is safe (it won’t endanger people); secure and resilient (it can withstand attacks and misuse); accountable and transparent (someone owns it, and its use is visible); explainable and interpretable (you can describe how it works and why it produced a given result); privacy-enhanced (it protects the data it touches); and fair, with harmful bias identified and managed.
These characteristics can also conflict. Increasing explainability may reduce accuracy, and tightening privacy can limit the data available for bias testing. The framework makes those tensions visible, without resolving them, so trade-offs are made deliberately rather than by accident.
The Core: Govern, Map, Measure, and Manage
The framework’s “Core” consists of four functions that organize the work of managing AI risk.
Three functions form a continuous cycle: Map establishes the context and frames the risks, Measure analyzes and assesses them, and Manage addresses them in priority order. Govern, the fourth function, sits at the center. It encompasses the culture, policies, roles, and accountability that make the other three work and applies across the entire lifecycle rather than as a single step. Because systems and operating conditions change, the cycle repeats: map, measure, and manage again.
Govern is the cross-cutting function. It asks whether the organizational conditions for responsible AI are in place: clear policies and a stated risk tolerance, defined roles and accountability, and a culture in which people can raise concerns and human judgment remains in the loop.
Map is about context. Before you build or buy an AI system, you establish what it is for, how it will be used, and what could go wrong, including risks introduced by third-party components and the potential impact on the people it affects.
Measure provides the evidence. You choose methods and metrics, then evaluate the system against the trustworthy characteristics above, testing factors such as accuracy, security, and bias and tracking how the results hold up over time.
Manage turns measurement into decisions. You prioritize the risks identified through Map and Measure, act on them, and continue monitoring after deployment, with a clear plan for who can intervene or shut the system down if something goes wrong.
For software and QA leaders, much of the testing discipline resides in Measure and Manage. Both functions call for validation and monitoring: independent testing, evaluation, and ongoing observation, the work quality organizations already do well.
The Playbook: From Outcomes to Practice
The Core defines what to achieve. NIST’s companion resource, the AI RMF Playbook, suggests how to achieve it. For each outcome in the Core, the Playbook offers suggested actions, prompts for documentation and transparency, and references to related standards and resources.
The Playbook is meant to be used selectively. NIST does not expect organizations to carry out every suggested action. Teams choose the actions that fit their system and level of risk. This makes the framework’s flexibility workable in practice and gives teams a practical starting point.
Consider a health system rolling out a model that flags patients at risk of sepsis. Using the Playbook, the team might:
- Govern: name an accountable owner, require sign-off before the model goes live, and document how much risk the organization is willing to accept for a tool that influences clinical decisions;
- Map: document the model’s intended use, the patient population it applies to, and the decisions a wrong prediction could affect;
- Measure: define test data and metrics, evaluate accuracy and false-alarm rates across patient groups, and set thresholds that trigger human review;
- Manage: decide in advance who can override or shut down the model and establish monitoring to detect performance drift after go-live.
A bank deploying a credit-decisioning model would apply the same four functions but substitute fair-lending testing, adverse-action explainability, and model-risk documentation for clinical validation. The framework remains the same, but the tailoring changes.
In financial services, the Cyber Risk Institute, working with more than a hundred institutions, has published a Financial Services AI RMF that is structurally aligned with the NIST framework and expands it with roughly 230 control objectives. It maps those controls to an organization’s AI adoption stage, so a firm just beginning to use AI and one already operating mature, high-impact systems receive recommendations suited to their level of adoption. For financial institutions, it provides a ready-made, sector-specific path for putting the NIST framework into practice.
Where to Start
For regulated organizations, I believe the NIST AI RMF provides a stable, widely recognized way to manage AI risk without imposing rigid rules. Sector-specific guidance and regulation can build on it while the framework remains the common foundation. For teams in healthcare and financial services, it is a sensible place to begin. Because much of the Measure and Manage work involves testing, evaluation, and ongoing monitoring, the framework also aligns naturally with the quality practices these organizations already rely on.
Where to Learn More
These primary references provide additional details.
NIST AI Risk Management Framework (AI RMF 1.0): https://www.nist.gov/itl/ai-risk-management-framework
NIST AI RMF Playbook: https://airc.nist.gov/AI_RMF_Knowledge_Base/Playbook
NIST Trustworthy & Responsible AI Resource Center: https://airc.nist.gov
Financial Services AI RMF (Cyber Risk Institute): https://cyberriskinstitute.org/artificial-intelligence-risk-management/
