Link Search Menu Expand Document
AI Alliance GitHub Organization

AI Safety, Governance, and Education

Collaborate on the necessary safety and governance enablers of successful AI applications. Ensure users and builders of AI systems have the knowledge they need.

Successful AI applications have fundamental requirements for safety and governance, as well as the expertise required to use AI technologies effectively.

AI Safety was one of the most pressing initial concerns for AI systems, encompassing classic cybersecurity, as well as AI-specific concerns, such as suppression of undesirable content and compliance with regulations and social norms. The tools and techniques developed have been generalized to address general concerns about trustworthiness and alignment, which are about ensuring accuracy (i.e., minimizing hallucinations), and ensuring specific application requirements are met, etc. Enterprises won’t deploy AI applications into production scenarios if they don’t trust them to behave as expected.

Governance is related to trustworthiness, specifically the assurances that all end-to-end processes used to create all AI application components are secure, all data is properly licensed and responsibly used, etc. AI models are created with data and they are mostly data themselves. Hence, models also need to be governed.

Finally, Education addresses the needs where organizations and society at large struggle to learn all the things they need to know in order to use AI safely and effectively. Not only has AI introduced new tools and techniques to software application development, it has fundamentally altered some of the ways software works, for example, introducing stochastic behaviors as core aspects of application features, where previously deterministic behaviors were the norm. Most AI Alliance projects have dual missions, not only to innovate and create, but to educate.

The following projects address these concerns.

Safety and Governance

See also the discussion of Testing Generative AI Applications in the Agents and Applications Projects.

Links Description
The AI Trust and Safety User Guide AI Alliance icon1
An introduction to trust and safety concepts from diverse experts, followed by recommendations for how to meet your application's needs. Start here if you are new to trust and safety, then leverage the projects discussed next to implement what you need.

1 The icon indicates an Alliance core project.

Notable AI Alliance Member Projects

The following notable, recommended projects are developed by Alliance members and affiliated organizations.

Links Description
The Coalition for Secure AI (CoSAI)
Several AI Alliance organizations are active participants in CoSAI work groups on AI security, including for MCP and agents. See, for example, this summary of CoSAI's work on MCP security in the AI Alliance guide MCP (and Beyond) in the Enterprise: A User Guide.
Red Hat Lightspeed
An end-to-end system management tool that predicts risks across Red Hat platforms, recommends actions, and tracks costs. Red Hat Lightspeed uses AI-powered package recommendations and planning capabilities to provide targeted guidance on increasing your systems’ day-to-day efficiency. (Principal developer: Red Hat)
unitxt
Unitxt is a Python library for enterprise-grade evaluation of AI performance, offering the world's largest catalog of tools and data for end-to-end AI benchmarking. (Principal developer: IBM Research)
Granite Trust Policy Tools
Granite Trust Policy Tools provides a tool kit for defining, implementing, and enforcing policies, open sourced by IBM. The AI Alliance repo trust-policy-contrib is for community-donated policy definitions that use this tool kit.
DoomArena

AI agents are becoming increasingly powerful and ubiquitous. They now interact with users, tools, web pages, and databases—each of which introduces potential attack vectors for malicious actors. As a result, the security of AI agents has become a critical concern. DoomArena provides a modular, configurable framework that enables the simulation of realistic and evolving security threats against AI agents. It helps researchers and developers explore vulnerabilities, test defenses, and improve the security of AI systems. The DoomArena architecture comprises several key components that work together to create a flexible, powerful security testing environment for AI agents:

  • Attack Gateway: Functions as a wrapper around original agentic environments (TauBench, BrowserGym, OSWorld), injecting malicious content into the user-agent-environment loop as the AI agent interacts with it.
  • Threat Model: Defines which components of the agentic framework are attackable and specifies targets for the attacker, enabling fine-grained security testing.
  • Attack Config: Specifies the AttackableComponent, the AttackChoice (drawn from a library of implemented attacks), and the SuccessFilter which evaluates attack success.

DoomArena offers several advanced capabilities that make it a powerful and flexible framework for security testing of AI agents:

  • Plug-in: Plug into to your favorite agentic framework and environments τ-Bench, BrowserGym, OSWorld without requiring any modifications to their code.
  • Customizable threat models: Test agents against various threat models including malicious users and compromised environments.
  • Generic Attacker Agents: Develop and reuse attacker agents across multiple environments.
  • Defense Evaluation: Compare effectiveness of guardrail-based, LLM-powered, and security-by-design defenses.
  • Composable Attacks: Reuse and combine previously published attacks for comprehensive and fine-grained security testing.
  • Trade-off Analysis: Analyze the utility/security trade-off under various threat models.

Legacy AI Alliance Projects

The following projects were created by the Alliance, but are not currently active. Most still welcome contributions in the form of pull requests. If you find them valuable, please help us restart work on them!

See also the discussion of CUBE - Common Unified Benchmark Environment in the Agents and Applications Projects.

Links Description
Evaluation Is for Everyone AI Alliance icon
Evaluation Is for Everyone addresses two problems: 1) many AI application builders don't know what they should do to ensure trust and safety, and 2) it should be as easy as possible to add trust and safety capabilities to AI applications. Many trust and safety evaluation suites are available that can be executed on the Evaluation Reference Stack. We are making it as easy as possible for AI application developers to find and deploy the evaluations they need. See also the companion Testing Generative AI Applications project. This project is part of the Safety and Governance initiative.
Evaluation Reference Stack AI Alliance icon
The companion projects Testing Generative AI Applications and Evaluation Is for Everyone require a runtime stack that is flexible and easy to deploy and manage. This project is collating popular tools for writing and running evaluations into easy-to-consume packages. This project is part of the Safety and Governance initiative.
Ranking AI Safety Priorities by Domain AI Alliance icon
What are the most important safety concerns for your specific domain and use cases? This project explores these questions in several industries, healthcare, finance, education, and legal, with more to come.

AI in Education and Research

Links Description
Academic Resources for AI AI Alliance icon
A growing catalog of publicly-available resources on how to safely and effectively use AI in education and research. These resources include guidelines adopted at various institutions, courseware, degree programs, research on AI effectiveness in these contexts, etc. It is far from complete and comprehensive, so your contributions are welcome!