Achieving Confidence in Enterprise AI Applications

(Previous Title: AI Application Testing for Developers)

How to Test AI Applications, for Enterprise Developers

If you are an enterprise developer, you know how to test your traditional software applications, but you may not know how to test your AI-powered applications, which are uniquely nondeterministic. This project is building the knowledge and tools you need to achieve the same confidence for your AI applications that you have for your traditional applications.

Welcome to the The AI Alliance project to advance the state of the art for Enterprise Testing of Generative AI (“GenAI”) Applications.

Tips:

1. Use the search box at the top of this page to find specific content.
2. Italicized terms link to a glossary of terms.

The Challenge We Face

We enterprise software developers know how to write Repeatable and Automatable tests. In particular, we rely on Deterministism when we write Unit, Integration, and Acceptance tests to verify expected behavior and to ensure that no Regressions occur as our code base evolves. These are core skills in our profession. They give us essential confidence that are applications meet our requirements and implement the use cases our customers expect.

Problems arise when we introduce Genenerative AI Models, which are inherently nondeterministic, into our applications. Can we write the same kinds of tests now? If not, what alternative approaches should we use instead?

In contrast, AI experts (model builders and data scientists) use different tools to build their confidence in how their models perform. Specifically, Probability and Statistics, tools that were developed long before Generative AI came along, are used to understand the probabilities of possible outcomes, analyze the actual behaviors, and to quantity their confidence in these models.

Figure 1: The spectrum between deterministic and probabilitistic behavior.

We have to bridge this divide. As developers, we need to be able to adapt these data science tools to meet our needs. We will need to learn some probability and statistics concepts, but we shouldn’t need to become experts. Similarly, our AI expert colleagues need to better understand our needs, in order for us to take their work and deliver reliable, trustworthy products that use AI and use it confidently.

Project Goals

The goals of this project are two fold:

1. Develop and document strategies and techniques for testing Generative AI applications that eliminate nondeterminism, where feasible, and where not feasible, still allow us to write effective, repeatable and automatable tests.
2. Publish detailed, reusable examples and guidance for developers and AI experts on these strategies and techniques.

NOTE: This is very much a work in progress. This site will be updated frequently to reflect our current thinking, emerging recommendations, and reusable assets. Your contributions are most welcome!

The website is organized into the following sections:

• The Problems of Testing Generative AI Applications - An explanation of the problems in detail.
• Testing Strategies - How to do effective testing of Generative AI Applications, despite the nondeterminancy.
• Glossary of Terms - Definitions of terms.
• References - Useful sources of additional information, some of which motivated the ideas here.

Getting Involved

Are you interested in contributing? If so, please see the Contributing page for information on how you can get involved. See the About Us page for more details about this project and the AI Alliance.

Additional Links

• This project’s GitHub Repo
• Companion projects:
  • Evaluation Is for Everyone
  • Evaluation Reference Stack
• The AI Alliance:
  • Website
  • The Trust and Safety Work Group
  • Trust and Safety Evaluation Initiative

---