---

AI Application Testing for Developers

GitHub Repo The AI Alliance

Tips:

1. Use the search box at the top of this page to find specific content.
2. Capitalized, italicized terms link to a glossary of terms.

Welcome to the The AI Alliance project to advance the state of the art for Developer Testing for Generative AI (“GenAI”) Applications.

Using nondeterministic, Genenerative AI Models in an application makes it difficult to write Deterministic, Repeatable, and Automatable tests. This is a serious concern for application developers, who are accustomed to and rely on determinism when they write Unit, Integration, and Acceptance tests to verify expected behavior and ensure that no Regressions occur as the application code base evolves.

What can be done about this problem?

Project Goals

The goals of this project are two fold:

1. Research strategies and techniques for testing Generative AI applications that eliminate nondeterminism, where feasible, and enable effective Repeatable and Automatable testing, where not feasible.
2. Publish guidance for developers on these strategies and techniques here and possibly other venues, like blogs and research papers.

This site will be updated regularly to reflect our current thinking and recommendations.

The content is organized into the following sections:

• The Problems of Testing Generative AI Applications - An explanation of the problems in detail.
• Testing Strategies - How to do effective testing of Generative AI Applications, despite the nondeterminancy.
• Glossary of Terms - Definitions of terms.

Additional links:

• Contributing: We welcome your contributions! Here’s how you can contribute.
• About Us: More about the AI Alliance and this project.
• The AI Alliance: The AI Alliance website.
• Project GitHub Repo

Version History

---