Link Search Menu Expand Document

References

References for more details on testing, especially in the AI context. Note that outside references to particular tools are not shown here.

Table of contents
  1. References
    1. Adrian Cockcroft
    2. Alignment Forum
    3. CVS Health
    4. Dean Wampler
    5. EleutherAI
    6. Evan Miller
    7. James Thomas
    8. John Snow Labs and Pacific.ai
    9. Merriam-Webster Dictionary
    10. Michael Feathers
    11. MLCommons Glossary
    12. Nathan Lambert
    13. NIST Risk Management Framework
    14. OpenAI
    15. Wikipedia

Adrian Cockcroft

Dean Wampler and Adrian Cockcroft exchanged messages on Mastodon about lessons learned at Netflix, which are quoted in several sections of this website. See also Dean Wampler

Alignment Forum

The Alignment Forum works on many aspects of Alignment.

CVS Health

CVS, the US-based retail pharmacy and healthcare services company, has a large data science team. They recently open-sourced uqlm, where UQLM stands for Uncertainty Quantification for Language Models. It is a Python package for UQ-based LLM hallucination detection.

Among the useful tools in this repo are:

Dean Wampler

In Generative AI: Should We Say Goodbye to Deterministic Testing? Dean Wampler summarizes the work of this project. After posting the link to the slides, he and Adrian Cockcroft discussed lessons learned at Netflix.

EleutherAI

EleutherAI’s definition of alignment is quoted in our glossary definition.

Evan Miller

Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations is a research paper arguing that evaluations (see the Trust and Safety Evaluation Initiative for more details) should use proper statistical analysis of their results. It is discussed in Statistical Tests.

James Thomas

James Thomas is a QA engineer who posted a link to a blog post How do I Test AI? that lists criteria to consider when testing AI-enabled systems. While the post doesn’t provide a lot of details behind the list items, the list is excellent for stimulating further investigation.

John Snow Labs and Pacific.ai

John Snow Labs has created langtest, a test generation and execution framework with “60+ test types for comparing LLM & NLP models on accuracy, bias, fairness, robustness & more.”

The affiliated company Pacific.ai offers a commercial testing system with similar features.

Merriam-Webster Dictionary

The Merriam-Webster Dictionary: is quoted in our glossary for several terms.

Michael Feathers

Michael Feathers gave a talk recently called The Challenge of Understandability at Codecamp Romania, 2024, which is discussed in Abstractions Encapsulate Complexities.

MLCommons Glossary

The MLCommons AI Safety v0.5 Benchmark Proof of Concept Technical Glossary is used to inform our Glossary.

Nathan Lambert

How to approach post-training for AI applications, a tutorial presented at NeurIPS 2024 by Nathan Lambert. It is discussed in From Testing to Training. See also this Interconnects post.

NIST Risk Management Framework

The U.S. National Institute of Science and Technology’s (NIST) Artificial Intelligence Risk Management Framework (AI RMF 1.0) is used to inform our Glossary.

OpenAI

An OpenAI paper on reinforcement fine tuning is discussed in From Testing to Training.

Wikipedia

Several Wikipedia articles are used as references in our Glossary and other places.