QuantPi is tasked to develop the first automated risk management platform to enable safety, fairness, explainability, and continuous monitoring of generative AI systems.
The Challenge:
Artificial Intelligence is the most profound transformation humanity has ever seen. In the era of AI-first organizations, all processes, and applications of companies will be driven by AI. With the rise of generative AI, intelligent systems not only determine the formats of human interaction but also its content. At the same time, these extremely powerful generative AI systems can be dangerous black boxes. The number of dramatic incidents has increased and can explode, putting the existence of entire organizations at risk.
Controlling AI black boxes with today’s testing tools is difficult. Existing testing approaches are highly complex, computationally expensive, and can only be partially applied to generative AI systems. Full risk assessments with quantifiable evidence of risk likelihood are expensive and time-consuming.
QuantPi’s Contribution:
In this project funded by the European Innovation Council, QuantPi is developing a software platform that automatically conducts technical risk assessments within the most important risk dimensions of generative AI systems. The goal of the project is to be able to detect unintended system behavior with a minimum number of model queries and provide quantitative assessments on concrete testing metrics. This information is intended to be mapped to a monetary business impact, and translated into technical documentation, as well as certification and audit-readiness reports.
For the first time, the software to be developed in the project enables companies developing or operating generative AI systems to automatically identify, assess, and mitigate major risks across dimensions such as performance, fairness, robustness, explainability, and more.
More Information:
This project is funded by the European Union. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Innovation Council and SMEs Executive Agency (EISMEA). Neither the European Union nor the granting authority can be held responsible for them.