Organizations that develop or deploy Generative AI solutions in healthcare are subject to more than 70 national and state laws, regulatory rules, and industry standards. Once an organization establishes an AI Governance framework, its policies will include dozens of controls that must be implemented for each AI project. This session describes a subset of these controls that can be automated with current tools:

• Automated execution of medical LLM benchmarks during system testing and when monitoring in production, including coverage of medical ethics, medical errors, fairness and equity, safety and reliability – using Pacific AI
• Automating generation and executing of LLM test suites for custom solutions, including testing for robustness, bias, fairness, representation, and accuracy – using LangTest
• Automated generation of model cards, complying with transparency laws and including explained benchmark results – based on the CHAI draft model card standard