AI Vendor Due Diligence Checklist

10 Questions for Healthcare AI Procurement

Instructions: Use this checklist to supplement standard security questionnaires (HECVAT, SIG, HITRUST) when evaluating AI vendors. These questions address inference-level evidence capabilities that standard questionnaires do not cover.

For any specific patient interaction, can you provide a tamper-evident trace showing which guardrails executed, with timestamps and pass/fail status?

Testing: Guardrail execution trace — proves controls ran, not just that they exist

Can you reconstruct the complete input context the model processed for any given output—including prompts, retrieved data, and applied redactions?

Testing: Decision rationale — enables root cause analysis when outputs are unexpected

Is your compliance evidence cryptographically signed and independently verifiable without access to your internal dashboards?

Testing: Independent verifiability — evidence third parties can validate

Can you prove that protected health information never left our infrastructure during AI inference?

Testing: Zero-egress architecture — reduces BAA scope and data residency risk

How do you demonstrate model version control—proving which exact code processed each request?

Testing: Configuration traceability — links incidents to specific model versions

What is your documented hallucination rate, and can you provide statistical confidence intervals based on production data?

Testing: Performance transparency — quantified risk, not marketing claims

How does your evidence map to specific control objectives in ISO 42001, NIST AI RMF, and EU AI Act Article 12?

Testing: Framework anchoring — accelerates audit and compliance assessment

What per-inference artifacts do you retain, for how long, and in what format are they available for audit?

Testing: Evidence retention — California ADMT requires 5+ years

If a patient files a complaint about AI-generated content, what evidence can you provide within 24 hours?

Testing: Incident response capability — operational readiness for investigations

Do your logs and attestations meet the evidentiary standards that would be required in regulatory proceedings or litigation?

Testing: Legal defensibility — admissibility in adversarial contexts

Notes

By Jennifer Shannon, MD | Chief Medical Officer, GLACIS