Why AI vendor due diligence is different
Traditional IT vendor assessments focus on infrastructure security, data protection, and operational reliability. These remain necessary but insufficient for AI vendors. AI systems introduce fundamentally new risk categories that standard frameworks don’t address:
Model Opacity (Black Box Risk)
Some models, particularly large neural networks, are not intrinsically interpretable. Ask what explanation is available for the relevant output, whether it has been validated for fidelity and stability, and what system-level documentation, human-review path, and limitations accompany it. A post-hoc explanation should not be presented as the model’s internal reasoning.
Hallucinations & Confabulation
Generative models can produce plausible but unsupported or false outputs. Ask for task-specific evaluation data, source-grounding behavior, abstention and escalation design, monitoring, known limitations, and evidence from the actual model and configuration you intend to use.
Training Data Provenance
Training data can introduce bias, errors, privacy concerns, and licensing questions. Vendors may limit disclosure for security, privacy, or commercial reasons, so determine what provenance, rights, composition, filtering, and evaluation evidence is necessary for your use case and what residual uncertainty remains.
Model Drift & Degradation
Model performance can change when data, users, dependencies, prompts, or the operating environment shift. Ask what the vendor monitors, which thresholds trigger review, how model or provider changes are controlled, and what performance commitments persist after deployment.
AI-Specific Attack Vectors
Prompt injection, data poisoning, model extraction, and adversarial examples may fall outside a conventional infrastructure assessment. Match testing to the AI components and reachable actions in scope. In 2024, the Texas Attorney General alleged that Pieces Technologies used inaccurate or misleading hallucination-rate metrics; the settlement required more accurate disclosure of product accuracy and appropriate reliance.[6]
The AI vendor risk framework
Effective AI vendor assessment requires expanding traditional third-party risk management to include AI-specific dimensions. We recommend evaluating vendors across eight core risk categories:
AI Vendor Risk Categories
| Risk Category | Key Questions | Evidence Required |
|---|---|---|
| Security & Infrastructure | SOC 2? Pen testing? Data encryption? | Certifications, audit reports, test results |
| Model Transparency | Can they explain decisions? Model architecture documented? | Model cards, technical documentation, interpretability demos |
| Data practices | Training data sources? Privacy controls? Retention policies? | Data lineage documentation, DPIAs, data flow diagrams |
| Bias & Fairness | Bias testing performed? Fairness metrics tracked? | Fairness assessments, demographic parity analysis, mitigation plans |
| Regulatory compliance | EU AI Act ready? GDPR compliant? Industry regulations? | Compliance roadmaps, legal opinions, certification status |
| Business Continuity | SLAs? Model versioning? Exit strategy? | Service agreements, disaster recovery plans, data portability specs |
| Contractual Protections | Liability caps? IP ownership? Audit rights? | MSA, DPA, indemnification terms, insurance coverage |
| Ongoing Governance | Performance monitoring? Incident response? Update cadence? | Monitoring dashboards, incident logs, change management process |
Security assessment
Start with security foundations, then add questions matched to the AI system and workflow. A certificate can be useful evidence when its scope is relevant, but no single certification is universally required or sufficient.
Standard Security Requirements
- SOC 2 Type II: If your assurance criteria call for it, inspect the actual report period, system boundary, exceptions, subservice organizations, and complementary user-entity controls. Do not infer AI-system safety from a “SOC 2 compliant” label.
- ISO 27001: International security standard. Indicates mature information security management. Check certificate scope covers AI operations.
- Penetration testing: Set scope and cadence from exposure and risk. Request evidence that testing covered relevant AI components and APIs, not only the surrounding web application.
- Data encryption: Document encryption in transit and at rest, key ownership and rotation, termination points, exceptions, and which stores contain customer data, model artifacts, and inference logs.
AI-Specific Security Requirements
Critical Gap: AI Attack Vector Testing
Standard penetration testing focuses on infrastructure vulnerabilities. AI systems require specialized testing for:
- Prompt Injection: Can attackers manipulate outputs through crafted inputs?
- Data Poisoning: Can training data be corrupted to degrade model performance?
- Model Extraction: Can attackers reconstruct proprietary models through API queries?
- Adversarial Examples: Can attackers craft inputs that fool the model?
Request evidence of AI-specific security testing. The OWASP Top 10 for LLM Applications provides a baseline for evaluating LLM-based vendors.[7]
Model transparency and explainability
Model transparency determines whether you can understand, validate, and defend the vendor’s AI decisions. This is critical for regulatory compliance and liability management.
Documentation Requirements
The vendor should provide comprehensive technical documentation covering:
Model Cards
Standardized documentation (proposed by Google) covering model architecture, intended use, performance characteristics, limitations, and ethical considerations. Model cards should specify:
- Model type and architecture
- Training data characteristics
- Performance metrics by demographic group
- Known limitations and failure modes
- Intended use cases and out-of-scope applications
Datasheets for Datasets
Documentation of training data including collection methodology, composition, preprocessing, and known biases. Datasheets should cover:
- Data source and collection process
- Demographic composition
- Labeling methodology and annotator instructions
- Data cleaning and preprocessing steps
- Legal basis for data use
Explainability & Interpretability
For consequential applications, determine what explanation, notice, review, and contestability duties apply to the actual decision, system role, and jurisdiction. Where an explanation is needed, evaluate whether the proposed method is faithful and usable for that purpose; no single technique universally explains a model’s internal reasoning. Candidate approaches include:
- Feature-attribution methods: SHAP, LIME, and related methods can estimate associations under stated assumptions; they do not prove causal influence or reveal internal model reasoning. Attention weights should not be treated as an explanation without method-specific validation.
- Example-based evidence: Similar or contrastive examples may aid review, but similarity does not establish causality or explain every component of a decision.
- Counterfactual explanations: Test what specified changes are associated with a different output under a named model and configuration; validate feasibility and stability.
- Natural-language rationales: Model-generated text may help communicate a result, but it is an output to validate and is not proof of the model’s internal reasoning.
Test this in practice. Ask the vendor to demonstrate the proposed explanation or evidence method on representative cases and disclose its intended use, assumptions, failure modes, and validation. A generic “explainable AI” label is not enough.
Data practices
Data practices determine privacy risk, regulatory exposure, and model quality. Inadequate data governance creates liability you may not discover until litigation or regulatory investigation.
Training Data Assessment
Critical Questions on Training Data
1. What data sources were used for training?
Public datasets? Licensed commercial data? Customer data? Scraped web content? Each carries different legal and quality implications.
2. What is the legal basis for using this data?
Under GDPR, AI training requires valid legal basis, typically consent or legitimate interest. Scraped data without consent creates regulatory risk.
3. Does training data contain personal information?
If yes, how was it de-identified? Can models leak training data through memorization? (Demonstrated risk with LLMs.)
4. What is the demographic composition of training data?
Underrepresentation of demographic groups leads to biased models. Vendor should provide demographic breakdowns.
5. How old is the training data?
Stale training data produces models that don’t reflect current patterns. Request data recency and refresh schedules.
Customer Data Handling
Understand exactly how the vendor uses your data:
- Training and improvement: Determine separately whether prompts, outputs, uploaded files, feedback, fine-tuning data, or other customer data may be used for training, improvement, evaluation, or abuse monitoring. Controls can differ by product, endpoint, account setting, and contract; document consent requirements and any exceptions rather than treating a “zero data retention” label as a no-training commitment.
- Data Residency: Where is data processed and stored? EU GDPR and China’s Personal Information Protection Law impose restrictions on cross-border data transfers.
- Retention Policies: Identify retention separately for prompts, outputs, files, inference logs, feedback, fine-tuning data, safety or abuse-monitoring records, and backups. Verify the product and endpoint configuration, contractual term, deletion behavior, and documented legal, security, or operational exceptions; do not infer retention from a separate training-use control.
- Deletion Rights: Can you request deletion of your data? How quickly is deletion executed? Is deletion cryptographically verifiable?
Bias and fairness testing
The roughly $2.2M SafeRent settlement resolved allegations concerning discriminatory tenant-screening outcomes and illustrates potential litigation and regulatory exposure; a settlement does not establish the merits of the allegations. Due diligence should examine the vendor’s testing, outcomes, controls, and evidence for the buyer’s intended use.[8]
Bias-assessment evidence
Request evidence of fairness testing across protected categories relevant to your use case:
Common Fairness Metrics
| Fairness Metric | Definition | When to Use |
|---|---|---|
| Demographic Parity | Equal positive prediction rates across groups | When groups should receive outcomes at equal rates (e.g., ad delivery) |
| Equal Opportunity | Equal true positive rates across groups | When cost of false negatives varies by group (e.g., disease screening) |
| Equalized Odds | Equal true positive and false positive rates | When both error types matter (e.g., credit decisions) |
| Calibration | Predicted probabilities match true outcome rates | When probability scores are used directly (e.g., risk scoring) |
Critical point: Different fairness definitions can be mathematically incompatible. The vendor must explain which fairness criteria they optimize for and why that choice is appropriate for your use case.[9]
Bias Mitigation Documentation
Testing for bias is insufficient. Vendors must demonstrate mitigation. Request documentation of:
- Pre-processing techniques: Data augmentation, reweighting, or resampling to balance training data
- In-processing techniques: Fairness constraints during model training
- Post-processing techniques: Threshold adjustment to equalize outcomes across groups
- Ongoing monitoring: Production dashboards tracking fairness metrics by demographic group
Regulatory compliance
Vendor and deployer obligations depend on the applicable law, system classification, role, territory, and effective date. Under the EU AI Act, supply-chain duties differ among providers, deployers, importers, distributors, and other actors.
EU AI Act Vendor Requirements
Article 26 places specified obligations on deployers of covered high-risk AI systems. A vendor review should establish the parties’ roles and identify the documentation and conformity artifacts relevant to the system; the list below is a review aid, not a statement that every item is a universal Article 26 verification duty.[10]
Potential high-risk-system review artifacts
- Vendor has conducted conformity assessment (self-assessment or notified body)
- CE marking is affixed and accompanied by EU declaration of conformity
- Technical documentation is available (Article 11)
- System has undergone appropriate risk management procedures (Article 9)
- Data governance meets Article 10 requirements (training/validation/testing data)
- Human oversight provisions are documented (Article 14)
- Accuracy, robustness, and cybersecurity requirements are met (Article 15)
Non-compliance penalties: Article 99 sets different ceilings by violation. Listed operator and transparency obligations are generally subject to up to €15M or 3% of worldwide annual turnover; the €35M or 7% tier applies to prohibited practices, and €7.5M or 1% applies to incorrect, incomplete, or misleading information supplied in response to an authority request. The AI Omnibus entered into force on 27 July 2026, and relevant Annex III high-risk obligations apply from 2 December 2027.[10]
Other Regulatory Frameworks
US State Laws
Colorado (SB 26-189, “Automated Decision-Making Technology”): Signed May 14 2026, this repealed and replaced the 2024 Colorado AI Act before it took effect. The “high-risk AI system” reasonable-care duty, impact assessments and the NIST/ISO safe harbor are gone; substantive obligations commence January 1 2027 (not yet enforceable). It now turns on disclosure and transparency for covered automated decision-making technology (ADMT) used to materially influence a consequential decision. Seek vendor documentation of intended uses, known limitations and how decisions can be reviewed.
California ADMT (January 2027): The CPPA regulations create risk-assessment, pre-use notice, access, and opt-out duties for specified ADMT uses, subject to the regulation’s scope, exceptions, and phased compliance dates. Review the intended use against the current rule rather than assuming every automated decision carries every duty.
Sector-Specific Regulations
Healthcare (FDA, HIPAA): FDA requirements depend on whether the software is a device, its classification, and the applicable 510(k), De Novo, PMA, or exempt pathway. HIPAA obligations depend on the parties’ roles and the PHI use; a BAA is required when the vendor is a business associate, and training or another secondary use must have an applicable permission or authorization.
Financial Services (SR 11-7, ECOA): Banking organizations within the scope of SR 11-7 should assess third-party models under their model-risk framework. ECOA and Regulation B can require adverse-action notices with specific principal reasons for credit denials, including when complex models are used.
Business continuity and operational risk
AI vendor failures create unique business continuity challenges. Model performance can degrade silently, vendors can deprecate API versions without notice, and model weights cannot be easily migrated to alternative providers.
Service Level Agreements
Negotiate specific SLAs covering AI-relevant metrics:
- Uptime guarantees: Set an availability target from the workflow’s tolerable downtime and fallback design. Ensure the measured service boundary covers the inference API or other production dependency you rely on.
- Latency commitments: P95 and P99 latency targets for inference requests. Critical for real-time applications.
- Model performance baselines: Minimum accuracy, precision, recall thresholds. What happens if production performance falls below these levels?
- Version stability: Negotiate notice and migration support appropriate to the workflow’s criticality, switching cost, and release model.
Model Versioning & Change Management
AI models change over time through retraining, fine-tuning, and updates. Uncontrolled model changes can break production systems or introduce new biases.
- Version pinning: Determine whether versions can be pinned or changes otherwise controlled. The risk from automatic updates depends on the workflow, validation design, and rollback options.
- Change notification: Negotiate notice, content, and emergency-change exceptions appropriate to the system, including expected performance and subgroup effects where relevant.
- Rollback capability: Ability to revert to previous model versions if new versions underperform.
Exit Strategy & Data Portability
AI switching costs can be material when a workflow depends on vendor-specific models, fine-tuning, prompts, evaluations, integrations, or data formats. Assess portability for the actual deployment rather than assuming either easy migration or unavoidable lock-in.
Negotiate exit protections:
- Data export: Negotiate the exact data, formats, delivery window, costs, retention exceptions, and deletion evidence required at termination; do not assume the vendor’s default covers inference logs, fine-tuning data, or feedback data.
- Model rights (if custom): For vendor fine-tuning or custom training, define ownership, licenses, access, deletion, and transition rights; transfer of weights may be unavailable or inappropriate.
- Transition assistance: Negotiate an appropriate transition period, continued access, data delivery, support, and wind-down terms for the workflow.
Contractual considerations
Standard vendor terms may not address the system’s AI-specific risks. Review liability, intellectual property, evidence access, audit rights, data use, change control, and indemnification in light of the actual workflow and bargaining position.
Liability & Risk Allocation
Key Contractual Provisions
1. Performance Warranties
Consider measurable commitments, evaluation protocols, remedies, and exclusions appropriate to the workflow. Thresholds should be validated for intended use and negotiated in the agreement.
2. Liability Caps & Carve-Outs
Liability caps and carve-outs vary by vendor, leverage, risk, insurance, and agreement. Negotiate treatment of gross negligence, data breaches, intellectual-property claims, regulatory exposure, and other material risks with counsel.
3. Indemnification
Assess whether and how indemnities should address intellectual-property claims, vendor-caused regulatory exposure, data incidents, or undisclosed model limitations. Scope, exclusions, procedure, and enforceability are agreement-specific.
4. Audit Rights
Negotiate evidence access, audit rights, third-party assurance, confidentiality protections, and remediation where appropriate. EU AI Act Article 26 assigns system- and role-specific deployer duties; it does not create a general contractual right to audit every vendor.
5. Intellectual Property Ownership
Clarify ownership of: your input data, vendor’s base models, fine-tuned models trained on your data, outputs generated by the system. Default SaaS contracts often grant vendor broad rights to your data. Modify this for AI systems.
Data Processing Agreements
Determine the parties’ roles, data flows, and applicable law, then review whether the contract, DPA, BAA, or other terms need AI-specific provisions. Relevant topics may include:
- Purpose limitation: Define permitted training, improvement, evaluation, and benchmarking uses, legal bases, instructions, and approval requirements appropriate to the relationship.
- Automated decision-making: Where applicable, allocate responsibilities for information, human intervention, contestability, and other rights under the relevant regime.
- Individual rights: Define the support needed for applicable access, correction, deletion, objection, information, and review rights without implying that every regime creates every listed right.
Ongoing monitoring and governance
Depending on the workflow and obligations, post-contract monitoring may be warranted because data, usage, dependencies, model versions, and risks can change after deployment. Set the cadence and measures from the actual risk and contract rather than assuming every system degrades continuously.
Performance Monitoring Requirements
Negotiate access to monitoring dashboards tracking:
Operational Metrics
- API uptime and latency (P50, P95, P99)
- Request success/failure rates
- Rate limit consumption
- Error codes and frequencies
Model Performance Metrics
- Accuracy, precision, recall over time
- Confidence score distributions
- Prediction distribution drift
- Model performance by demographic group
Incident Response & Escalation
Define clear incident response procedures for AI-specific failures:
- Incident classification: What constitutes a P0/P1/P2 incident for AI systems? Include: accuracy drops below SLA thresholds, bias metric violations, hallucination rates exceeding baselines, security breaches affecting model integrity.
- Response SLAs: Time to acknowledge, time to mitigation, time to resolution by incident severity.
- Post-incident reporting: Root-cause analysis where feasible, corrective actions, retesting, residual-risk analysis, and evidence intended to reduce recurrence; no process can guarantee that a similar incident will not recur.
Periodic and Event-Driven Reviews
Set review cadence and triggers from the workflow’s consequence, vendor access, change rate, incidents, contract, and applicable duties. Topics can include:
- Performance against SLAs and model accuracy baselines
- Bias metrics and fairness assessment results
- Security incidents and remediation status
- Regulatory landscape changes and vendor compliance roadmap
- Model updates, deprecations, and roadmap
Illustrative due diligence checklist
Use these prompts as a starting point, then tailor them to the system, role, data, harms, deployment, contract, and applicable law. They are not universal minimum controls or legal requirements. The example priorities are planning aids, not a risk classification or assurance conclusion.
Evidence-Based Vendor Assessment
Vendor questionnaires often rely on self-attestation. For configured in-scope events, GLACIS can preserve signed records of what vendor controls reported, adding operational evidence without claiming complete coverage or effectiveness.
From Questionnaires to Evidence
Instead of asking only “Do you perform bias testing?”, request scoped records identifying the model version, configured test, timestamp, and reported result. Selected signed records can be assembled manually into a scoped review packet whose supported integrity and provenance properties are independently checkable; test quality, effectiveness, and coverage require separate assessment.
Supply Chain Transparency
EU AI Act Article 26 assigns deployers specific duties. Vendor documentation and signed operational records may inform due diligence, but neither independently establishes vendor compliance or satisfies every deployer obligation. Legal scope and evidence sufficiency require separate review.
Request from vendors: Ask for scoped operational records in addition to policy and compliance documents. Independently checkable signatures can support integrity and provenance, while due-diligence teams still assess effectiveness, coverage, safety, and compliance.
Security & Infrastructure: 10 illustrative prompts
| # | Illustrative review prompt | Evidence Type | Example priority |
|---|---|---|---|
| 1 | Current independent assurance whose scope, period, criteria, and exceptions address the relevant service and risks | Audit report | Critical |
| 2 | Relevant information-security certification or equivalent control evidence, with scope and validity checked | Certificate | High |
| 3 | Security and adversarial testing with cadence and scope based on exposure, changes, threat model, and obligations | Test summary | Critical |
| 4 | Risk-appropriate protection for stored training data and model artifacts, including disclosed cryptographic design where used | Technical spec | Critical |
| 5 | Risk-appropriate transmission protection for relevant data flows, with supported protocols and exceptions disclosed | Technical spec | Critical |
| 6 | Strong authentication and access controls appropriate to privileged and administrative paths | Policy doc | High |
| 7 | Vulnerability management program with defined SLAs for patching | Process doc | High |
| 8 | Incident response plan including AI-specific incident types | IR plan | High |
| 9 | Business continuity and disaster recovery plan with tested procedures | BC/DR plan | High |
| 10 | Security awareness training for personnel with access to models/data | Training records | Medium |
Model Transparency & Explainability: 8 illustrative prompts
| # | Illustrative review prompt | Evidence Type | Example priority |
|---|---|---|---|
| 11 | Model card documenting architecture, performance, limitations | Model card | Critical |
| 12 | Technical documentation covering model development lifecycle | Technical docs | High |
| 13 | Explainability methods for individual predictions (SHAP, LIME, etc.) | Demo + docs | Critical |
| 14 | Performance benchmarks on standard datasets relevant to use case | Benchmark results | High |
| 15 | Known limitations and failure modes documented | Limitation docs | High |
| 16 | Out-of-scope use cases explicitly identified | Use case doc | High |
| 17 | Model versioning strategy with semantic versioning | Versioning policy | Medium |
| 18 | Change log documenting model updates and performance impacts | Change log | Medium |
Data Practices: 9 illustrative prompts
| # | Illustrative review prompt | Evidence Type | Example priority |
|---|---|---|---|
| 19 | Datasheet for training datasets documenting composition and provenance | Datasheet | Critical |
| 20 | Legal basis for using training data (licenses, consent, etc.) | Legal analysis | Critical |
| 21 | Customer data usage policy (training prohibition unless consented) | Policy + contract | Critical |
| 22 | Data residency and cross-border transfer documentation | Data flow diagram | High |
| 23 | Data retention policy with defined retention periods | Retention policy | High |
| 24 | Data deletion procedures supporting right to erasure (GDPR Article 17) | Deletion process | High |
| 25 | Data Processing Agreement (DPA) covering AI-specific processing | DPA | Critical |
| 26 | De-identification methods for personal data in training sets | Technical docs | High |
| 27 | Data access controls limiting personnel access to training data | Access policy | High |
Bias & Fairness: 7 illustrative prompts
| # | Illustrative review prompt | Evidence Type | Example priority |
|---|---|---|---|
| 28 | Bias assessment across protected categories (race, gender, age, etc.) | Bias report | Critical |
| 29 | Fairness metrics tracked (demographic parity, equal opportunity, etc.) | Fairness metrics | Critical |
| 30 | Bias mitigation techniques documented and implemented | Mitigation docs | High |
| 31 | Demographic composition of training data documented | Data composition | High |
| 32 | Ongoing fairness monitoring in production deployments | Monitoring dashboard | High |
| 33 | Disparate impact testing for high-stakes decisions | Impact analysis | Critical |
| 34 | Third-party fairness audit (for high-risk use cases) | Audit report | Medium |
Regulatory Alignment: 8 illustrative prompts
| # | Illustrative review prompt | Evidence Type | Example priority |
|---|---|---|---|
| 35 | EU AI Act risk classification and compliance roadmap (if applicable) | Classification + roadmap | Critical |
| 36 | Technical documentation per EU AI Act Article 11 (for high-risk) | Technical docs | Critical |
| 37 | Quality management system documentation (EU AI Act Article 17) | QMS docs | High |
| 38 | GDPR compliance documentation (DPIAs for automated decisions) | DPIA | Critical |
| 39 | Sector-specific compliance (HIPAA, FDA, FCA, etc. as applicable) | Certifications | High |
| 40 | ISO 42001 certification or roadmap (AI management system) | Certificate | Medium |
| 41 | NIST AI RMF alignment documentation | Framework mapping | Medium |
| 42 | Regulatory monitoring process for emerging AI regulations | Process doc | Medium |
Business Continuity & Operations: 8 illustrative prompts
| # | Illustrative review prompt | Evidence Type | Example priority |
|---|---|---|---|
| 43 | SLA availability target and measurement boundary matched to the workflow | SLA document | Critical |
| 44 | Latency commitments (P95, P99) appropriate for use case | SLA document | High |
| 45 | Model performance baseline guarantees (accuracy, precision, recall) | Performance SLA | Critical |
| 46 | Version pinning capability with controlled upgrade path | Technical spec | High |
| 47 | Change and deprecation notice matched to switching time, fallback design, emergency changes, and contractual risk | Contract term | High |
| 48 | Data export and portability specifications | Export specs | High |
| 49 | Transition assistance and continued access for a negotiated period appropriate to the exit plan | Contract term | Medium |
| 50 | Financial stability assessment (funding, runway, profitability) | Financial disclosure | Medium |
Contractual Protections: 6 illustrative prompts
| # | Illustrative review prompt | Evidence Type | Example priority |
|---|---|---|---|
| 51 | Performance warranties with specific accuracy/bias thresholds | Contract term | Critical |
| 52 | Liability caps appropriate for risk level (or uncapped for critical risks) | Contract term | Critical |
| 53 | Indemnification for IP infringement, regulatory penalties, discrimination | Contract term | Critical |
| 54 | Evidence access and audit rights with scope, cadence, triggers, confidentiality, and remediation negotiated for the relationship | Contract term | High |
| 55 | Clear IP ownership terms (inputs, outputs, fine-tuned models) | Contract term | High |
| 56 | Insurance coverage (E&O, cyber) with AI-specific coverage confirmed | COI + policy | Medium |
Frequently asked questions
How do I prioritize vendors when I have limited assessment resources?
Use a risk-based approach grounded in the workflow’s consequences, affected people, data, autonomy, vendor access, legal classification, and existing controls. Select the review prompts and evidence depth that fit that analysis rather than applying a fixed 56-item minimum or universal high/medium/low taxonomy.
What if the vendor refuses to provide technical documentation?
Treat unavailable documentation as a risk and evidence gap. Decide whether alternative evidence, contractual protections, narrower scope, additional controls, or another vendor can address it. Legal classification does not change merely because human oversight is added, and some proprietary material may require controlled review rather than public disclosure.
Should I require third-party audits of AI vendors?
Consider independent assurance where its scope, criteria, period, and methodology address the risks you need reviewed. A SOC 2 report or bias audit supports only the claims within its stated scope and does not validate every vendor assertion. Audit type, cadence, review depth, and timing should follow the workflow, changes, contract, and applicable duties.
How often should I reassess AI vendors?
Set the reassessment cadence from the workflow’s consequence, vendor access, rate of model or product change, contractual duties, and applicable law. Reassess after material model or architecture changes, disclosed incidents, changes in legal requirements, or performance outside agreed thresholds.
References
- [6] Texas Attorney General. “Attorney General Ken Paxton Reaches Settlement in First-of-its-Kind Healthcare Generative AI Investigation.” September 18, 2024. Settlement filing.
- [7] OWASP Foundation. “OWASP Top 10 for Large Language Model Applications.” owasp.org
- [8] SafeRent Solutions settlement. November 2024. Roughly $2.2M settlement resolving tenant-screening discrimination allegations; the settlement is not a merits determination.
- [9] Kleinberg, Jon, et al. “Inherent Trade-Offs in the Fair Determination of Risk Scores.” ITCS 2017. Demonstrates mathematical impossibility of satisfying all fairness definitions simultaneously.
- [10] European Union. “Regulation (EU) 2024/1689 - Artificial Intelligence Act.” Official Journal of the EU, July 2024. eur-lex.europa.eu
- [11] Colorado General Assembly. “Automated Decision-Making Technology” (SB 26-189), signed May 14, 2026, repealing and replacing the 2024 Colorado AI Act (SB 24-205); substantive compliance January 1, 2027. leg.colorado.gov
- [13] U.S. Food and Drug Administration. “Artificial Intelligence and Machine Learning in Software as a Medical Device.” fda.gov
- [14] Federal Reserve. “SR 11-7: Guidance on Model Risk Management.” April 2011. Applies to third-party models used by financial institutions.
- [15] Equal Credit Opportunity Act (ECOA), 15 U.S.C. § 1691. Requires specific reasons for adverse credit decisions.
- [16] NIST. “AI Risk Management Framework (AI RMF 1.0).” January 2023. nist.gov
- [17] ISO/IEC. “ISO/IEC 42001:2023 - Information technology — Artificial intelligence — Management system.” iso.org
- [18] Mitchell, Margaret, et al. “Model Cards for Model Reporting.” FAT* 2019. Proposes standardized documentation for machine learning models.
- [19] Gebru, Timnit, et al. “Datasheets for Datasets.” Communications of the ACM, 2021. Framework for documenting datasets used in ML.
- [20] Lundberg, Scott M., and Su-In Lee. “A Unified Approach to Interpreting Model Predictions.” NIPS 2017. SHAP (SHapley Additive exPlanations) framework.