AI Procurement: What to Demand in a Vendor's Governance Documentation

When an organisation procures traditional software, the governance due diligence is well-established: security review, data processing agreements, SLA terms, exit provisions. The checklist is mature and the risks are understood.

AI procurement is different. The systems are not deterministic — their outputs depend on training data, model architecture, deployment context, and ongoing updates that the purchasing organisation does not control. The governance questions are more complex, and the consequences of inadequate due diligence are higher.

A growing number of enterprise organisations are discovering this after procurement — when an AI system produces a discriminatory output, creates a compliance gap, or proves impossible to audit. The time to address governance in AI procurement is before the contract is signed.

What to Demand From AI Vendors

1. Model documentation. Can the vendor provide technical documentation covering training data sources, performance benchmarks across different populations, known limitations, and intended use cases? Absence means the organisation is deploying a system it does not understand.

2. Data provenance and consent. What data was used to train the model? Is it processed in accordance with applicable data protection law, including India's DPDP Act and the EU's GDPR? Are there copyright claims on training data that could create downstream liability?

3. EU AI Act classification. Has the vendor assessed the system's risk classification under the EU AI Act? If the system falls into a high-risk category, what compliance documentation exists? Has a conformity assessment been completed?

4. Performance disparities. Does the system perform consistently across different demographic groups, languages, and input types? Performance disparity data — showing how accuracy varies across populations — should be a standard deliverable for any AI used in consequential decisions.

5. Logging and audit capability. Can the system produce logs sufficient to reconstruct its decision process after the fact? This is required for EU AI Act compliance on high-risk systems and increasingly required by internal governance standards.

6. Human override capability. Can a human review and override the system's output in normal operation? Is this usable in practice, or is it a nominal feature requiring significant technical effort to invoke?

7. Update and change management. How does the vendor handle model updates? Will an update change system behaviour in ways that affect the organisation's compliance position? What notice is provided before material changes?

8. Incident response SLAs. What is the vendor's process for reporting and remediating AI incidents — outputs that cause harm, discriminate, or produce systematic errors? What are the SLA commitments?

9. Data processing terms. What happens to the data the organisation inputs into the system? Is it used for training? Under what conditions? What are the data residency requirements?

Red Flags in Vendor Responses

Vendors who cannot answer these questions without lengthy escalation typically have not operationalised AI governance themselves. That is not necessarily disqualifying — but it is important information about the residual risk the buying organisation will need to manage independently.

Vendors who actively resist transparency questions, claim proprietary constraints prevent disclosure, or provide inconsistent answers represent a higher risk category. An AI system whose governance documentation cannot be reviewed is not an AI system that can be governed.

Contractual commitments are not a substitute for operational transparency. A vendor can commit contractually to standards they cannot demonstrate — and that commitment will not protect the organisation when a governance failure occurs.