How should a business evaluate an AI vendor?
Evaluate the vendor in the context of the exact workflow, data, users, actions, and consequences you plan to deploy. Require evidence for performance, control, operations, economics, and exit.
An AI vendor can have strong infrastructure controls and still be wrong for your use case. It can pass a polished benchmark while failing on your documents, vocabulary, permissions, or high-impact edge cases.
Strategic Brief
Due diligence is a decision about residual business risk. Certifications and questionnaires are inputs; representative testing, architecture, contract rights, and operating evidence complete the decision.
What should happen before the questionnaire?
Write a use-case brief:
- intended users and affected parties;
- the business decision or action;
- data sources and sensitivity;
- whether the system reads, recommends, drafts, or acts;
- consequence and reversibility of error;
- required quality, latency, availability, and geography;
- applicable customer, legal, sector, and retention requirements;
- expected volume and budget;
- accountable business owner.
Without this context, reviewers cannot distinguish a relevant control from a generic claim.
Set the diligence depth
Risk follows the deployed use, not the vendor’s marketing category.
Use a streamlined review with firm data boundaries
Human review limits consequence, but confidential data, retention, and unsupported claims still require control.
Test representative drafts, administration, data handling, deletion, and user guidance.
Employees may expand a drafting tool into unapproved decisions or sensitive uses.
Which eight diligence domains matter?
1. Task performance
Test on your representative data. Include ordinary cases, difficult cases, missing information, policy conflicts, adversarial inputs, and protected cohorts where relevant. Measure task success, supported claims, consistency, abstention, and reviewer effort.
2. Data lifecycle
Map collection, transmission, storage, retrieval, logging, support access, retention, backups, deletion, analytics, and model improvement. Identify every subprocessor, region, and purpose. Clarify who owns prompts, outputs, feedback, embeddings, configurations, and derived data.
3. Security
Review identity, single sign-on, role mapping, encryption, tenant isolation, secrets, vulnerability management, testing, secure development, business continuity, and incident notification. For agents, inspect each tool permission and policy boundary.
4. Model supply chain
Identify model providers, versions, hosting, fine-tuning, safety layers, open-source components, and upstream dependencies. Determine how changes are announced, evaluated, rolled back, and reflected in service commitments.
5. Responsible AI and governance
Ask how the vendor identifies intended use, limitations, misuse, affected people, fairness, transparency, human oversight, and complaints. Translate its process into evidence for your deployment.
6. Operations
Verify service levels, rate limits, support, monitoring, status communication, recovery, observability export, and degradation behavior. Test failure, not just success.
7. Economics
Model licenses, consumption, premium models, connectors, storage, support, implementation, review, and exit. Require usable cost and usage exports.
8. Contract and exit
Address data use, confidentiality, audit evidence, subprocessor changes, model changes, security notification, indemnity, limitations, service levels, termination, export, transition, and deletion.
Score the vendor evidence, not the sales response
What evidence should you request?
Request only evidence connected to risk:
- architecture and data-flow diagrams;
- security and privacy documentation;
- current independent audit or certification reports and relevant scope;
- penetration-test summary and remediation process;
- model and subprocessor inventory;
- evaluation methods, limitations, and release criteria;
- service history, continuity plans, and recovery objectives;
- incident notification and response procedures;
- access, usage, audit, and cost export examples;
- retention, deletion, and backup behavior;
- customer references with comparable use;
- sample export and termination procedure.
A refusal may be reasonable for sensitive reports. The vendor should still provide a secure review path, scoped attestation, or alternative evidence.
Investigate the highest-impact vendor risks
A provider update changes quality, refusal, latency, or tool behavior in a critical workflow.
Production metrics move without an application or data release.
Require material-change notice, version options where available, external regression tests, canary release, and rollback.
How should the proof of value be run?
- Agree on the workflow, cohort, data, duration, and support conditions.
- Lock the success, risk, latency, adoption, and cost thresholds.
- Use a held-out case set and realistic permissions.
- Include red-team and failure cases appropriate to risk.
- Capture user edits, rejection, escalation, and downstream outcomes.
- Price expected production volume and a downside scenario.
- Reconcile technical findings with contract commitments.
- Record the decision, limitations, conditions, and residual risk owner.
Do not let the vendor perform all scoring. Domain experts and your product team should own the answer key and outcome interpretation.
What belongs in the final decision memo?
The approval should state:
- selected use and prohibited extensions;
- evidence reviewed and unresolved gaps;
- performance thresholds and operating limits;
- required configuration and controls;
- accountable internal owners;
- contract conditions;
- monitoring and review cadence;
- material-change triggers;
- incident and stop authority;
- exit and replacement plan;
- residual risk and the person accepting it.
Repeat diligence when the use, data, autonomy, model supply chain, subprocessor, or consequence changes materially. Vendor approval is not a permanent passport for every future use.