You're Building on Someone Else's AI
Know What Their Terms Actually Say
Every AI vendor has different data retention policies, training practices, and liability terms. This kit gives you a 60-question scored assessment, a vendor comparison matrix, and pre-researched profiles for the 7 providers your team is actually evaluating.
Why AI Vendor Selection Goes Wrong
Most companies evaluate AI vendors the same way they'd evaluate any SaaS tool: demos, feature comparison, pricing. Security and data handling are an afterthought, or they're delegated to someone who doesn't have a structured way to evaluate them. By the time legal looks at the contract, the engineering team has already committed to a vendor and a timeline.
The stakes are higher than with traditional software. AI vendors often retain prompts and completions by default. Some use your data for model improvement unless you explicitly opt out - and the opt-out process varies by vendor, is often buried in the API documentation, and changes when vendors update their terms. Training on customer data can create GDPR obligations, HIPAA liability, and confidentiality risks that your security review should catch before you sign.
The 12 deal-breaker questions are the ones that matter most. They're the questions that identify vendors whose data practices create liability you can't accept - training on your data without explicit consent, breach notification timelines longer than your regulatory requirements, indemnification gaps that leave you exposed if the model produces harmful outputs. Most vendor evaluations never ask them.
The pre-researched profiles for OpenAI, Anthropic, Google, AWS Bedrock, Azure OpenAI, Cohere, and Mistral save the research that would otherwise take your team days per vendor. Data retention defaults, training policies, BAA availability, compliance certifications - already documented.
Everything You Need to Evaluate AI Vendors
Built by a CISSP/CISM-certified security professional. Not a summary of blog posts.
60-Question Scored Assessment
Six categories covering data handling, model training, security controls, compliance, incident response, and contractual terms. 12 questions flagged as deal-breakers with automatic scoring.
Vendor Comparison Matrix
Side-by-side comparison template for up to 5 vendors. Score, rank, and present findings to leadership with a clear recommendation framework.
7 Pre-Researched Provider Profiles
OpenAI, Anthropic, Google, AWS Bedrock, Azure OpenAI, Cohere, and Mistral. Data retention, training policies, compliance certifications, and contract gotchas - already documented.
DPA Requirements Checklist
45-item checklist covering everything your Data Processing Agreement needs. Subprocessor clauses, data deletion, breach notification, and cross-border transfer requirements.
12 Deal-Breaker Questions
The questions that separate acceptable vendors from liability risks. Training on your data, data retention defaults, breach notification timelines, and indemnification gaps.
Vendor Evaluation Guide
Step-by-step process for running the assessment: who to involve, how to score, when to escalate, and how to present findings to leadership and legal.
Built for Teams Choosing AI Providers
If your company is evaluating AI vendors, this kit turns weeks of research into a structured decision.
Engineering Leads
You're evaluating AI vendors for your product. Leadership wants a comparison. This kit gives you the framework to assess security, not just features.
Security Teams
Procurement dropped an AI vendor on your desk for review. The 60-question assessment and DPA checklist give you a structured evaluation instead of ad-hoc Googling.
CTOs & VPs of Engineering
You're choosing between OpenAI, Anthropic, and Google. The pre-researched profiles and comparison matrix give you a defensible decision - not a gut call.
60 questions. 7 provider profiles. One clear decision.
- 60-question scored assessment (6 categories, 12 deal-breakers)
- Vendor comparison matrix template
- 7 pre-researched AI provider profiles
- 45-item DPA requirements checklist
- Vendor evaluation guide with scoring rubric
- Deal-breaker question reference card
- Single-user commercial license
Instant download. Professional Word documents (.docx) for easy customization.
Questions
Is this just for OpenAI?
No. The 60-question assessment framework works for any AI vendor - cloud APIs, self-hosted models, or embedded AI features. The kit includes 7 pre-researched profiles (OpenAI, Anthropic, Google, AWS Bedrock, Azure OpenAI, Cohere, Mistral), but the assessment applies to any provider.
How often should we re-evaluate?
Quarterly for critical vendors (those processing customer data or PII), annually for others. AI vendor terms change frequently - the monitoring guide covers what to watch for and how to set up review triggers.
What if a vendor fails the assessment?
The kit includes a risk acceptance and mitigation path. If a vendor scores poorly on data retention but is otherwise the right technical fit, the DPA requirements checklist shows you what contractual protections to negotiate and what compensating controls to put in place. Not every failure is a disqualifier - but you should be making that decision consciously, not accidentally.
Does this cover self-hosted and open-source models?
Yes. The 60-question assessment framework applies to any AI provider - cloud API, self-hosted model, or open-source weights. For self-hosted models, the vendor risk sections shift focus from data retention policies to model supply chain security and integrity verification. The framework adapts with guidance for each scenario.
We're a small team - who should run this assessment?
The evaluation guide is written for whoever handles security at your company. At an early-stage startup that might be the CTO or a senior engineer. At a larger company it's likely the security team with input from legal. The guide includes a RACI matrix and stakeholder involvement recommendations based on company size.
David A. Moline, CISSP | CISM
Your AI automation, built by someone who secures DoD systems.
Stop Guessing About AI Vendor Risk
Your AI vendor's terms matter more than their benchmarks. Know exactly what you're agreeing to before you build on their platform.