AI Governance for Companies That Don't Have a Compliance Team

The governance gap nobody talks about

Enterprises have AI governance because they have compliance functions, regulatory exposure, and lawyers on staff. Mid-market companies — call it $20–250M revenue — typically have none of that, but they have most of the AI risk. They’re shipping AI features faster than enterprises (less process), with less mature data hygiene (smaller eng teams), and with less insurance against the consequences.

When something goes wrong — a customer’s data leaks into a prompt, an autonomous agent makes a refund it shouldn’t, a hallucinated response triggers a regulatory complaint — the mid-market response is usually “we’ll figure it out.” Sometimes that works. Increasingly it doesn’t.

This post is about the minimum-viable governance stack for that company. It is not a substitute for actual legal advice. It is the baseline that prevents the most common, most preventable failure modes. Five documents, three runtime controls, one quarterly cadence.

The five documents

You need exactly five governance documents. They should each be 1–3 pages. Pages, not slides. If you can’t articulate your AI governance in 15 pages total, you don’t actually have governance.

1. Acceptable Use Policy (internal)

What employees can and cannot do with company data and external AI tools. Specifically:

• Which tools are approved (ChatGPT Enterprise, Claude.ai with SSO, etc.) and which are banned for company data.
• What data categories cannot be put into a public LLM (customer PII, financial detail, anything under NDA).
• What approval is required to deploy AI in a customer-facing surface.
• Who employees ask if they’re unsure.

Two pages, signed at hire, refreshed annually.

2. AI System Inventory

A spreadsheet — yes, a spreadsheet — listing every AI system in production and pilot. For each:

• What it does (one sentence).
• What model powers it.
• What data it has access to.
• Who owns it.
• What its failure mode is (best-case “wrong output,” worst-case “regulatory issue”).

This inventory is the document that prevents you from being surprised when something breaks. If you cannot produce this list in an hour, you do not know what AI is running in your company.

3. Data Boundary Document

A 1-page document describing which data categories can flow to which model providers. Looks like a table. Customer PII → only Bedrock with VPC isolation. Internal company data → approved frontier providers under DPAs. Marketing copy and PR drafts → anything goes.

The document doesn’t have to be perfect. It has to be decided so that engineers don’t have to make the call each time.

4. Incident Response Runbook

A short, specific runbook for when an AI system misbehaves. What gets paged. Who has the kill switch. How to roll back. How to do the post-mortem. This document gets used about once a quarter. The cost of not having it is measured in the time between “something is wrong” and “we have it contained.”

5. Vendor & Model Provider Register

A list of every model provider and AI vendor you do business with, the DPAs in place, the data categories they’re permitted to process, and the contractual exit terms. This is what your CFO will ask for the first time a board member asks about AI risk.

The three runtime controls

Documents are necessary and insufficient. The runtime controls below are the ones that actually prevent harm. You don’t need more than these three to start.

1. A kill switch on every autonomous agent

Every agent that can take action without human approval must have a kill switch that can be activated in under five minutes by an on-call engineer. The switch should be tested quarterly. The agent should be observable enough that a human can tell whether to flip the switch.

If you can’t kill an agent in five minutes, you don’t have an agent — you have a long-running process that pretends to be an agent and is going to embarrass you eventually.

2. Cost & rate guardrails

Every system that calls a paid model API has hard-coded daily cost ceilings, per-tenant rate limits, and a circuit breaker that kicks in when the cost spikes 5x over baseline. This is governance disguised as ops. The most common “AI incident” we see at mid-market companies is a runaway loop that runs up a $40k bill overnight. Easily preventable. Routinely missed.

3. Audit logs that survive a subpoena

Every model call, every tool call, every autonomous decision is logged. Inputs are redacted; outputs are sampled; metadata (user, tenant, model, cost, latency) is complete. The logs live in an immutable store with a defined retention window. The cost is small. The downside of not having them is unbounded.

The quarterly cadence

Once a quarter, an hour-long review covering:

1. The inventory. New systems added? Old systems retired? Anything we missed?
2. The incident log. What broke this quarter? What did we learn? What document or control gets updated as a result?
3. The vendor register. New providers? Any contracts up for renewal? Any DPAs to re-paper?
4. The roadmap. What’s coming next quarter, and what governance work needs to land before it?

That’s the entire cadence. One hour. Four times a year. The single highest-leverage hour on the executive calendar in most mid-market AI programs.

Who owns this when there’s no compliance team

The honest answer: someone has to. If there’s no head of compliance, the work falls to one of three roles depending on the company.

• CTO/VP Engineering. Most common. Pairs naturally with the runtime controls. Often weak on the documentation side.
• COO. Good fit when AI affects operations heavily. Strong on documentation, sometimes weak on the runtime side.
• Fractional CAIO. Externalized ownership. Good fit when the company doesn’t have internal capacity and doesn’t want to hire a full-time function. See our Fractional CAIO practice for what this looks like.

The wrong answer is “nobody owns this.” Documents without an owner go stale. Stale documents are worse than no documents because they create the illusion of governance.

What this protects against

The governance stack above is not designed to make you compliant with the EU AI Act. It is not designed to win a SOC 2 audit. It is designed to prevent the four most common harms we see in mid-market AI programs:

1. Customer data leaking into a public LLM (acceptable use + data boundary).
2. Runaway costs (rate guardrails).
3. Autonomous agents taking actions nobody can roll back (kill switch + audit logs).
4. Discovering after the fact that you have no idea what AI is running in production (inventory + quarterly review).

If you cover those four, you have eliminated about 80% of the realized risk we see in our practice. The remaining 20% requires actual compliance work, which is what real lawyers and auditors are for.

The starter kit

If you’re starting from zero today, here’s the order:

1. Today: write the AI System Inventory. One hour, spreadsheet.
2. This week: write the Acceptable Use Policy. Send it to legal for a 30-minute pass.
3. This month: ship the three runtime controls. Most are 1–3 days of engineering each.
4. Next quarter: stand up the cadence.

That’s a complete minimum-viable governance program in 90 days, with no hires.

---

We help mid-market companies stand up governance as part of Fractional CAIO engagements, or as a one-off AI Readiness Audit. Schedule a call if you want a second opinion on where your program sits today.