The ritual most teams skip or botch
Most companies don’t run an annual AI program review. Of those who do, most produce a retrospective deck that no one reads beyond the executive offsite where it gets presented. The ritual is theater.
This is a missed opportunity. The annual review is the moment the executive team has the most context, the most data, and the most permission to make hard calls. Used well, it sharpens the program for the next year by killing what isn’t working, doubling down on what is, and setting the trajectory for what comes next.
Below is the structure we use for the year-end review. It’s three questions, three artifacts, and one meeting. The whole thing fits in 90 minutes plus a few hours of prep.
The three questions
The review is organized around three questions. Each gets honest treatment.
Question 1: What did we ship, and what did we learn?
Not “what was on the roadmap” — what actually shipped. Not “what did we accomplish” — what did we learn from each thing.
For each initiative that shipped:
- What was the success definition at the start of the year?
- Did it meet that definition? (Honest yes/no.)
- What did the team learn about AI capability, the business, the customer that wasn’t known a year ago?
The “what we learned” part is the heart of the question. Organizations that learn faster compound. Organizations that ship without learning end up replaying the same mistakes.
For each initiative that didn’t ship:
- Why didn’t it?
- Was it the right call to not ship?
- What does the not-shipping tell us about how to plan the next year?
Question 2: What’s working, and what should we do more of?
The instinct is to focus on failures in retrospectives. The mature move is to invest equally in understanding success.
For the initiatives or patterns that worked:
- What specifically made them work? Talent? Process? Architecture? Customer fit?
- Is the success transferable to other initiatives?
- What would it look like to ship more of this kind of work in the next year?
This question is uncomfortable for two reasons. First, success is often attributed to “the team’s effort” — true but not useful. The discipline is to extract specific replicable patterns. Second, doubling down requires saying no to other things. The conversation produces strategic clarity that purely retrospective conversations don’t.
Question 3: What should we kill?
The hardest question. Every program has initiatives that are taking resources and not producing returns. They are not visibly failing — visibly failing things get killed naturally. They are zombie — running, costing money, producing modest value, blocking new initiatives.
Specific things to look for:
- Initiatives still in Section 1 of the roadmap after more than 2 quarters.
- Vendor contracts that are renewing but produced less value than expected.
- Internal projects that nobody can defend with current data.
- Pilot programs that have outlived their pilot status without becoming production.
For each candidate kill, the discipline is to write down what’s blocking the kill. Often it’s organizational — someone’s identity is attached, someone made a commitment, the kill would acknowledge a previous mistake. These are real barriers and they cost real money to maintain. The annual review is the moment to push past them.
The three artifacts
Each question produces an artifact. The artifacts are short, written, and survive past the meeting.
Artifact 1: The capabilities log
A list of what the program is capable of doing now that it wasn’t a year ago. Things like:
- We can ship a customer-facing AI feature with appropriate governance in 8 weeks.
- We have an internal knowledge agent serving 200+ employees weekly.
- We’ve operated autonomous agents safely in production for 6 months.
- Our engineering team includes 4 engineers who’ve shipped AI to production.
This is institutional memory. The next leader, the next investor, the next board meeting can refer to this. It’s also the basis for next year’s ambition — what becomes possible because of what’s now established.
Artifact 2: The strategic priorities
A short document — one page — naming the 3-5 strategic priorities for the next year. Each priority:
- One sentence.
- The reasoning (why this, why now).
- The rough scope (what does success look like).
- Who owns it.
This document sits above the quarterly roadmap. It’s the strategy the roadmap implements. Without it, the roadmap drifts toward whatever felt urgent that quarter.
Artifact 3: The kill list
The list of things being killed, with rationale. Each item:
- What it is.
- What it was supposed to do.
- Why it’s being killed.
- What we learned from running it.
The kill list serves two purposes. It documents the decisions so they don’t get reversed quietly later, and it preserves the learning so the program doesn’t make the same mistake.
The meeting
90 minutes, executive team, agenda fixed.
Minutes 0-10: Welcome and the three questions. Framing. Set the expectation that this is a working meeting, not a presentation.
Minutes 10-30: Question 1 (what did we ship, what did we learn). Initiative-by-initiative walk-through. The program lead or fractional CAIO presents; the executive team reacts and adds context.
Minutes 30-50: Question 2 (what’s working). Discussion. The goal is identifying 3-5 specific patterns that worked, not just listing wins. The output is an outline of the capabilities log.
Minutes 50-70: Question 3 (what to kill). This is the longest segment because it’s the hardest. Candidate kills are presented; objections are heard; decisions are made or deferred.
Minutes 70-85: Strategic priorities draft. With the prior 70 minutes of context, the team drafts the 3-5 priorities for the next year. The output is bullet points, not finished prose.
Minutes 85-90: Next steps. Who writes up the artifacts. When the next review happens. When the next quarterly roadmap incorporates the priorities.
What goes wrong without this
Three patterns we see in companies that skip the review:
Roadmap drift. Without explicit strategic priorities, the quarterly roadmaps drift. Initiatives are added because they sounded good in the moment. The cumulative direction loses coherence.
Zombie initiatives compound. Without an annual kill conversation, projects survive past their useful life. Resources are spread across too many things. The program’s velocity declines.
Learning doesn’t transfer. Without the explicit “what did we learn” exercise, lessons stay with individuals instead of becoming institutional. Turnover erases hard-won understanding.
The cost of skipping is gradual. The benefit of running is also gradual. The compounding effect across years is large in both directions.
What to do if you’ve never run one
If this is the first time:
- Schedule the meeting for 4 weeks out. Long enough to do real prep, short enough to maintain energy.
- Assign the prep. The fractional CAIO or VP AI does most of the data gathering. Each initiative owner contributes their input.
- Pre-circulate the data. Don’t show numbers for the first time in the meeting. Pre-read materials should be 5-10 pages, mostly written prose plus a few key charts.
- Run the meeting. Use the structure above. Resist the urge to make it presentation-heavy.
- Produce the artifacts within a week. Three short documents, circulated, archived. The first time it’ll feel rough. The second time it’ll feel essential.
What to expect in years 2 and 3
The review gets more valuable each year because the historical context compounds. Year 1: you’re producing the first set of artifacts. Year 2: you can compare to year 1, see trends, refine. Year 3: you have institutional patterns and the review becomes a strategic instrument, not an inventory.
Companies that run this for 3+ years consistently outperform companies of comparable size and ambition that don’t. The mechanism is mostly about the kill discipline and the strategic-priority discipline. Both compound.
The take
The annual AI program review is one of the highest-leverage executive rituals in a maturing AI program. Three questions — what did we ship and learn, what’s working, what to kill — produce three artifacts — capabilities log, strategic priorities, kill list — in one 90-minute meeting. The discipline is small. The annual compounding is enormous. Programs that run this beat programs that don’t, in every meaningful metric, by year 3.
The year-end review is part of the cadence we set up in Fractional CAIO engagements. If your AI program is overdue for an honest annual look, schedule a call.
