The model that worked then and doesn’t work now
Big companies adopted the “Center of Excellence” pattern for analytics around 2014–2017. A small centralized team of data scientists served the rest of the business by running analyses on request, building dashboards, and developing models that operating teams then deployed. The CoE pattern worked because analytics had a specific shape: discrete projects, well-defined deliverables, modest integration with operating systems.
When AI started showing up on executive agendas in 2023, the natural move was to apply the same pattern: stand up an AI CoE, staff it with specialists, route AI requests through it. Most large companies did this. Most are now realizing it doesn’t work.
The reasons are structural and worth understanding because they shape what the right org pattern actually looks like.
Why CoE breaks for AI
Four reasons:
1. AI isn’t a discrete project. It’s an ambient capability.
Analytics produces reports and dashboards — discrete deliverables that the operating team consumes. AI produces capabilities that have to be woven into operating workflows. You can’t deliver “an AI capability” to a team the way you delivered “a sales forecasting model.”
The capability has to live inside the product or operating workflow. The team that owns the product or workflow has to operate the AI. The CoE delivering AI to another team is a structural mismatch.
2. The development cycle is too tight for handoffs.
AI development is iterative on a cycle measured in hours. A CoE that’s a separate team from the operating team adds days of coordination cost per iteration. Twenty iterations to ship a prompt becomes “twenty handoffs” which becomes “we’ll get to it next quarter.”
Analytics could tolerate slow handoffs because the cycles were weekly or monthly. AI can’t.
3. The operational responsibility doesn’t transfer cleanly.
Analytics dashboards rarely break in ways that need urgent response. AI systems do — they drift, they fail subtly, they have failure modes that require domain context to diagnose. The CoE can build the system but can’t operate it without the domain team. The operating team can’t operate it without understanding how it was built. Without a single team owning end-to-end, the system rots.
4. The talent doesn’t want the CoE seat.
The strongest AI engineers want to ship products, not deliver to internal customers. They are looking for the seat closest to the actual product or operation. CoE roles end up filled with engineers who are willing to be in a service org — which is a meaningful but different talent population, often less senior than the work requires.
The result: large companies with CoEs report that their best AI engineers leave for product teams (internal or external) within 12–18 months. The CoE becomes a turnstile.
The federated pattern that works
The pattern that’s emerging — and that we recommend — is federated AI capability with thin coordination.
Federated means: AI engineers embed in operating teams. The customer support team has an AI engineer. The product team has an AI engineer. The ops team has an AI engineer. Each engineer ships end-to-end for their domain, owns operations, and is accountable to the operating team’s leadership.
Thin coordination means: a small platform group (1–3 people) owns shared infrastructure — gateway, observability, governance, eval harness. They do not build features. They do not staff operating-team AI initiatives. They provide the platform that all federated AI engineers use.
The roles, concretely:
- Federated AI engineers (embedded). 1–2 per significant operating team. Build and operate AI for that team’s domain. Report to the operating team’s lead, not to a central AI org.
- Platform team (central, small). Owns gateway, observability, model client, eval framework, security and governance tooling.
- Fractional CAIO or VP AI (central, leadership). Sets policy, owns the program-level roadmap, coordinates across federated engineers. Does not manage them day-to-day.
- No CoE. No central “AI delivery team.” No internal customers.
This pattern produces several outcomes that the CoE doesn’t:
- Cycle time stays tight because the engineer building the system is in the room with the operators.
- Operational accountability is clear because the same team that builds the system runs the system.
- The platform team scales without bloating because they don’t deliver features.
- Talent stays because the engineers are shipping product work, not service work.
How to migrate from CoE to federated
If your company has an existing CoE, the migration is a quarter to two quarters of focused work. The sequence:
Phase 1: Identify the right operating teams
Not every team needs an embedded AI engineer. Pick the 3–5 teams where AI has the highest cost-of-delay (see the CoD post). These are the candidates.
Phase 2: Move people
Pair each candidate team with an AI engineer from the CoE (or hire one if needed). The engineer reports into the operating team. The CoE shrinks. Some current CoE staff may move to operating teams; others may move to the platform team or to other roles.
Phase 3: Stand up the platform team
Take the most platform-shaped work out of the CoE — the model gateway, the eval harness, the shared observability — and form a 1–3 person platform team around it. They serve the federated engineers, not the operating teams directly.
Phase 4: Sunset the CoE
The remaining CoE shell either becomes the platform team or dissolves into program leadership. There is no “AI service team” by the end of the migration.
The migration is socially awkward. CoE leadership may resist losing scope. Talent in the CoE may have mixed feelings about being assigned to operating teams. Both are managed through clarity about why the change is happening: the CoE pattern doesn’t ship AI at the rate the business needs.
What about company-wide policy and standards?
A reasonable concern about federation: how do you maintain coherent standards across teams without a CoE?
The answer is two-part:
Policy and standards come from leadership (CAIO/VP AI level), not from the CoE. A small leadership function sets policy, holds the program roadmap, runs the quarterly cadence. This is one or two people, not a team.
Standards are enforced through the platform. The model gateway, the eval harness, the observability stack — these enforce uniform behavior at the technical level. Engineers using the platform are automatically using the standards. No coordination meeting required.
The combination works. Policy + platform = standards without the CoE overhead.
Where CoE still works
Not every aspect of AI fits the federated pattern. Two areas where centralization is correct:
Research and exploration. A small research function looking at frontier capabilities, evaluating models, doing technical due-diligence on vendors. This is naturally central because the work doesn’t fit any single operating team.
Specialized compliance work. When AI work touches highly regulated domains (healthcare, finance), a small central compliance function makes sense. They don’t build the systems — they review the federated work.
Both are small. Both are advisory, not operational. Both fit inside the platform/leadership shell, not as a separate CoE.
The take
The CoE pattern was right for analytics and is wrong for AI. The right pattern is federated capability — AI engineers embedded in operating teams — supported by a thin central platform team and led by a small policy function. Companies running this pattern ship AI 2–4x faster than companies running CoEs, with better operational outcomes and better talent retention. The migration from CoE to federated takes a quarter or two. The compounding benefit is years.
Org design is a recurring topic in Fractional CAIO engagements. If you’re running a CoE that isn’t shipping and want a second opinion, schedule a call.
