Rolling out AI beyond the engineering team

For the first two years of the LLM era, AI spending was a CTO problem: pick tools, run pilots, report progress. In 2026 it is a CFO problem, and the numbers explain why:

• Enterprise LLM spend roughly doubled in two years (about $4.5M to $7M on average in a16z’s enterprise survey, with a further 65% rise expected this year), and the share of it classed as innovation experiment collapsed from 25% to 7%. The rest is operating expense, and operating expense gets reviewed.
• MIT’s “GenAI Divide” report put a number on the review’s fear: roughly 95% of enterprise GenAI pilots showed no measurable P&L impact, against $30 to 40B of spend.¹
• In a 2026 survey of over 5,000 finance leaders, 51% named proving ROI as the top blocker to further AI spend, and a third of finance leaders now control AI purchase approvals. Forrester reports enterprises postponing about a quarter of planned AI spend to 2027 under that scrutiny.
• The FinOps Foundation found 98% of organizations now manage AI spend as a cost discipline, up from 31% two years ago, the fastest adoption curve they have recorded.

I run AI adoption for a 200-person regulated fintech, 60 of them in engineering. The field guide covered the engineering side. This article covers the other 140 people, and it is written for the version of the question the CFO asks: how do you give a whole company AI in a way that can attribute its cost, govern its data, and prove what it produced?

The default answer, buy everyone a seat, fails that question four ways. The rest of this article is the architecture that passes it.

Why seats fail the CFO’s question

The list-price math first, because it is the smallest problem. ChatGPT Business is $20 per user per month billed annually ($25 monthly, after OpenAI’s April 2026 price cut), and Claude Team standard seats match. Sixty-five seats is $15,600 a year. Two hundred is $48,000. ChatGPT Enterprise is quote-only, reported around $45 to 75 per seat with a ~150-seat minimum, which lands a 200-person company near $108,000 to $144,000.

Affordable either way. The four real failures:

1. You pay for seats nobody uses. No vendor publishes seat-utilization data for ChatGPT or Claude, so the best proxy is Microsoft 365 Copilot, where independent 2026 surveys put weekly active usage at 20 to 30% of purchased seats, enterprises average 36% adoption, and Gartner estimates $124K per year of license waste at enterprise scale. Two-thirds of the spend produces nothing, and you cannot tell which two-thirds.

2. The usage you do get is unattributable. A seat tells you who has access. It cannot tell you which team’s usage grew, what it cost per department, or what it correlates with. The FinOps Foundation states the structural reason plainly: LLM calls are transactions, not taggable cloud resources, so attribution has to be captured at the application layer. A consumer seat has no application layer you control.

3. The sensitive traffic runs elsewhere anyway. UpGuard’s shadow-AI research found over 80% of workers using unapproved AI tools. Verizon’s 2026 DBIR measured shadow-AI detections up 4x year over year, with two-thirds of AI access on corporate devices via personal accounts. IBM’s 2026 breach report connects the cost: 43% of AI-related breaches now involve shadow AI, at $5.39M per incident on average, and 92% of organizations with an AI-related breach lacked AI access controls.

4. A seat is a bet that one vendor’s models, at one vendor’s prices, stay the right answer. Today’s seat pricing looks great because it is subsidized: the labs are selling below cost to win distribution, the way every platform land-grab has, and repricing arrives once the workflows are locked in. The restructuring has started; Anthropic moved its Enterprise seats from bundled usage toward seat-plus-usage billing this year. And a seat covers one vendor’s catalog, when the better answer for a given job is often an open-weight model served by competing inference providers at a fraction of frontier prices.

Put together: the seat model buys idle licenses, no attribution, no control over the traffic that carries the risk, and a pricing exposure that grows with adoption. The MIT study’s sharpest finding fits here too: workers’ personal-account AI usage succeeded at the same companies where official pilots stalled. Demand is not the problem. The sanctioned path is.

The requirement list, and the architecture

Written as requirements, the CFO’s question plus the shadow-AI reality produce a short list:

Requirement	Why
One interface people already use	Adoption dies on new logins; shadow AI wins on convenience
Every model call through one control point	Attribution is only complete if nothing bypasses it
Spend attributed to person and team	Showback first, chargeback when coverage is high
Hard budget caps, enforced before the provider	A cap that alerts after the fact is a report
PII controls in the pipe	Policy documents do not scan traffic
Model catalog and residency you choose	Regulated data, EU residency
Audit trail for every change	The board delegated AI to the audit committee for a reason

The architecture that satisfies it:

flowchart TD
  E["All 200 employees: Slack assistant"] --> GW["Self-hosted LLM gateway"]
  P["Internal tools portal (SSO)"] --> GW
  D["60 devs: coding agents"] --> GW
  GW --> V["Virtual keys
  budget, model allowlist, rate limits per person"]
  V --> PII["PII scan: redact deterministic classes"]
  PII --> M["Model providers, EU residency terms"]
  GW --> L["Spend ledger: per person, per team, per model"]

Everything converges on one self-hosted gateway (ours is Bifrost; LiteLLM and Portkey occupy the same slot). No employee or service ever holds a raw provider key. Each holds a virtual key: a scoped credential carrying its own budget, rate limits, and model allowlist, minted and revoked centrally.

The gateway is also what turns model choice from a procurement decision into a routing decision:

• The default tier runs a capable open-weight model through an inference vendor on EU terms, at a fraction of frontier per-task cost. Frontier models sit in higher tiers for the jobs that measurably need them.
• Teams get the best model for each job, coding included, because the catalog spans providers instead of one vendor’s lineup.
• When a provider reprices, degrades, or has an outage, the response is a config change, not a renegotiation. We have exercised all three cases in the first month.

Prior art exists and is worth reading: Wealthsimple, also a regulated fintech, published the closest architecture, a self-hosted LLM gateway with PII redaction serving over half the company daily, built on the principle that the secure path must be the path of least resistance. Where ours differs is delivery (Slack-native rather than a custom web app), an explicit org, team, person budget hierarchy, and the scale point: at 200 people you get no platform team, so everything below had to be operable by approximately one person.

Slack is the front door because it is already open

Rolling out a new AI app to 140 non-technical people means a new login, a training session, and an adoption chart that decays weekly. We skipped the app. The assistant lives in Slack:

• Zero onboarding. A non-technical employee needs an identity account and Slack membership. The onboarding doc’s line for them: “non-developers stop here.” Identity is workspace membership, so there is no user database and nothing to configure.
• Ambient discovery. People see colleagues using the assistant in channels. That is the adoption mechanism; announcements are a supplement.
• Mobile comes free, which for operations and support staff decides whether the tool exists at all.

This stopped being a contrarian choice in 2026: Salesforce relaunched Slackbot as a personal AI agent in January and reports custom Slack agents growing 300% in months. The build-vs-buy line runs where it always runs in regulated companies: the vendor version ties you to their plans, their models, and their governance surface. A Slack bot backed by your own gateway keeps model choice, data control, and attribution while sitting in the same channels.

The runtime behind ours is thin: stateless workers that rebuild conversation context from the Slack thread each turn (a restart loses nothing, scaling is a replica count), an application-level gate that caps concurrent turns and declines quickly past the queue limit, and an outbound-only connection so there is no public URL to defend. Every agent is a registry entry plus a prompt file. The interesting engineering is model selection: each agent runs a cheap base model and escalates to a stronger one on four deterministic triggers (the user asks, one retry after a failed call, long context, or a task-class match), and every selection is logged with its reason, so the ledger can attribute spend to routing causes. Published router research (FrugalGPT, RouteLLM) points toward learned routing; deterministic rules run first because they generate the outcome data a learned router would need.

Governance is a config file with a policy engine

The part the audit committee cares about:

• Budgets are a hierarchy: org, team, person. A request is refused if any level is exhausted, and refused means the gateway returns an error before the provider is ever called. We verified the refusal by exhausting a key and watching the 402 come back, rather than trusting documentation, a habit the rest of this section justifies.
• One reviewed file is the source of truth. People, teams, tiers, and budgets live in a single config file. Merge triggers an idempotent apply, so merge is the deploy and a failed apply is a failed deploy. An admin console has grown into a PR factory over that file: a team lead requests a change in a form, the console validates it against policy and opens the PR, and a policy engine decides who merges. Within-team changes inside budget auto-merge with a signed change record. Structural changes (a new team, a cross-team grant) wait for a human. Requests exceeding caps are refused before a PR exists.
• Model tiers are allowlists, and friction is a control. The default tier is a capable open-weight model on EU inference terms; the premium tier adds Claude. After one frontier-model binge drained the inference balance in days, the strongest models moved to a separate opt-in tier that nobody holds by default. Making expensive models an explicit, logged request changed behavior more than any usage policy had.
• Data-handling promises are probed, not assumed. Our inference vendor’s no-training setting is one dashboard click from silently off, so a gateway plugin stamps the no-training directive on every outbound request and fails closed if it cannot, and a watchdog probes a known training-enabled endpoint every ten minutes to prove the exclusion holds. A promise you cannot probe is a setting.
• Prompts are not a database. The spend ledger keeps metadata (about 1KB per request) for years. Post-redaction request content goes to a private EU object store that physically expires it after 48 hours.

The PII scanning taught us the most, so here is the whole story. The gateway scans every request with Microsoft Presidio: 14 deterministically verifiable classes (card numbers, IBANs, government ids, API keys) are redacted, statistical classes (names, locations) are logged only, and there is no blocking mode, because a false-positive block converts the AI platform into an outage generator. That design was right, and it still failed without a trace: for nine days, nearly half the traffic the scanner was asked to check passed through unscanned, because a fixed scan deadline timed out on any long prompt and the plugin failed open, as designed. The fix was a size-scaled deadline. The bigger fix was instrumentation: a startup canary that sends a synthetic card number and reports whether redaction is armed, and a watchdog that scans a realistic long payload every ten minutes. Fail-open is only defensible while you measure how often “open” happens.

One more 2026 fact for the same audit committee: the EU AI Act’s AI-literacy duty (Article 4) became enforceable this month, with deployer-tier penalties up to 3% of global turnover. It applies to everyone operating AI on the company’s behalf, all 200 people, and a governed rollout with training built into onboarding is the compliance evidence, produced as a side effect.

The first production month, measured

Designing this took weeks. Operating it produced the lessons, and they are the part I have not seen written elsewhere:

1. A router must compose with per-key allowlists. We run complexity routing at the gateway: classify each request, send it to the cheapest model that handles that tier. It shipped dark behind a flag (which caught two config-validation surprises no dry run sees), then went live, and then one internal service’s requests started failing: the router escalated them to models that service’s key disallows by design, and its daily job returned 403 for three days. Routing and authorization are two policies on the same request. The router has to be key-aware, or it routes traffic into walls you built on purpose.
2. Every routing rung needs a fallback on a different provider. The inference vendor’s prepaid balance ran dry one morning and every tier failed until top-up, because all rungs sat on one provider. Each rule now carries a cross-provider fallback to a directly contracted model.
3. Scope spend caps to the provider. A daily cap meant for one expensive model was applied without a provider scope and blocked an entire routing tier org-wide for a day. The cap survived the incident review; the missing scope did not.
4. Choose models by blind comparison, and let people opt into the loser. Rung assignments come from blind bake-offs: candidate models judged on a fixed task set with per-task cost recorded. One fashionable model lost 12 of 15 blind comparisons at 4x the incumbent’s cost and was excluded from the default ladder; we still published an explicitly labeled opt-in ladder carrying it, to collect a week of real feedback from the people who wanted it. The benchmark bounds the decision without having to end the argument.
5. Report subtractions. We had bundled two prompt-compression tools on the theory they would cut token spend. Measured on real sessions, the content they compress is about 2.4% of prompt tokens, so the saving rounded to zero and both came out of the installer. A rollout that only ever adds tooling is not measuring its tooling.

On ROI reporting itself, the sequencing that survives CFO review: showback before chargeback. The ledger attributes every request to a person, team, and routing cause from day one, and the first weeks of data have already changed tier design and caps. Publishing team-level dashboards comes first; charging budgets back to departments waits until attribution coverage is high enough that nobody can dispute the bill.

Attribute every request → Showback dashboards → Chargeback to teams → CFO review

↺ budgets, tiers, and caps adjust from what the ledger shows

And the honest position on the MIT 95% number: much of what it measured is measurement failure, pilots with no baseline and no attribution, which is a failure you can architect out before the first prompt is sent.

Who owns what

The rollout stabilized when ownership matched the stakeholder map rather than the org chart:

Stakeholder	What they own	What the platform gives them
CFO / finance	AI budget, ROI review	Spend ledger by team and person, showback dashboards, hard caps
CISO / risk	Data governance, incidents	PII scan logs, audit trail, model allowlists, no-training probes
HR / compliance	AI-literacy duty (EU AI Act Art. 4)	Training evidence produced by onboarding
Platform (one person)	The gateway and Slack runtime	Config-as-code, policy-engine merges, canaries
Team leads	Their team’s access and budgets	The request form; within-policy changes self-serve

Key takeaways

• AI spend is now reviewed as operating expense. An architecture that cannot attribute spend to people, teams, and outcomes fails the review regardless of how good the models are.
• Seats fail four ways: 20 to 30% weekly utilization on the best available proxy data, no attribution layer, 80%+ shadow-AI usage carrying the sensitive traffic around them, and a subsidized price that locks you to one vendor’s catalog before the repricing arrives.
• A gateway makes model choice a routing decision: open-weight models for the default tier, frontier models where measurement justifies them, and provider changes handled in config.
• Put the assistant where people already work and make the sanctioned path the convenient one. Governance that adds friction loses to a personal browser tab.
• Converge every surface on one self-hosted gateway with per-person virtual keys, and enforce budgets by refusal before the provider is called. Verify the refusal yourself.
• Run governance as config with a policy engine: routine changes auto-merge with signed records, structural changes wait for humans, over-cap requests are refused before a PR exists.
• Instrument the guardrails. Our PII scanner failed open on long prompts for nine days; a canary and a watchdog now make its armed state observable. The same applies to no-training settings: probe them.
• Routers, caps, and fallbacks interact with authorization in ways only production reveals: make routing key-aware, scope caps to providers, and give every rung a cross-provider fallback.
• Sequence ROI reporting as showback then chargeback, and treat the famous 95%-of-pilots-fail number as what it mostly is: unmeasured pilots, a failure you can design out on day one.

Footnotes

1. “The GenAI Divide: State of AI in Business 2025,” MIT NANDA, July 2025. Seat prices from vendor pricing pages, August 2026; ChatGPT Enterprise figures are procurement-report estimates. Copilot utilization figures are third-party survey aggregates. ↩