--- title: "FinOps for AI" slug: finops-for-ai category: cost term_status: emerging evidence_status: verified summary: "FinOps for AI applies the FinOps operating model to AI spend (tokens, GPU hours, provisioned model capacity and AI inside SaaS) so that the cost of each useful outcome is visible, owned, forecast and decided by the people who can change it." audience: [FinOps practitioners, AI and platform engineers, Product managers, Finance partners, Leaders, Beginners] last_verified: 2026-10-18 key_takeaways: - "FinOps for AI is FinOps applied to a new technology category. The label is emerging, and in 2026 it overlaps with the FinOps Foundation's 'AI Value' and the Tokenomics Foundation's 'AI tokenomics'." - "A price per token is a poor unit of management: tokenizers, cached and uncached input, output, hidden reasoning tokens, tiers and context length all change it. Manage the cost per successful outcome." - "Architecture sets most AI cost: requests × model calls per request × agent depth × tokens per call. Bound those multipliers before you negotiate rates." - "Provisioned model capacity saves money unit by unit: buy a unit only if it alone runs above break-even utilisation, let bursts spill to pay-as-you-go, and match the term to how long you will keep the model." - "Allocation needs metadata on every model call. Billing exports alone cannot say which feature, agent or team spent the tokens." - "A budget control is only as strong as its preconditions: gateways that fail open, keys outside the gateway and uncapped agents defeat it." - "Judge a cheaper model by total cost per resolved case, including human review: in this page's worked example, tokens are about 11% of that cost." related: [finops, llmops, mlops, modelops, greenops, kubernetesops, procurementops, platformops] --- ## What FinOps for AI is ### In 30 seconds **FinOps for AI is the FinOps operating model applied to AI spend.** Engineering, product, finance and procurement share timely data on what AI costs and produces, and decide together what is worth paying for. In the FinOps Foundation's words, it "focuses on addressing the cost complexity, faster development cycle, spend unpredictability, and the need for a greater degree of policy and governance to support innovation through allocation, forecasting, and optimization decisions that align consumption, investment, and business value."[^fin-ai-cat] What is new is the meter: tokens, GPU hours, capacity paid around the clock, and AI credits in SaaS. Tokens are the pieces of text, image or audio a model reads and writes, priced per million and by type. **In one line: put the cost of each useful AI outcome in front of the people who choose the model, the prompt and the capacity, early enough to change them.** **For** FinOps, engineering, product, finance and leaders ([routes by role](#learning-path)); example prices are illustrative. ::: note Term status: emerging, and the names are moving "FinOps for AI" names one of five technology categories in the FinOps Framework;[^fin-ai-cat] its definitions are still settling, so this Atlas classes it as **emerging**. In 2026 the labels moved. The Foundation's FinOps-for-AI topic page now redirects to "AI Value",[^fin-ai-value] and its certification is "FinOps Certified: AI Value".[^fin-cert] In June 2026 the Linux Foundation announced its intent to launch the Tokenomics Foundation "in close partnership with the FinOps Foundation";[^lf-tokenomics] that foundation treats FinOps for AI as the *consumption* part of a wider "AI tokenomics" (a draft definition).[^tok-about][^tok-what] This page's boundaries with [LLMOps](../llmops/) and [FinOps](../finops/) are an **editorial distinction**, not a standard. ::: ### At a glance | Item | Summary | |---|---| | Goal | A known cost per useful AI outcome; AI spend that is owned, forecast and justified by value | | Users | FinOps, engineering, product, finance, procurement, leaders | | In scope | Token, GPU and capacity economics; cost per outcome; service tiers; allocation of shared models and GPUs; budgets and caps; forecasting; value reviews | | Handed elsewhere | Generic mechanics, building caching and routing, training, GPU clusters, model approval, contracts, carbon ([neighbours](#how-finops-for-ai-relates-to-other-ops)) | | Prerequisites | FinOps basics; how one LLM API or GPU platform bills | | Key outputs | Request metadata standard; AI cost data joined to outcomes; unit-cost scorecards; budgets and caps; capacity decisions; stage-gated funding | | Term status | **Emerging** (note above); billing-data standards for AI are still drafts[^focus-ai] | ### The problem it solves AI spend breaks four habits formed on virtual machines. 1. **The unit hides multipliers.** Output tokens often cost "three to five times more than input tokens", and stateless APIs resend the history each turn.[^fin-token] Reasoning tokens are invisible yet "billed as output tokens".[^oai-reasoning] Anthropic's tokenizer for Claude 4.7 and later "produces approximately 30 percent more tokens for the same text than earlier models".[^anth-pricing] 2. **Usage compounds.** Agent multipliers "show up only on the invoice",[^tok-bigt] and unit cost "is not a fixed price. It is a distribution."[^fin-token] 3. **New spenders, channels and lifetimes.** Product, marketing and sales create AI spend, often on SKUs that cannot be tagged natively.[^fin-ai-overview] From June 2026 all GitHub Copilot plans bill by "the model and the number of tokens consumed".[^gh-copilot] Provisioned capacity is charged "whether or not the deployment is handling requests",[^az-ptu] yet "You cannot commit for three years to something with a six-month shelf life."[^tok-what] 4. **Value is unproven.** In the State of FinOps 2026, 98% of respondents (self-selected FinOps practitioners) manage AI spend, up from 31% in 2024. Their top AI challenges are visibility, allocation and ROI. One said: "Is your AI providing value? No one can answer that question yet."[^sof-2026] | Operating reality | Without FinOps for AI | With FinOps for AI | |---|---|---| | Unit of management | A monthly bill per provider | Cost per resolved case or document | | Model choice | The most capable model everywhere | The cheapest model that clears a measured quality floor | | Allocation | One shared key | Owner metadata on every call | | Capacity | Bought for the peak, or never | Bought unit by unit while each clears break-even; spillover | | Runaway agent | Found on the invoice | Stopped by step limits, caps and budgets | ### What it is not | Belief | Reality | |---|---| | "A new discipline." | FinOps applied to a technology category: "Basic Price * Quantity = Cost equation still applies".[^fin-ai-overview] | | "Using AI to do FinOps." | That is "AI for FinOps", which the Foundation treats separately;[^fin-ai-overview] see [FinOps](../finops/#automation-and-ai). | | "AIOps." | [AIOps](../aiops/) applies machine learning to IT-operations events. | | "LLMOps with a budget." | [LLMOps](../llmops/) builds the controls; this page prices them and settles who decides. | | "A crypto word." | In Web3, tokenomics means a coin's supply design; the AI usage is unrelated.[^tok-what] | ## Where it comes from and where it sits ### Origins and evolution Per-token pricing made design a cost decision; GPU scarcity made capacity a commitment. Traceable milestones: - **2023:** FrugalGPT reports LLM API fees that "can differ by two orders of magnitude" and proposes model cascades.[^frugal] - **2024–2025:** Foundation working groups map AI services (December 2024), infrastructure choices and forecasting.[^fin-ai-overview][^fin-choosing][^fin-forecast] - **2026:** AI is a technology category of the Framework;[^fin-ai-cat] on 3 June the Linux Foundation announces its intent to launch the Tokenomics Foundation;[^lf-tokenomics] from July the working draft of FOCUS (the FinOps Open Cost and Usage Specification) gains model and token properties, not yet released.[^focus-ai] The evidence is mostly Foundation guidance, provider documentation and a few papers; no peer-reviewed study of the practice was found, so this page omits unsupported savings figures. ### Glossary and mental model An LLM API counts each call's tokens by type and multiplies by a price per million; images and audio become tokens too, usually at higher prices.[^fin-token] - **The cost of one model call** is *(uncached input + cache reads × r + cache writes × w) × input price + output × output price*, times a tier factor (batch, flex, priority) and any context-length or regional modifier. Typically r = 0.1 and w = 1.25 (October 2026).[^anth-pricing][^oai-cache] Most APIs count hidden reasoning inside output, but not all ([conventions](#methods-standards-and-frameworks)). - **The cost of an outcome** adds capacity paid whether used or not, retrieval, tools and people, divided by successful outcomes. - **Call growth:** the Tokenomics Foundation's informal "Big-T" notation writes consumption as `T(n · k · a)`, "requests × model calls per request × agent depth", from T(1), a cache hit, to T(∞), endless loops.[^tok-bigt] ::: figure Where the cost of an AI outcome comes from ```mermaid %%{init: {"flowchart": {"wrappingWidth": 170, "nodeSpacing": 28, "rankSpacing": 34}}}%% flowchart TB O["Outcome: one resolved case, document or decision"] --> N["n: requests per outcome"] N --> K["k: model calls per request: retries, tool turns, reasoning steps"] K --> A["a: agent depth: sub-agents that call models"] A --> T["Tokens per call: uncached input, cache read, cache write, output with reasoning"] R["Retrieved context: chunks × chunk size; images resent each turn"] --> T T --> P["Price per token type × tier × context length × region"] P --> S["Cost per successful outcome"] C["Capacity: provisioned units, GPU-hours, vector store; paid whether used or not"] --> S H["People: review, escalation, operations"] --> S classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029 classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029 classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029 class O,S decision class N,K,A,R risk class T,P core class C,H neutral ``` Text equivalent: An outcome needs n requests, each making k model calls, with agent depth a. Each call consumes uncached input, cache reads and writes, and output with reasoning; retrieved context and resent images enlarge the input. Tokens are priced by type, tier, context length and region. Token cost, capacity paid whether used or not (provisioned units, GPU-hours, a vector store) and people's time give the cost per successful outcome. The multipliers and retrieved context are marked as risks. ::: - **Cache read and write; context window creep:** reusing a stored prompt prefix at a discount after paying a premium to store it; input growing as each turn resends the history.[^fin-token] - **Service tier:** one model priced by speed: batch, flex, priority or fast, reserved ([comparison](#deep-comparisons)). - **Provisioned throughput:** capacity reserved for a term in provider units (PTUs on Azure; model units or reserved tokens per minute on Bedrock);[^az-ptu][^br-pt][^br-tiers] **spillover** sends the excess to pay-as-you-go.[^az-spill] - **Idle allocated and unallocated capacity:** deployed but unused; reserved but assigned to no model.[^fin-capacity] ### How FinOps for AI relates to other Ops | Neighbour | Handoff rule | |---|---| | [FinOps](../finops/) | Allocation, commitments and cadence there; AI meters, capacity units and outcome cost here | | [LLMOps](../llmops/), [KnowledgeOps](../knowledgeops/) | They build routing, caching, retrieval and evaluations, and own the corpus; this page prices their effect and owns who decides | | [MLOps](../mlops/), [KubernetesOps](../kubernetesops/) | They run [training jobs](../mlops/#operating-training-jobs) and GPU pools; this page values and allocates GPU hours | | [ModelOps](../modelops/) | Approval and risk tier there; pricing and commitments here | | [ProcurementOps](../procurementops/), [PlatformOps](../platformops/) | Contracts and the AI gateway as a product there; forecasts, metadata rules and budgets from here | | [GreenOps](../greenops/) | Carbon trade-offs there, from this page's usage data | ::: figure FinOps for AI and its neighbours: what crosses each boundary ```mermaid %%{init: {"flowchart": {"wrappingWidth": 150, "nodeSpacing": 30, "rankSpacing": 50}}}%% flowchart TB FO["FinOps: allocation rules, commitments, cadence"] -->|"generic mechanics"| FA["FinOps for AI: unit economics, capacity and budget decisions"] MO["ModelOps: approved models, risk tiers"] -->|"allowed models"| FA ML["MLOps and KubernetesOps: training jobs, GPU pools"] -->|"GPU-hours, utilisation"| FA LL["LLMOps: routing, caching, evaluations"] <-->|"quality results and cost targets"| FA FA -->|"demand forecasts, unit cost"| PR["ProcurementOps: contracts"] FA -->|"usage and cost data"| GR["GreenOps: energy and carbon"] classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029 class FA core class FO,MO,ML,LL,PR,GR neutral ``` Text equivalent: FinOps for AI receives generic mechanics from FinOps, approved models from ModelOps, and GPU-hours and utilisation from MLOps and KubernetesOps; it trades quality results and cost targets with LLMOps; it sends forecasts and unit cost to ProcurementOps and usage data to GreenOps. ::: ## Principles and operating model ### Charter and principles The six FinOps principles apply unchanged ([FinOps](../finops/#charter-and-principles)). **This Atlas proposes** six AI-specific ones: 1. **Price the outcome, not the token:** "Consider the total cost to get a successful outcome."[^fin-token] 2. **Quality floor first:** benchmark quality against cost "for their specific use case, not against generic benchmarks".[^fin-tools] 3. **Bound the multipliers:** OWASP's controls against "Denial of Wallet" include input validation, rate limits, timeouts and limits on queued and total actions;[^owasp] step, retry and output caps are this Atlas's agent version. 4. **An owner on every call:** multi-agent allocation lacks a "generally accepted" framework,[^fin-ai-cat] so tag at the call. 5. **Commit only to what you will still use:** each unit above break-even, for no longer than you keep the model. 6. **Fund in stages and count people:** set "the limits of both costs and time" before a pilot,[^fin-ai-overview] and include human review in outcome cost. ### Roles, ownership and RACI The Foundation recommends an "AI Investment Council or similar body to regularly approve, evaluate, and track the impact of AI projects".[^fin-ai-cat] **RACI for recurring decisions (this Atlas's proposal).** R = responsible, A = accountable, C = consulted, I = informed. | Decision | FinOps | Engineering | Product owner | Finance | AI investment council | |---|---|---|---|---|---| | Request metadata and allocation rules | A/R | R | C | C | I | | Model and tier for a feature | C | R | A | – | I | | Budgets, caps and step limits per feature | R | R | A | C | I | | Buying provisioned capacity or GPU reservations | R | C | C | A | I (A above threshold) | | Funding a use case through a stage gate | C | C | R | C | A | | Spend anomaly triage | C | R | A | I | – | ### Lifecycle: from idea to retirement The Foundation describes Crawl (validate feasibility, minimal cost, fail fast), Walk (integrate, budgets split) and Run (core processes, higher non-functional requirements) for AI initiatives,[^fin-ai-overview] and investment statuses from proof of concept to scaling.[^fin-ai-cat] The gates below are this Atlas's proposal. ::: figure A stage-gated lifecycle for an AI use case ```mermaid %%{init: {"flowchart": {"wrappingWidth": 170, "nodeSpacing": 26, "rankSpacing": 34}}}%% flowchart TB I["Intake: value hypothesis, outcome unit, quality floor"] --> P1["Proof of concept: time box and cost cap"] P1 --> G1{{"Gate 1: does it work at all?"}} G1 -->|yes| P2["Proof of value: real traffic, cost per successful outcome"] G1 -->|no| X["Stop and record the lesson"] P2 --> G2{{"Gate 2: value exceeds cost with a margin?"}} G2 -->|no| X G2 -->|yes| SC["Scale: capacity model, budgets, forecast"] SC --> RN["Run: weekly unit cost, monthly review"] RN -->|"model, price or volume change"| RE["Re-evaluate cost and quality"] RE --> RN RN -->|"value gone or model retired"| RT["Retire: release capacity, revoke keys"] classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029 classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029 class I,P1,P2,SC,RN,RE core class G1,G2 decision class X,RT risk ``` Text equivalent: Intake records the value hypothesis, outcome unit and quality floor. A capped proof of concept faces gate 1 (does it work?) and a proof of value on real traffic gate 2 (value above cost with a margin?); failing either stops the work. Scale adds capacity, budgets and a forecast; run adds weekly unit costs and re-evaluation after model, price or volume changes; retirement releases capacity and revokes keys. ::: ### Operating model and governance The cadence (this Atlas's proposal) is faster than general FinOps because the Foundation expects "shorter forecasting windows" for AI[^fin-ai-cat] and suggests weekly or monthly forecasting.[^fin-forecast] | Cadence | Inputs | Decisions | Owner | |---|---|---|---| | Continuous | Gateway spend-rate alerts per key and feature | Contain or explain within hours | Product owner; engineer on call | | Weekly | Cost per successful outcome, p95 per request, quality | Prompt, cache, routing or cap changes | Product owner with engineering | | Monthly | Showback, forecast against actual by driver, provisioned utilisation | Re-forecast ([template](#templates)); capacity changes at term ends | FinOps with finance | | On a model or price change | Evaluation on the team's test set | Switch, keep or renegotiate | Engineering with FinOps | | Quarterly | Use cases by stage; value against cost | Fund, scale or stop | AI investment council | ## Architecture and workflow ### Reference architecture and building blocks The architecture is a **cost pipeline joined to request telemetry and business outcomes**, with controls acting on the gateway and the providers (figure below). Its inputs: - **Provider cost APIs:** OpenAI's report daily buckets by project, line item or key;[^oai-costs] Anthropic's report USD cents, typically within five minutes, but leave out Priority Tier costs (October 2026).[^anth-admin] - **Cloud billing exports:** FOCUS 1.4 has no AI columns, so tokens appear through SKU, `ConsumedUnit` and `ConsumedQuantity`.[^fin-ai-cat] - **Everything else:** SaaS invoices, gateway logs or OpenTelemetry GenAI spans,[^otel-genai] GPU telemetry, outcome events. **Trust boundary:** prompts can contain personal data, and the cost pipeline needs only counts and metadata; agree retention with [PrivacyOps](../privacyops/). ::: figure Reference architecture: from billing and request data to decisions and controls ```mermaid %%{init: {"flowchart": {"wrappingWidth": 120, "nodeSpacing": 16, "rankSpacing": 40}}}%% flowchart TB B["Provider invoices and cost APIs"] --> N["Normalise usage conventions; price with a versioned rate card"] F["Cloud billing exports in FOCUS"] --> N L["Gateway logs and GenAI spans"] --> N G["GPU telemetry and cluster allocation"] --> N O["Outcome events from business systems"] --> N N --> R{{"Reconciles to invoices?"}} R -->|yes| A["Attribute to team, feature and use case; idle and unallocated lines"] R -->|no| X["Investigate the gap: usage conventions, unpriced lines, price changes"] X -->|"fix parser or rate card"| N A --> U["Unit cost, forecasts, anomalies, utilisation"] U --> D["Weekly reviews and the AI investment council"] D --> C["Controls: gateway budgets and caps, routing, provider spend limits, capacity purchases"] classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029 classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029 classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029 class N,A,U core class R,D decision class X risk class B,F,L,G,O,C neutral ``` Text equivalent: Five sources (provider cost data, FOCUS billing exports, gateway logs and spans, GPU telemetry, outcome events) are normalised and priced with a versioned rate card, then reconciled to invoices. A gap goes to an investigation (conventions, unpriced lines, price changes) that fixes the parser or rate card and normalises again. Reconciled data is attributed, with idle and unallocated lines, and turned into unit costs, forecasts and anomalies for the reviews and the council, whose decisions act through budgets, caps, routing, provider spend limits and capacity purchases. ::: ### End-to-end workflow **Happy path.** The product owner records the outcome unit, quality floor and volume; engineering and FinOps estimate cost per outcome for two or three models; the platform issues a key or profile per feature with metadata and a budget; the gateway logs every call and the product emits outcome events; the pipeline joins them daily and reconciles monthly; the weekly review acts on the largest lever; model and price changes trigger a re-decision. **Exception path.** At 80% of a period budget the gateway alerts the product owner; at 100% it applies the feature's agreed action (degrade to a smaller model, queue in a batch tier, or reject with a clear error). The owner explains or fixes the cause, and the breach is recorded; the [runbook](#worked-runbook-an-agent-loop-burns-a-month-of-budget-overnight-illustrative) follows this path. ## Practices, frameworks and tools ### Core practices and playbooks | Practice | Trigger | Owner | Verification | |---|---|---|---| | Owner metadata on every call | New key, feature or agent | Platform | Attributed share; log reconciles to invoice | | Compare models on cost per success at a quality floor | New feature or model | Product with engineering | Re-run on every model or prompt change | | Cache stable prefixes | Prefix reused within the cache lifetime | Engineering | Cache-read tokens | | Batch or flex for non-urgent work | Answers not needed within minutes | Engineering | Tier in logs (Bedrock: `ResolvedServiceTier`)[^br-tiers] | | Bound output, steps, retries and input size | Every agent and long-output feature | Engineering | A load test hits the limit | | Route by difficulty | A cheaper model meets the floor on some traffic | Product with engineering | Cost per resolved case falls ([judgement call](#a-judgement-call-route-easy-conversations-to-a-cheaper-model)) | | Size provisioned capacity | Steady demand, or a latency SLO worth a premium | FinOps with engineering | Each unit above break-even | | Budget retrieval | A feature uses RAG | Engineering with FinOps | Retrieved tokens per call; re-indexing cost | | GPU efficiency and shared-capacity allocation | Self-hosted serving or training | Platform with FinOps | SM activity; allocated plus idle equals the bill | ### Playbook: buying provisioned model capacity Provisioned capacity buys predictable latency. In the Foundation's words it "can sometimes cost more per token than shared tiers", for bursty traffic "may be a trap", and without latency requirements "can solely be used to reduce costs", bought like a savings plan to a coverage target.[^fin-capacity] 1. **Prove steady demand:** four or more weeks of hourly tokens by type (Bedrock's Reserved tier also counts cache writes).[^br-tiers] 2. **Normalise the price and buy unit by unit:** break-even utilisation = hourly unit price ÷ (tokens per hour per unit × pay-as-you-go price per token). Buy a unit only if its own utilisation, the traffic it would serve beyond the units already bought, clears break-even ([Lab 3](#lab-3-advanced-pay-as-you-go-provisioned-or-self-hosted)). 3. **Load-test your mix:** throughput per PTU varies by model,[^az-ptu] and a vendor's figure "is best treated as a rough estimate".[^fin-capacity] 4. **Match the term to the model's life** (October 2026): Bedrock Provisioned Throughput runs without commitment, one or six months (customised models need it unless base model and Region allow on-demand deployment: some Nova and Llama models in two US Regions);[^br-pt] Bedrock's Reserved tier one or three months;[^br-tiers] Azure reservations one month or one year, with the advice to "always create deployments first, then purchase the Azure Reservation";[^az-ptu] Google sells a "fixed-cost, fixed-term subscription" with weekly and monthly prices.[^gcp-genai] AWS reservations are model-specific, the Foundation reports;[^fin-capacity] Azure's are "a financial discount applied to PTU billing meters", covering any supported model in scope.[^az-ptu] 5. **Plan the overflow:** Azure spillover[^az-spill] and Bedrock's Reserved tier, which "automatically overflows to the Standard tier",[^br-tiers] send excess to pay-as-you-go; elsewhere build it into the gateway. Forecast spilled tokens. 6. **Report idle and unallocated capacity; separate preparer and approver;** file the [decision record](#templates). Do not resize production with traffic: on Azure, scaling down "releases capacity permanently".[^az-ptu] ### Playbook: allocating a shared model endpoint or GPU pool | Method | How it works | Fits | Watch out for | |---|---|---|---| | Tokens consumed | Price each tenant's tokens by type from gateway logs | Shared pay-as-you-go endpoints | Ignores unused reserved capacity | | Reserved share | Tenants reserve and pay for part of the capacity | Provisioned capacity, per-tenant SLOs | Over-reservation stays with the tenant | | Usage plus idle | Usage at the effective rate; idle spread or kept by the platform | Platform-run pools | Agree the idle rule before the first showback | | GPU pool | Charge `max(request, usage)`; the rest is idle | Self-hosted serving and training | Whole-device requests unless partitioned | OpenCost defines workload cost for GPUs as "max(request, usage)".[^opencost] Kubernetes requests GPUs through `limits` (requests, if set, must be equal),[^k8s-gpu] so a pod holding a GPU pays for all of it, unless MIG slices (hardware isolation) or time-slicing ("no memory or fault-isolation between replicas") divide it.[^nv-sharing] That is the device-plugin path; with Dynamic Resource Allocation, stable since Kubernetes v1.35, Pods can share a device through one claim, so allocate by claim ([KubernetesOps](../kubernetesops/)).[^k8s-gpu] **Multi-agent workloads** have no accepted allocation framework;[^fin-ai-cat] this Atlas proposes charging every call in a run to the feature that started it, via a propagated root request ID (Lab 1). ### Methods, standards and frameworks | Method or standard | Status (October 2026) | Use it for | |---|---|---| | FinOps Framework, AI technology category | Community framework, CC BY 4.0 | Scope, personas, KPIs[^fin-ai-cat] | | FOCUS 1.4 (June 2026) | Released; no AI columns | Billing ingestion[^focus-ai] | | FOCUS working-draft AI properties | Unreleased | Planning your schema[^focus-ai] | | OpenTelemetry GenAI conventions | Development | Request-level token telemetry[^otel-genai] | | OWASP LLM10:2025 | Security guidance | Denial-of-wallet controls[^owasp] | | Big-T notation; seven-layer tokenomics stack | Informal community models | Architecture reviews[^tok-bigt][^tok-stack] | **Usage conventions differ.** OpenTelemetry's `gen_ai.usage.input_tokens` "SHOULD include all types of input tokens, including cached tokens", with reasoning inside output;[^otel-genai] OpenAI matches.[^oai-cache] Anthropic's `input_tokens` counts only tokens after the last cache breakpoint.[^anth-cache] One parser for both misprices (Lab 1). Reasoning differs too: OpenAI and Anthropic count it inside output (OpenAI's example: 1,186 output tokens, 1,024 of them reasoning); Gemini reports thinking tokens separately and prices "the sum of output tokens and thinking tokens".[^oai-reasoning][^anth-thinking][^gem-thinking] **Draft FOCUS properties** `ModelDeveloper`, `ModelFamily`, `ModelId`, `ModelVersion`, `TokenCacheAction` and `TokenDirection` arrive in `SkuPriceDetails` with "No new columns";[^focus-ai] name your fields to match. ### Tooling landscape | Tool category | Verified examples | Job | Limits to check | |---|---|---|---| | Provider cost APIs and limits | OpenAI Costs API;[^oai-costs] Anthropic Usage and Cost API, workspace spend limits[^anth-admin] | Authoritative cost; per-provider caps | Bucket sizes; conventions; excluded tiers | | Provider-side tagging | Bedrock application inference profiles[^br-aip] | Allocation inside one cloud | Tags must be activated for billing | | AI gateways and proxies | LiteLLM proxy (open source)[^litellm]; cloud gateways | Metadata, budgets and routing across providers | Budgets depend on the gateway's database | | GPU telemetry and allocation | NVIDIA DCGM;[^dcgm] OpenCost[^opencost] | Utilisation and per-workload GPU cost | Node access; partitioning changes the unit | | FinOps platforms | FOCUS-capable tools ([FinOps tooling](../finops/#tooling-landscape)) | AI next to other technology spend | Request detail rarely reaches the bill | ### Automation and AI Automate the guardrails (budgets, caps, step limits, spend-rate alerts, re-evaluation after model releases); keep people on decisions that trade cost against quality or commit money. ## FinOps for AI in practice ### Reference implementation Four artefacts; all values are placeholders. **1. A request metadata contract:** the fields every call must carry, and those it must not. _Example status: YAML syntax-checked; a design artefact, not tied to one product._ ```yaml # ai-request-metadata.yaml: owner fields every model call must carry (version 1) version: 1 required: team: "Owning team ID from the service catalogue, e.g. support" feature: "Product feature, e.g. assistant" use_case: "Funded use case ID from the investment register, e.g. UC-0142" environment: "prod | staging | dev" budget_id: "Budget the call draws from, e.g. support-assistant-2026q4" root_request_id: "ID of the user request the call serves; joins calls to outcomes" optional: agent_step: "Step number inside an agent run" outcome_event: "Business event that marks success, e.g. case_resolved" forbidden: - prompt_text # never in the cost dataset; content stays in application logs - user_email # use a pseudonymous user ID if per-user cost is needed mapping: gateway_header_prefix: "x-ai-meta-" # e.g. x-ai-meta-team: support otel_span_attributes: # OpenTelemetry GenAI conventions (Development) - gen_ai.request.model - gen_ai.usage.input_tokens # includes cached tokens - gen_ai.usage.cache_read.input_tokens - gen_ai.usage.cache_write.input_tokens - gen_ai.usage.output_tokens # includes reasoning tokens enforcement: "Gateway rejects calls without required fields; dev environments only warn" ``` **2. Provider-side attribution on Amazon Bedrock.** Application inference profiles let you "track usage and costs when invoking a model" and accept tags;[^br-aip] activate the keys as cost allocation tags first ([FinOps](../finops/#2-ownership-metadata)). Tags see only calls made through the profile, so use the `bedrock:InferenceProfileArn` IAM condition key to allow the model only through it; for cross-Region routing, copy from a system-defined (cross-Region) profile instead of a foundation model.[^br-aip] _Example status: syntax-checked with `bash -n`; not executed (needs Bedrock access)._ ```bash # Create an application inference profile that carries cost allocation tags (AWS CLI v2). # Placeholders: REGION and MODEL_ID; use a model your organisation has approved. PROFILE_ARN=$(aws bedrock create-inference-profile \ --inference-profile-name support-assistant-prod \ --description "Support assistant, production" \ --model-source copyFrom="arn:aws:bedrock:REGION::foundation-model/MODEL_ID" \ --tags key=team,value=support key=feature,value=assistant \ key=budget_id,value=support-assistant-2026q4 \ --query inferenceProfileArn --output text) # Check the profile by its ARN (the identifier is the profile ID or ARN, not its name). # Applications then invoke the model with this ARN as the model ID. aws bedrock get-inference-profile --inference-profile-identifier "$PROFILE_ARN" ``` **3. A gateway budget with LiteLLM.** The proxy tracks spend per key, user and team, but checks budgets against its database: without one "none of them cap anything" and `max_budget` "fails open".[^litellm] Keys that bypass the gateway are not capped either, so add provider-side spend limits. Also:[^litellm] the proxy-wide budget covers "all calls on the proxy", so reaching it would stop every team (our inference; keep it above the team budgets); checks trust a Redis counter that after a restart "can come back lower than the spend already recorded" unless `fail_closed_budget_enforcement` reads the database (more load; unverifiable requests get a 503); batch jobs are costed only on completion. _Example status: syntax-checked (YAML, `bash -n`); not run against a proxy (LiteLLM 1.105.1, PyPI 2026-10-14)._ ```yaml # config.yaml for the LiteLLM proxy: central spend tracking and a global ceiling. model_list: - model_name: large-default litellm_params: model: provider/large-model-id # placeholder: provider prefix and model ID api_key: os.environ/PROVIDER_API_KEY - model_name: small-default litellm_params: model: provider/small-model-id api_key: os.environ/PROVIDER_API_KEY general_settings: master_key: os.environ/LITELLM_MASTER_KEY database_url: os.environ/DATABASE_URL # required: without it, budgets fail open fail_closed_budget_enforcement: true # check spend in the database if Redis is stale litellm_settings: max_budget: 20000 # USD per period for ALL calls; keep above the team budgets budget_duration: 30d ``` ```bash # A team with its own monthly budget; keys issued to the team draw from it. curl -sS -X POST "${LITELLM_URL:?}/team/new" \ -H "Authorization: Bearer ${LITELLM_MASTER_KEY:?}" \ -H "Content-Type: application/json" \ -d '{"team_alias": "support-assistant", "max_budget": 2500, "budget_duration": "30d"}' ``` **4. A versioned rate card.** Prices change, so price usage from a dated file, not code constants. _Example status: JSON syntax-checked; values illustrative._ ```json { "rate_card_version": "2026-10-17", "currency": "USD", "unit": "per 1M tokens", "source_note": "Illustrative. Copy real prices from provider pages, with URL and date.", "models": { "large-v1": {"input": 4.0, "output": 20.0, "cache_read": 0.1, "cache_write": 1.25}, "small-v1": {"input": 0.4, "output": 2.0, "cache_read": 0.1, "cache_write": 1.25} }, "tier_factors": {"standard": 1.0, "batch": 0.5, "flex": 0.5, "priority": 1.75}, "stacking_rule": "Check per provider: some stack cache and batch discounts, some do not" } ``` The tier factors echo published examples (October 2026): OpenAI and Google discount batch by 50%, OpenAI prices flex at batch rates,[^oai-tiers][^gcp-genai] and Bedrock lists Flex at 50% off and Priority at a "75% premium" for many models.[^br-pricing] Anthropic's multipliers "stack with" the batch discount;[^anth-pricing] Google's "don't stack".[^gcp-genai] ### Lab 1 (beginner): price, attribute and reconcile LLM requests **Objective.** Price calls in two usage conventions, reconcile and explain a gap, find unowned spend, and compare cost per call with cost per successful outcome. **Prerequisites.** Python 3.8+, standard library; no cost or risk. **Data.** Fourteen illustrative calls: `inclusive` rows count cached tokens inside input (OpenTelemetry, OpenAI), `exclusive` rows report them separately (Anthropic).[^otel-genai][^oai-cache][^anth-cache] Each call carries the `root_request_id` of the user request it served ([metadata contract](#reference-implementation)); a second file records each request's outcome. In the support assistant a small model triages each conversation first. _Example status: verified by execution on 2026-10-18 (Python 3.8.10). Data and prices are illustrative._ Save this as `calls.csv`: ```csv call_id,root_request_id,style,model,tier,team,feature,input,cache_read,cache_write,output r01,c1,inclusive,small-v1,standard,support,assistant,900,0,0,40 r02,c1,inclusive,large-v1,standard,support,assistant,14200,12000,0,420 r03,c2,inclusive,small-v1,standard,support,assistant,950,0,0,35 r04,c2,inclusive,large-v1,standard,support,assistant,15100,12000,0,610 r05,c2,inclusive,large-v1,standard,support,assistant,16300,15100,0,450 r06,c3,inclusive,small-v1,standard,support,assistant,920,0,0,38 r07,c3,inclusive,large-v1,standard,support,assistant,13900,0,12000,380 r08,c4,inclusive,large-v1,standard,support,assistant,41800,12000,0,2900 r09,e1,exclusive,large-v2,standard,sales,email-drafts,1800,6000,0,900 r10,e2,exclusive,large-v2,standard,sales,email-drafts,2100,0,6000,1100 r11,e3,exclusive,large-v2,standard,sales,email-drafts,1500,6000,0,850 r12,i1,exclusive,large-v2,batch,finance,invoice-extract,52000,0,0,6000 r13,i2,exclusive,large-v2,batch,finance,invoice-extract,48000,0,0,5500 r14,,inclusive,large-v1,standard,,,30000,0,0,8000 ``` Save this as `outcomes.csv`: ```csv root_request_id,outcome c1,resolved c2,resolved c3,escalated c4,resolved e1,accepted e2,rejected e3,accepted i1,accepted i2,accepted ``` Save this as `lab1_price.py`: ```python """Lab 1: price, reconcile, attribute and join LLM calls to outcomes (stdlib).""" import csv import statistics import sys from collections import defaultdict # Illustrative rate card: USD per 1M tokens; cache multipliers relative to input price. RATES = { "large-v1": {"input": 4.00, "output": 20.00, "read": 0.10, "write": 1.25}, "small-v1": {"input": 0.40, "output": 2.00, "read": 0.10, "write": 1.25}, "large-v2": {"input": 3.00, "output": 15.00, "read": 0.10, "write": 1.25}, } TIER = {"standard": 1.0, "batch": 0.5} # e.g. a 50% batch discount # The provider's cost report for the same period (illustrative). INVOICED = {"large-v1": 0.6084, "small-v1": 0.00133, "large-v2": 0.35343} SUCCESS = {"resolved", "accepted"} INT_FIELDS = ("input", "cache_read", "cache_write", "output") def split_input(r): """(uncached, cache_read, cache_write) input tokens in either provider convention.""" if r["style"] == "inclusive": # input already contains the cached tokens uncached = r["input"] - r["cache_read"] - r["cache_write"] else: # input excludes them; they are reported separately uncached = r["input"] if uncached < 0: raise ValueError(f"{r['call_id']}: cached tokens exceed input; wrong style?") return uncached, r["cache_read"], r["cache_write"] def price(r): rate = RATES[r["model"]] uncached, read, write = split_input(r) tokens_usd = (uncached + read * rate["read"] + write * rate["write"]) * rate["input"] tokens_usd += r["output"] * rate["output"] return tokens_usd / 1e6 * TIER[r["tier"]] with open("calls.csv", newline="") as f: calls = [dict(r, **{k: int(r[k]) for k in INT_FIELDS}) for r in csv.DictReader(f)] with open("outcomes.csv", newline="") as f: outcome = {r["root_request_id"]: r["outcome"] for r in csv.DictReader(f)} for c in calls: c["usd"] = price(c) # 1. Reconcile computed cost with the provider's cost report, model by model. by_model = defaultdict(float) for c in calls: by_model[c["model"]] += c["usd"] for model, usd in sorted(by_model.items()): gap = INVOICED[model] - usd status = "ok" if abs(gap) < 0.0005 else f"INVESTIGATE: invoice {gap / usd:+.0%}" print(f"{model:9s} computed {usd:.4f} | invoiced {INVOICED[model]:.4f} | {status}") # 2. Attribution: how much has no owner? total = sum(c["usd"] for c in calls) unowned = sum(c["usd"] for c in calls if not c["team"]) print(f"Total {total:.4f} USD; unattributed {unowned:.4f} ({unowned / total:.1%})") # 3. Per call versus per outcome: join every call to the outcome of the request that # started it (root_request_id), then divide by successful outcomes. groups = defaultdict(list) for c in calls: groups[(c["team"] or "UNKNOWN", c["feature"] or "UNKNOWN")].append(c) print("team feature calls median_call roots per_root wins per_success") for (team, feature), cs in sorted(groups.items()): usd = sum(c["usd"] for c in cs) roots = {c["root_request_id"] for c in cs if c["root_request_id"]} wins = sum(outcome.get(root) in SUCCESS for root in roots) per_root = f"{usd / len(roots):.4f}" if roots else "n/a" per_win = f"{usd / wins:.4f}" if wins else "n/a" median = statistics.median(c["usd"] for c in cs) print(f"{team:8s} {feature:15s} {len(cs):5d} {median:11.4f} {len(roots):5d}" f" {per_root:>8s} {wins:4d} {per_win:>11s}") # 4. The classic mistake: one parser for both conventions bills cached tokens twice. inclusive = [c for c in calls if c["style"] == "inclusive"] wrong = sum(price(dict(c, style="exclusive")) for c in inclusive) right = sum(c["usd"] for c in inclusive) print(f"Inclusive rows read as exclusive: {wrong:.4f} instead of {right:.4f} " f"({wrong / right - 1:+.0%})") ``` **Steps.** Save the three files in an empty folder and run `python3 lab1_price.py` in it. **Expected output** (actual output of the run on 2026-10-18): ```text large-v1 computed 0.6084 | invoiced 0.6084 | ok large-v2 computed 0.3213 | invoiced 0.3534 | INVESTIGATE: invoice +10% small-v1 computed 0.0013 | invoiced 0.0013 | ok Total 0.9311 USD; unattributed 0.2800 (30.1%) team feature calls median_call roots per_root wins per_success UNKNOWN UNKNOWN 1 0.2800 0 n/a 0 n/a finance invoice-extract 2 0.1181 2 0.1181 2 0.1181 sales email-drafts 3 0.0207 3 0.0284 2 0.0425 support assistant 8 0.0209 4 0.0824 3 0.1099 Inclusive rows read as exclusive: 0.8622 instead of 0.6098 (+41%) ``` **Validation.** (a) `large-v1` and `small-v1` reconcile; the `large-v2` report is 10% higher because those calls ran with a 1.1× data-residency multiplier (Anthropic charges one for US-only inference on Claude 4.6 and later)[^anth-pricing] missing from the rate card. (b) The unlabelled call r14 is 30.1% of spend and serves no outcome. (c) The assistant's median call costs 0.0209, a conversation 0.0824 (two calls on average, triage included; long calls such as r08 lift the mean) and a resolved conversation 0.1099, because the escalated conversation's cost is spread over the three that succeeded: per call understates the unit cost five times. (d) Reading inclusive rows with the exclusive rule bills cached tokens twice: the log runs 41% above the truth. **Troubleshooting.** A `ValueError` about cached tokens means a wrong `style`. `INVESTIGATE` means a missing row, a mid-period price change or an unpriced line: longer-lived cache writes, a regional or long-context multiplier, a separate thinking field. **Teardown.** Delete the folder. ### Lab 2 (intermediate): context creep and the caching break-even **Objective.** See why conversation cost grows faster than turns, and when caching pays. **Prerequisites.** Python 3.8+. **Assumptions.** A stateless API resends the history each turn; turns arrive within the cache lifetime; each conversation starts with a cold cache (no prefix shared across users) except in the line after the table; write 1.25× and read 0.1×, as Anthropic and OpenAI publish for most current models (some newest models read at 0.05× or 0.025×, October 2026).[^anth-pricing][^oai-cache] Real caches have minimum lengths and can miss.[^br-cache] _Example status: verified by execution on 2026-10-18 (Python 3.8.10). Prices are illustrative._ ```python """Lab 2: context-window creep and prompt-caching break-even (Python 3.8+, stdlib).""" S, U, B = 3000, 150, 350 # system prompt, user turn, assistant reply (tokens) P_IN, P_OUT = 4.00, 20.00 # illustrative USD per 1M input / output tokens READ, WRITE = 0.10, 1.25 # cache read, 5-minute cache write (x input price) def conversation(turns, cached, warm_prefix=False): """Total cost of one conversation in which every turn resends the full history.""" usd = 0.0 for k in range(1, turns + 1): prompt = S + (k - 1) * (U + B) + U # input tokens sent at turn k if not cached: usd += prompt * P_IN elif k == 1 and not warm_prefix: usd += prompt * WRITE * P_IN # cold: turn 1 writes it all else: # Reused part: the previous turn's prompt, or at turn 1 a system prompt that # other users' traffic has already cached. reused = S if k == 1 else S + (k - 2) * (U + B) + U usd += (reused * READ + (prompt - reused) * WRITE) * P_IN usd += B * P_OUT return usd / 1e6 def total_input(turns): return sum(S + (k - 1) * (U + B) + U for k in range(1, turns + 1)) print("turns input_tokens input_share no_cache_usd cached_usd saving") for t in (1, 5, 10, 20, 40): plain, cached = conversation(t, False), conversation(t, True) share = total_input(t) * P_IN / 1e6 / plain print(f"{t:5d} {total_input(t):12,d} {share:11.0%} {plain:12.4f} {cached:10.4f}" f" {1 - cached / plain:6.0%}") warm = conversation(1, True, warm_prefix=True) print(f"1 turn, system prompt already cached by other users: {warm:.4f}" f" ({1 - warm / conversation(1, False):.0%} saving)") # Closed form: T*S + T*U + (U+B)*T*(T-1)/2 grows with the square of the turns. t = 40 assert total_input(t) == t * S + t * U + (U + B) * t * (t - 1) // 2 # Break-even: a cached block of n reads costs WRITE + n*READ instead of (1 + n) uncached. for write in (1.25, 2.0): n = 0 while write + n * READ >= 1 + n: n += 1 print(f"write multiplier {write}: caching pays from read number {n}") ``` **Steps.** Save as `lab2_context.py` and run `python3 lab2_context.py`. **Expected output** (actual output of the run on 2026-10-18): ```text turns input_tokens input_share no_cache_usd cached_usd saving 1 3,150 64% 0.0196 0.0227 -16% 5 20,750 70% 0.1180 0.0670 43% 10 54,000 76% 0.2860 0.1268 56% 20 158,000 82% 0.7720 0.2614 66% 40 516,000 88% 2.3440 0.5906 75% 1 turn, system prompt already cached by other users: 0.0089 (54% saving) write multiplier 1.25: caching pays from read number 1 write multiplier 2.0: caching pays from read number 2 ``` **Validation.** (a) From 20 to 40 turns, input tokens more than triple: total input *T·S + T·U + (U+B)·T(T−1)/2* grows with the **square** of the turns. The Foundation's paper says "exponentially";[^fin-token] quadratic is accurate, and still severe. (b) Output costs five times more per token, yet input is 82% of cost by turn 20. (c) One-turn conversations cost 16% *more* with caching when cold, but 54% less when other users keep the system prompt cached, as scenario A assumes. (d) The break-even matches Anthropic's statement: one read for a 1.25× write, two for 2×.[^anth-pricing] **Troubleshooting.** Low real savings usually mean a changing prefix or traffic too sparse for the cache lifetime. **Teardown.** Delete the file. ### Lab 3 (advanced): pay-as-you-go, provisioned or self-hosted? **Objective.** Size provisioned capacity unit by unit, find the break-even, and stress a self-hosting case. **Prerequisites.** Python 3.8+. **Assumptions (illustrative).** 9M tokens per hour in weekday business hours, 1.5M at night, 2M at weekends; pay-as-you-go at 6.00 per million; a provisioned unit of 2.5M tokens per hour for 10.00 per hour, with spillover; GPUs of 3M tokens per hour at 4.00 per hour, at least two, plus 1,500 a week of people. Self-hosting assumes an open-weights model meets the quality floor and on-demand GPUs at this price every hour (reserved GPUs are paid all week). _Example status: verified by execution on 2026-10-18 (Python 3.8.10). All prices and throughputs are illustrative._ ```python """Lab 3: pay-as-you-go, provisioned units with spillover, or self-hosted GPUs?""" # One week of demand in millions of tokens per hour (illustrative profile). def demand(hour_of_week): day, hour = divmod(hour_of_week, 24) if day < 5 and 8 <= hour < 18: return 9.0 # weekday business hours return 2.0 if day >= 5 else 1.5 HOURS = range(168) PAYG = 6.00 # blended USD per 1M tokens on the shared tier (illustrative) UNIT_CAP, UNIT_USD = 2.5, 10.00 # provisioned unit: capacity (1M tokens/hour), USD/hour GPU_CAP, GPU_USD = 3.0, 4.00 # self-hosted GPU: capacity at target batch, USD/hour MIN_GPUS, OPS_WEEK = 2, 1500.0 # availability floor; people USD per week total = sum(demand(h) for h in HOURS) print(f"Weekly demand {total:.0f}M tokens; peak {max(map(demand, HOURS)):.1f}M/hour") print(f"Pay-as-you-go only: {total * PAYG:9,.0f} USD/week = {PAYG:.2f} per 1M") print("units served_by_units utilisation marginal_util weekly_usd usd_per_1M") best, served_before = None, 0.0 for units in range(0, 5): cap = units * UNIT_CAP served = sum(min(demand(h), cap) for h in HOURS) spill = total - served # overflow billed on the shared tier usd = units * UNIT_USD * 168 + spill * PAYG util = served / (cap * 168) if units else 0.0 marginal = (served - served_before) / (UNIT_CAP * 168) if units else 0.0 print(f"{units:5d} {served:15.0f} {util:11.0%} {marginal:13.0%} {usd:10,.0f}" f" {usd / total:10.2f}") best = min(best or (usd, units), (usd, units)) served_before = served print(f"Cheapest: {best[1]} unit(s) at {best[0]:,.0f} USD/week") # Break-even utilisation of one provisioned unit against pay-as-you-go: print(f"A unit pays for itself above {UNIT_USD / (UNIT_CAP * PAYG):.0%} utilisation") # Self-hosted: GPUs scale per hour between a floor and the peak; people cost is fixed. print("gpu_capacity people_usd gpu_hours weekly_usd usd_per_1M") for gpu_cap in (GPU_CAP, GPU_CAP / 2): # measured vs. pessimistic throughput for ops in (OPS_WEEK, OPS_WEEK * 2): gpu_hours = sum(max(MIN_GPUS, -(-demand(h) // gpu_cap)) for h in HOURS) usd = gpu_hours * GPU_USD + ops print(f"{gpu_cap:12.1f} {ops:10,.0f} {gpu_hours:9.0f} {usd:10,.0f}" f" {usd / total:10.2f}") ``` **Steps.** Save as `lab3_capacity.py` and run `python3 lab3_capacity.py`. **Expected output** (actual output of the run on 2026-10-18): ```text Weekly demand 651M tokens; peak 9.0M/hour Pay-as-you-go only: 3,906 USD/week = 6.00 per 1M units served_by_units utilisation marginal_util weekly_usd usd_per_1M 0 0 0% 0% 3,906 6.00 1 326 78% 78% 3,630 5.58 2 451 54% 30% 4,560 7.00 3 576 46% 30% 5,490 8.43 4 651 39% 18% 6,720 10.32 Cheapest: 1 unit(s) at 3,630 USD/week A unit pays for itself above 67% utilisation gpu_capacity people_usd gpu_hours weekly_usd usd_per_1M 3.0 1,500 386 3,044 4.68 3.0 3,000 386 4,544 6.98 1.5 1,500 536 3,644 5.60 1.5 3,000 536 5,144 7.90 ``` **Validation.** (a) The first unit runs at 78%, above the 67% break-even, and saves 7%; a second would serve only 30% of its capacity, so stop at one. (b) Four units, sized to the peak, cost 72% *more* than pay-as-you-go. (c) Break-even is 10.00 ÷ (2.5 × 6.00) = 67%. (d) Self-hosting wins only if throughput and people cost hold; halve one or double the other and it loses. Throughput depends on batching and the serving stack (vLLM's authors reported 2–4 times earlier systems' throughput),[^vllm] so measure it on your traffic. **Troubleshooting.** If many units look best, check the profile has quiet hours. **Teardown.** Delete the file. ### Use cases and scenarios All three are **illustrative**; the facts they use are sourced. | Scenario | Method | Trade-off | Success metric | |---|---|---|---| | A. Support assistant on a pay-per-token API | Cache the policy prefix; cap output and retrieved chunks; summarise long chats | Caching needs steady traffic; routing can lower resolution | Cost per resolved conversation, escalations included | | B. Shared GPU platform for open-weights models | `max(request, usage)` allocation; MIG for small models; batching; idle line | Isolation against density; people cost | Allocated plus idle equals the bill | | C. Fine-tuning to shorten prompts | 6ND training estimate; compare with prompt changes; price serving | Dedicated serving, depending on provider and base model | Cost per outcome, serving included | **A.** Resent history and the policy prompt dominate (Lab 2), then escalations. **B.** Small models waste a whole GPU unless partitioned ([allocation](#playbook-allocating-a-shared-model-endpoint-or-gpu-pool)). Judge efficiency by DCGM SM activity, the share of time multiprocessors had work (0.8 is "necessary, but not sufficient"), not by `nvidia-smi`, which counts time "during which one or more kernels was executing".[^dcgm] **C.** Try prompts first, the Foundation advises; for tuning it asks about data preparation ("often underestimated and sometimes exceeding the compute cost"), retraining frequency and serving: serving your own tuned model means "taking on the operational costs of inference infrastructure permanently".[^fin-tools] Flow: price the prompt tokens tuning would remove; add training (the [6ND estimate](#cost-capacity-and-sustainability), about 200 for an illustrative 8B run) per retrain, plus data preparation; then price serving. On Bedrock a customised model may need Provisioned Throughput ([playbook](#playbook-buying-provisioned-model-capacity));[^br-pt] one unit at Lab 3's illustrative 10.00 an hour costs 1,680 a week, about eight times that training run. **Pitfall:** approving the cheap training line and missing the permanent serving line. ## Measure, govern and pay ### Metrics and KPIs The Foundation lists cost per inference, cost per token, utilisation, anomaly cost, ROI and time to value.[^fin-ai-cat] Manage by the first row below. | KPI | Formula and data source | Pitfall | Cadence | |---|---|---|---| | Cost per successful outcome | (token + capacity + tool + review cost) ÷ successful outcomes; logs joined to outcomes | Counting attempts; omitting escalations | Weekly | | Cost per request, median and p95 | Request logs per feature | Averages hide long chats and loops | Weekly | | Attributed AI spend | Cost with owner metadata ÷ all AI cost | Forgetting SaaS and marketplaces | Monthly | | Provisioned utilisation and spillover | Tokens served by reserved capacity ÷ capacity; spilled share | 100% with heavy spillover: too little | Daily | | GPU SM activity | DCGM `PROF_SM_ACTIVE` per pool | `nvidia-smi` read as efficiency | Daily | | Forecast variance by driver | Actual − forecast by volume, tokens per outcome and price | Drivers that cancel out | Monthly | | Value ratio | Value per outcome ÷ cost per outcome, against the replaced process | Value nobody can audit | Quarterly | **Worked example (illustrative, from the [judgement call](#a-judgement-call-route-easy-conversations-to-a-cheaper-model)).** Per 100 conversations, (9.13 tokens + 73.00 escalations) ÷ 85.4 resolved = **0.96 per resolved conversation**. Tokens are 11% of it: a 20% token cut saves about 2%; one more escalation costs 5.00, almost three times as much. **Leading:** metadata coverage, cache-read share, p95 tokens per request, step-limit hits. **Lagging:** cost per successful outcome, forecast accuracy, value ratio. **Vanity trap:** "tokens saved" without the quality effect. **Data limits:** provider cost data arrives in buckets,[^oai-costs][^anth-admin] so real-time control needs gateway logs. ### Quality, security, privacy and compliance | Risk (likelihood / impact) | Signal | Mitigation | Owner | |---|---|---|---| | Unbounded consumption, "Denial of Wallet" (medium / high)[^owasp] | Spend rate per key far above baseline | OWASP's controls ([Bound the multipliers](#charter-and-principles)); Atlas: step and retry caps | Engineering | | Budget control fails open (medium / high) | Gateway database down; calls bypass the gateway | Quarterly cap tests; provider-side limits[^litellm][^anth-admin] | Platform | | Shared or leaked keys (medium / high) | Spend without owner metadata | A key per feature; rotation; revoke on retirement | Platform | | Prompt content in cost data (medium / medium) | Prompt text in the cost store | Counts and metadata only; retention agreed with [PrivacyOps](../privacyops/) | FinOps | | Residency raises price (high / low) | Regional endpoints | Price it in (October 2026): OpenAI +10% for regional processing on models released on or after 5 March 2026; Anthropic 1.1× for US-only inference on Claude 4.6 and later[^oai-tiers][^anth-pricing] | Finance | | Cost cuts lower quality (medium / high) | Escalations rise | Quality floor as a release gate | Product owner | Regulatory duties belong to [ModelOps](../modelops/) and [ComplianceOps](../complianceops/); their cost belongs to the use case. ### Cost, capacity and sustainability **Training cost, to an order of magnitude.** Training needs about *6 × parameters × tokens* FLOPs: Kaplan and colleagues write C = 6NBS, where "the factor of 6 accounts for the forward and backward passes".[^kaplan] Divide by peak FLOPs times model FLOPs utilisation (MFU), the observed share of peak throughput; PaLM reported 46.2%,[^palm] Meta 38–43% for Llama 3 405B, trained on 15.6T tokens with 3.8×10²⁵ FLOPs.[^llama3] _Example status: verified by execution on 2026-10-18 (Python 3.8.10); the price per GPU-hour is illustrative._ ```python """Order-of-magnitude training cost: FLOPs ~ 6 x params x tokens (Kaplan et al., 2020).""" def train_cost(params, tokens, peak_flops, mfu, usd_per_gpu_hour): flops = 6 * params * tokens gpu_hours = flops / (peak_flops * mfu) / 3600 return flops, gpu_hours, gpu_hours * usd_per_gpu_hour # Check against a published run: Llama 3 405B, 15.6T tokens, reported 3.8e25 FLOPs. print(f"Llama 3 405B: {6 * 405e9 * 15.6e12:.2e} FLOPs by the 6ND rule") # Illustrative full fine-tune: 8B params, 2B tokens, H100 SXM at 40% MFU, 3 USD/hour. DENSE_BF16 = 989e12 # H100 SXM dense BF16 peak; 1,979 TFLOPS assumes sparsity for label, peak in (("dense peak", DENSE_BF16), ("sparse peak (wrong)", 2 * DENSE_BF16)): flops, hours, usd = train_cost(8e9, 2e9, peak, 0.40, 3.00) print(f"{label:20s} {flops:.2e} FLOPs {hours:6.1f} GPU-hours {usd:6.0f} USD") ``` ```text Llama 3 405B: 3.79e+25 FLOPs by the 6ND rule dense peak 9.60e+19 FLOPs 67.4 GPU-hours 202 USD sparse peak (wrong) 9.60e+19 FLOPs 33.7 GPU-hours 101 USD ``` The rule reproduces Meta's figure. The second line is a common error: NVIDIA lists 1,979 BF16 teraFLOPS for the H100 SXM "With sparsity", and its architecture post pairs each dense figure with a sparse one twice as large;[^nv-h100] using it halves the estimate. The formula omits evaluations, restarts (Meta recorded 466 job interruptions in 54 days, 419 unexpected),[^llama3] data preparation and people. **Capacity.** GPU capacity is often bought ahead: AWS Capacity Blocks reserve GPU instances up to eight weeks ahead, and SageMaker AI Savings Plans are their own commitment type.[^aws-gpu] Interruptible capacity suits training only with checkpointing ([MLOps](../mlops/#operating-training-jobs)). In serving, batching sets throughput per GPU, so optimising only for single-request latency can raise cost.[^fin-tools][^vllm] **Retrieval and multimodal inputs.** Retrieval-augmented generation (RAG) adds retrieved text to every call: input grows with chunks per query × chunk size. Retrieving snippets "dramatically reduces" tokens compared with whole documents, but the Foundation calls RAG infrastructure costs "frequently underestimated in initial proposals": embeddings for every document, "re-done when documents change" or the embedding model is upgraded, "vector database storage and query costs", and pipeline upkeep.[^fin-tools] Give the vector store its own idle line, price re-indexing as a recurring cost driven by document churn, and count retrieved tokens per outcome. An image in the history is "silently re-billed" on every later turn.[^fin-token] **Sustainability.** Energy is layer zero of the Tokenomics Foundation's seven-layer stack, whose "lower three layers set what AI costs".[^tok-stack] Carbon accounting belongs to [GreenOps](../greenops/); idle reserved capacity wastes energy as well as money. ### Assurance of the operating model Monthly: request-log cost reconciles with every provider report (Lab 1), and attribution covers every channel. Quarterly: a test key pushed past its budget and an agent past its step limit both stop, and last quarter's capacity and funding decisions are checked against results. ## When things go wrong ### Common failures and troubleshooting | Symptom | Likely cause | Diagnosis | Fix and prevention | |---|---|---|---| | Request-log cost above the invoice | Cached or reasoning tokens counted twice (one parser for two conventions) | Usage fields by token type | A parser per provider convention (Lab 1) | | Invoice above request-log cost | Unpriced cache writes, separate thinking field, tier, long-context or regional prices, price change | Invoice lines by SKU against the rate card | Price every line type; dated rate card | | Overnight spend spike | Agent loop or retry storm | Spend by key and root request | Budget to zero; step and retry caps | | Cache savings vanish | Prefix changed, or traffic split across models or Regions (cross-Region inference "may lead to increased cache writes")[^br-cache] | Cache fields in responses | Stable prefix; pinned routing | | Cost with no visible answer | `max_output_tokens` reached during reasoning; you "could incur costs for input and reasoning tokens without receiving a visible response"[^oai-reasoning] | Response marked incomplete | Raise the cap or lower reasoning effort | | Provisioned capacity idle | Sized to the peak; traffic or model moved | Utilisation by hour | Shrink at term end; buy only units above break-even | | Cheaper model, higher total cost | More retries and escalations | Outcome rates before and after | Revert; quality floor as a gate | ### Worked runbook: an agent loop burns a month of budget overnight (illustrative) A research agent went live on Thursday. At 02:10 on Saturday the gateway alerts: spend at 12 times baseline, 70% of the month's budget gone, nobody using the product. ::: figure Runbook: containing and fixing a runaway agent ```mermaid %%{init: {"flowchart": {"wrappingWidth": 180, "nodeSpacing": 26, "rankSpacing": 32}}}%% flowchart TB A["Alert: feature spend rate 12 times baseline"] --> B["Triage: which key, agent run and model; calls per request"] B --> C{{"Is the spend serving real users?"}} C -->|no| D["Contain: set the feature budget to zero; revoke or rotate the feature's key"] C -->|yes| E["Throttle and page the product owner"] D --> F["Verify in gateway logs that spend stopped or fell to the throttled rate"] E --> F F --> G["Root cause: no step limit; a failing tool triggered retries with full context"] G --> H["Fix: step, retry and output caps; circuit breaker; test the limit"] H --> I["Review: incident cost, re-forecast, add a control test"] classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029 classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029 classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 class A,G risk class C decision class B,D,E,F,H,I core ``` Text equivalent: An alert reports spend at 12 times baseline. Triage finds the key, run, model and calls per request. With no real users, zero the feature budget and revoke or rotate its key; otherwise throttle and page the product owner. Verify that spend stopped or fell to the throttled rate, fix the root cause (no step limit) with caps and a circuit breaker, test them, and review the cost. ::: 1. **Triage (10 minutes):** one run made 3,000 calls, each resending 40,000 tokens: the T(∞) class.[^tok-bigt] 2. **Contain (5 minutes):** a provider-side spend limit is the backstop if the gateway fails. 3. **Verify** in gateway logs; provider cost data arrives in buckets.[^anth-admin][^oai-costs] 4. **Root cause:** OWASP's "unbounded consumption".[^owasp] 5. **Fix and review:** back-off on retries; drive a test agent into the limit; add it to quarterly control tests. ### Anti-patterns and myths - **One API key for the company:** cheap to start, impossible to allocate later. - **Comparing price per token across models:** tokenizers, reasoning and output length differ;[^anth-pricing][^oai-reasoning] compare cost per successful outcome. - **Cost targets without quality floors,** and **pilots without caps or end dates**. - **Half-truth: AI is "simply another bucket of spend".** A respondent quoted in the State of FinOps 2026 (labelled Financial Services) wrote that AI "is simply another bucket of spend that requires the same discipline and governance as any other technology"; the report concludes that "AI is not rewriting FinOps, but it is testing how flexible the operational framework can become".[^sof-2026] The discipline carries over; the unit economics do not. ## Deciding and adopting ### When to use and when not to use **Build the practice** when AI spend is material or growing fast, several teams or providers are involved, capacity commitments are on the table, agents run in production, or leadership asks whether AI pays. **Do not** build a separate practice for small AI spend with one provider: add AI lines and caps to existing FinOps reviews (not all AI spend needs its own Scope, the Foundation notes).[^fin-ai-cat] **Hidden costs:** metadata discipline, evaluation sets, review time. ### Decision framework How a workload consumes models decides most of its cost. The tree is this Atlas's proposal; thresholds come from Lab 3. ::: figure Choosing how a workload consumes models ```mermaid %%{init: {"flowchart": {"wrappingWidth": 160, "nodeSpacing": 24, "rankSpacing": 34}}}%% flowchart TB Q0{{"Does the task need a model call at all?"}} -->|no| R0["Rules, search or a cached answer"] Q0 -->|yes| Q1{{"Is an answer needed within seconds?"}} Q1 -->|no| R1["Batch or flex tier"] Q1 -->|yes| Q2{{"Steady demand, and the model (or a model-independent pool) kept for the term?"}} Q2 -->|no| R2["Pay-as-you-go with caching, caps and routing"] Q2 -->|yes| Q3{{"Next unit above break-even utilisation, or a latency guarantee worth its premium?"}} Q3 -->|no| R2 Q3 -->|yes| Q4{{"An open-weights model meets the quality floor and you can run GPUs?"}} Q4 -->|no| R3["Provisioned units, each above break-even or worth its latency premium; bursts spill over"] Q4 -->|yes| R4["Compare self-hosting with provisioned at measured throughput"] classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029 classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 class Q0,Q1,Q2,Q3,Q4 decision class R0,R1,R2,R3,R4 core ``` Text equivalent: No model call needed: rules, search or a cached answer. No answer needed within seconds: batch or flex. Without steady demand and a model (or model-independent pool) kept for the term, or when the next provisioned unit would run below break-even and no latency guarantee is worth its premium: pay-as-you-go with caching, caps and routing. Otherwise buy provisioned units one by one (each above break-even or worth its latency premium), with spillover, unless an open-weights model meets the quality floor and you run GPUs; then compare self-hosting with provisioned capacity at measured throughput. ::: **Self-hosting needs all of these (Atlas checklist):** utilisation above the managed break-even; an open-weights model that passes *your* evaluations; a team already running GPUs; and one of the Foundation's reasons to own a model: privacy, a specialised domain or "very high inference volumes".[^fin-tools] The tree puts self-hosting behind the managed break-even on purpose (Atlas reasoning): it adds fixed people cost, a GPU floor and throughput risk (Lab 3). ### Maturity model **This Atlas's proposal**; the Foundation assesses maturity per capability, not for AI. | Level | Observable behaviour | Evidence to move on | |---|---|---| | 0. Unmanaged | AI spend found on invoices; shared keys | Every AI channel inventoried | | 1. Visible | AI cost by provider and account; caps on pilots | All AI channels in one monthly report | | 2. Attributed | Most AI cost tagged by team and feature; reconciled monthly | Agreed attributed share held for a quarter | | 3. Outcome-managed | Cost per successful outcome per feature; quality floors; tested caps | Every production feature has a unit cost and an evaluation | | 4. Governed | Stage-gated funding; capacity above break-even; driver-based forecasts | Decisions reviewed against outcomes quarterly | ### Adoption roadmap: 30, 60 and 90 days | Window | Actions | Milestone | |---|---|---| | Days 1–30 (FinOps leads) | Inventory every AI channel; a key or profile per feature; caps on every pilot; the metadata contract | One report shows all AI spend; no uncapped pilot | | Days 31–60 (platform leads) | Gateway or instrumentation on every call; rate card; reconciliation; one outcome event per feature | Reconciled report; first cost per outcome | | Days 61–90 (product and council) | Weekly unit-cost reviews; caching and tiers for the top two features; capacity analysis; first council review | Two features with unit-cost targets; one pilot stopped or re-scoped | **Quick wins:** batch tiers, stable prefixes, output and step caps, closed unowned keys. ## Toolkit ### Checklists **Quick start** - [ ] Every AI channel listed: cloud AI services, direct APIs, marketplaces, SaaS credits, GPUs - [ ] Every key has an owner, a feature and a budget; unowned keys have a removal date - [ ] Every pilot has a time box, a cost cap and a value hypothesis **Launch readiness for an AI feature** - [ ] Outcome event emitted; quality floor and evaluation set agreed; cost per outcome estimated - [ ] Output, step, retry and input-size limits configured and tested - [ ] Owner metadata on every call; budget alert at 80% and a defined action at 100% - [ ] Stable, cacheable prompt prefix; non-urgent paths on batch or flex tiers **Capacity purchase preflight** ([playbook](#playbook-buying-provisioned-model-capacity)) - [ ] Each unit above break-even on four weeks of hourly data; load test with the real token mix - [ ] Term within the model's life; overflow configured; Azure deployment before reservation; preparer is not approver ### Templates **AI use-case funding request (Atlas template)** ```text Use case ID / name: UC-____ / ______________________ Owner (product) / engineer: ______________ / ______________ Outcome unit: e.g. "support case resolved without a person" Value per outcome: ____ (method and source: ______________________) Baseline: cost per outcome of the current process: ____ Quality floor: metric ____ >= ____ on evaluation set ____ Candidate models and tiers: 1. ________ 2. ________ (approved by ModelOps: Y/N) Estimated cost per outcome: model 1: ____ model 2: ____ (tokens/call x calls/outcome) Expected volume: ____ outcomes per month; growth assumption ____ Stage requested: proof of concept / proof of value / scale Cost cap and time box: ____ per month until ____ (YYYY-MM-DD) Kill criteria: stop if ______________________________________ Controls: output cap __ / step limit __ / budget __ / alert at 80% Data classification: ____ ; residency needs: ____ ; prompts logged: Y/N Decision: fund / fund with changes / decline Date: __________ ``` **Provisioned capacity decision record** ```text Decision ID / date: CAP-____ / YYYY-MM-DD Provider, model, region: ______________________________________________ Unit and quantity: ____ units (PTU / model units / reserved TPM) Term and renewal: ____ months; auto-renew: Y/N; end date: ________ Demand evidence: weeks of data __; floor __ tokens/h; peak __ tokens/h Load test result: __ tokens/h per unit at the real input/output mix Normalised price: __ per 1M tokens at __% expected utilisation Last unit's own utilisation: __% (buy only if above break-even) Pay-as-you-go comparison: __ per 1M tokens; break-even utilisation __% Overflow plan: spillover / overflow tier / gateway fallback Model-change risk: model can / cannot be changed during the term Waste lines reported: idle allocated / unallocated Prepared by / approved by: ______________ / ______________ Review date and result: ________ ; utilisation achieved __% ``` **AI cost forecast (Atlas template).** It applies the Foundation's "quantity * rate = cost" approach per driver; the Foundation's outcome is forecasts "within 5% of actuals per month".[^fin-forecast] Use mean, not median, tokens and take the range from low and high scenarios. ```text Feature / use case: ____________ Period: ______ Rate card version: ______ Driver Low Expected High Source Successful outcomes _____ _____ _____ product plan Requests per successful outcome _____ _____ _____ logs, last 4 weeks Mean tokens per request, by type _____ _____ _____ logs (mean, not median) Price per 1M tokens, by type _____ _____ _____ rate card; announced changes = Token cost (product / 1M) _____ _____ _____ + Provisioned units x hours _____ _____ _____ capacity decision records + Retrieval, tools, vector store _____ _____ _____ invoices; re-index plan + People (escalations x cost) _____ _____ _____ escalation rate = Forecast _____ _____ _____ Re-forecast when: a driver moves more than __% in a week; a new model, price or tier Last period's variance: volume __ tokens per outcome __ price __ capacity __ ``` **Monthly AI cost review agenda (45 minutes)** ```text 1. Reconciliation and coverage (5 min): gaps to invoices; unowned spend and owners 2. Unit costs (15 min): cost per successful outcome by feature; p95 outliers; quality 3. Capacity (10 min): utilisation, spillover, idle and unallocated; expiries in 60 days 4. Forecast (10 min): variance by volume, tokens per outcome and price; re-forecast 5. Decisions and owners (5 min): actions with dates; items for the council ``` ### Deep comparisons Consumption options compared (October 2026): | Option | Pays for | Best when | Main risk | |---|---|---|---| | Pay-as-you-go | Tokens used | Variable or new traffic | No latency guarantee; price changes | | Batch | Discounted tokens, results within 24 hours[^oai-tiers] | Backlogs, evaluations, enrichment | Latency; tool calling can be limited[^br-batch] | | Flex | Discounted tokens, slower, occasionally unavailable[^oai-tiers][^br-tiers] | Low-priority interactive work | Retries | | Priority or fast | A premium (75% on Bedrock's listed models)[^br-pricing] | Latency-critical paths, no reservation | Premium on all routed traffic | | Provisioned | Capacity per hour or month[^az-ptu][^br-pt] | Steady traffic above break-even, or a latency SLO | Idle capacity; on Bedrock, tied to one model for the term[^fin-capacity] | | Self-hosted | GPU-hours and people | Very high steady volume, open-weights model fits | Throughput, operations, quality | ### FAQ **Do we need a separate FinOps-for-AI team?** Usually not: survey respondents run lean central teams with a wider scope.[^sof-2026] Add AI skills and an investment council; dedicate an analyst once AI is a top line. **Should we charge AI costs back?** Show back by feature for a quarter first; charge back only when attribution is reliable and the rules for shared and idle capacity are accepted ([FinOps](../finops/)). **Why does our token log not match the invoice?** A log above the invoice usually counts cached or reasoning tokens twice; an invoice above it has unpriced lines or a price change ([failures](#common-failures-and-troubleshooting), Lab 1).[^otel-genai][^anth-cache][^gem-thinking] **Will provisioned capacity give us a discount?** Only for units above break-even utilisation and, where reservations are model-specific (Bedrock), only while you keep the model ([playbook](#playbook-buying-provisioned-model-capacity)). **How do we treat AI inside SaaS tools?** As SaaS spend with an AI meter: budgets, burn by model and value per active user. Since 1 June 2026 every GitHub Copilot plan includes "GitHub AI Credits", with budgets at enterprise, cost-centre and user level and a choice to cap spend or allow more usage at published rates;[^gh-copilot] the licence is no longer the whole cost. ## Going further ### Advanced topics and emerging direction - **Billing data for AI.** FOCUS draft model and token properties (reasoning tokens still an open request)[^focus-ai] are work the Tokenomics Foundation says it will help fund.[^lf-tokenomics] - **Price direction.** Epoch AI found the price of a fixed benchmark level fell 9 to 900 times a year (median 50) up to early 2025, excluding reasoning models, which generate far more tokens.[^epoch] The Linux Foundation asserted in June 2026, without data, that per-token costs "have leveled off" and "new model token prices are rising".[^lf-tokenomics] Both can hold: a fixed capability gets cheaper while the newest costs more, so forecast them separately. - **Routing research.** FrugalGPT reported "up to 98% cost reduction" and RouteLLM cuts of "over 2 times in certain cases", on the authors' 2023–2024 benchmarks.[^frugal][^routellm] In the Tokenomics Foundation's release-candidate survey, 86% were evaluating or using a router.[^tok-sot] Treat published savings as hypotheses to test. ### Learning path | Stage | What to do | |---|---| | Prerequisites | [FinOps](../finops/) basics; one LLM API's pricing and usage fields | | One day | The first two parts; Labs 1 and 2 | | One week | Metadata and cost per successful outcome for a real feature; Lab 3 on your traffic | | Thirty days | A reconciled pipeline, tested caps, a monthly review, a capacity decision record | | Specialise | [LLMOps](../llmops/), [MLOps](../mlops/), [KubernetesOps](../kubernetesops/), [ProcurementOps](../procurementops/), [GreenOps](../greenops/) | **Routes by role.** *Beginners:* the first part, Lab 1. *Engineers:* labs, reference implementation, failures. *FinOps and finance:* operating model, metrics, capacity playbook, templates. *Leaders:* problem, decision tree, judgement call. ## Wrap-up ### Executive recap and cheat sheet - **Decide by cost per successful outcome,** people included. - **Bound the multipliers** (calls, agent depth, retrieved context, output, retries) before negotiating rates. - **Tiers before commitments:** batch and flex, then caching, then provisioned units that each clear break-even. - **Test every control's preconditions:** gateway databases, direct keys, provider limits. - **Re-evaluate on change:** a new model, price or tokenizer is a cost event. Fund in stages with kill criteria. ### A judgement call: route easy conversations to a cheaper model? **Situation (illustrative).** A support assistant runs four-turn conversations on a large model: a 12,000-token cached prefix, 1,500 new input and 400 output tokens per turn, earlier turns resent from cache. About 70% of conversations are "easy"; a small model costs a tenth as much per token. An escalation to a person costs 5.00; the large model escalates 8% of easy and 30% of hard conversations, and a pilot suggests 11% for the small model on easy ones. **Option A, route:** token cost falls by over 60%. **Option B, keep one model:** simpler, quality kept. _Example status: verified by execution on 2026-10-18 (Python 3.8.10). All numbers are illustrative._ ```python """Judgement call: route easy conversations to a small model, or keep one large model?""" TURNS, PREFIX, NEW_IN, OUT = 4, 12_000, 1_500, 400 # tokens per turn MODELS = {"large": (4.00, 20.00), "small": (0.40, 2.00)} # USD per 1M in/out READ, WRITE = 0.10, 1.25 # cache read and write multipliers ROUTER_USD, HUMAN_USD = 0.0004, 5.00 # per conversation (illustrative) EASY, HARD_FAIL = 0.70, 0.30 # easy share; hard escalation rate def ai_usd(model): """One conversation; each turn resends the history, read from cache after turn 1.""" p_in, p_out = MODELS[model] usd = 0.0 for k in range(1, TURNS + 1): history = (k - 1) * (NEW_IN + OUT) # earlier turns, resent cached = PREFIX + max(0, history - OUT) # prefix (warm) + earlier prompts written = PREFIX + history + NEW_IN - cached # last reply + new input usd += (cached * READ + written * WRITE) * p_in + OUT * p_out return usd / 1e6 def per_100(route, easy_fail_large=0.08, easy_fail_small=0.11): easy_model = "small" if route else "large" ai = 100 * (EASY * ai_usd(easy_model) + (1 - EASY) * ai_usd("large")) ai += 100 * ROUTER_USD if route else 0.0 easy_fail = easy_fail_small if route else easy_fail_large escalations = 100 * (EASY * easy_fail + (1 - EASY) * HARD_FAIL) return ai, escalations * HUMAN_USD for label, route in (("one large model", False), ("route easy to small", True)): ai, people = per_100(route) print(f"{label:20s} AI {ai:5.2f} escalations {people:6.2f}" f" total {ai + people:6.2f}") saving = per_100(False)[0] - per_100(True)[0] points = 100 * saving / HUMAN_USD / (100 * EASY) print(f"AI saving per 100 conversations: {saving:.2f} USD") print(f"Routing wins only if easy-case escalations rise by less than {points:.1f} points") ``` ```text one large model AI 9.13 escalations 73.00 total 82.13 route easy to small AI 3.42 escalations 83.50 total 86.92 AI saving per 100 conversations: 5.71 USD Routing wins only if easy-case escalations rise by less than 1.6 points ``` **Decision: not yet.** Routing cuts token cost by 63% but raises total cost by 6%: three extra escalations per 100 easy conversations cost more than all the tokens saved. It wins only if the small model's easy-case escalation rate is within about 1.6 points of the large model's, so the next step is a better evaluation (a larger sample, a sharper "easy", a tuned prompt), not a launch. **Uncertainty.** The 3-point gap comes from a small pilot whose confidence interval may include 1.6. Cheaper human handling or a better router favour routing; low volume does not: routing "saves money until it breaks your cache".[^tok-what] **Why the attractive option misleads:** it optimises the visible line (tokens, 11% of cost) at the expense of the dominant one (people). ### Learning objectives and mastery check You should now be able to **explain** FinOps for AI against FinOps, LLMOps and tokenomics; **calculate** call and outcome costs; **design** metadata, budgets and caps; **decide** how a workload consumes models; and **evaluate** savings against quality. ::: answer Q1. Two providers each report 15,000 input tokens and 12,000 cached tokens. Are the requests the same size? No. Where input includes cached tokens (OpenTelemetry, OpenAI), it sent 15,000, 3,000 not read from cache (uncached or cache writes); where `input_tokens` counts only tokens after the last cache breakpoint (Anthropic), it sent 27,000.[^otel-genai][^anth-cache] ::: ::: answer Q2. Conversations grew from 10 to 20 turns. Finance expects input cost per conversation to double. Right? No, it roughly triples: resent history makes input grow with the square of the turns (Lab 2: 54,000 to 158,000 tokens). Caching lowers the level; summarising changes the shape. ::: ::: answer Q3. A unit costs 12 per hour for 3M tokens per hour; pay-as-you-go costs 5 per million. Buy it for traffic filling 60%? Break-even is 12 ÷ (3 × 5) = 80%, so not on price; only if its latency guarantee is worth the difference.[^fin-capacity] ::: ::: answer Q4. A GPU pool shows 95% in `nvidia-smi`, yet cost per million tokens is double the estimate. Why? `nvidia-smi` counts time when any kernel ran, not how busy the GPU was. Check DCGM SM activity (below 0.5 suggests ineffective use),[^dcgm] batch size and small models on whole GPUs. ::: ::: answer Q5. Every team has a gateway budget, yet one overspent by 40%. Name two ways that happens. The gateway's budgets fail open without its database (or trust a stale Redis counter),[^litellm] or the team used a provider key outside the gateway. Prevent both with provider-side limits and quarterly control tests. ::: ::: answer Q6. Leadership wants one number showing that AI pays. What do you give? Cost per successful outcome beside value per outcome and the replaced process's cost, per use case; a portfolio ROI mixes pilots with mature features. The Foundation suggests a common ROI template.[^fin-ai-cat] ::: ### Conclusion AI did not change the economics of FinOps: price times quantity still makes the bill. What changed is where the quantity comes from: design choices that multiply quietly (resent history, retrieved context, agent calls, hidden reasoning, idle reservations), made by engineers and product owners in an afternoon. If you remember one thing: **measure the cost of each successful outcome, people included, and bound the multipliers before you buy capacity.** ## Sources and further reading FinOps Foundation material (CC BY 4.0) is quoted briefly with attribution. Evidence ledger: `research/finops-for-ai-ledger.md`. **Re-check quarterly and on provider releases:** prices, tier factors, cache and residency multipliers, tokenizers, capacity terms, cost-API scope, LiteLLM budgets, the FOCUS draft. [^fin-ai-cat]: [FinOps for AI – FinOps Framework Technology Category](https://www.finops.org/framework/technology-categories/ai/) and [Technology Categories](https://www.finops.org/framework/technology-categories/) — FinOps Foundation, undated pages. Checked 2026-10-17. [^fin-ai-value]: [AI Value](https://www.finops.org/topic/ai-value/) (the old FinOps-for-AI topic URL redirects here) — FinOps Foundation. Checked 2026-10-17. [^fin-cert]: [FinOps Certified: AI Value](https://learn.finops.org/path/certified-finops-for-ai) — FinOps Foundation learning platform, as of 2026-10-17. Checked 2026-10-17. [^fin-ai-overview]: [FinOps for AI Overview](https://www.finops.org/wg/finops-for-ai-overview/) — FinOps Foundation working group, page states last updated 2026-02-17. Checked 2026-10-17. [^fin-token]: [GenAI FinOps: How Token Pricing Really Works](https://www.finops.org/wg/genai-finops-how-token-pricing-really-works/) — FinOps Foundation working group, last updated 2026-05-29. Checked 2026-10-17. [^fin-capacity]: [GenAI Capacity Options](https://www.finops.org/wg/genai-capacity-options/) — FinOps Foundation working group, last updated 2026-02-04. Checked 2026-10-17. [^fin-tools]: [FinOps for AI: Tools & Services Considerations](https://www.finops.org/wg/finops-for-ai-tools-services-considerations/) — FinOps Foundation working group, last updated 2026-04-23. Checked 2026-10-18. [^fin-choosing]: [Choosing an AI Approach and Infrastructure Strategy](https://www.finops.org/wg/choosing-an-ai-approach-and-infrastructure-strategy/) — FinOps Foundation working group, last updated 2025-05-22. Checked 2026-10-17. [^fin-forecast]: [How to Forecast AI Services Costs in Cloud](https://www.finops.org/wg/how-to-forecast-ai-services-costs-in-cloud/) — FinOps Foundation working group, last updated 2025-07-07. Checked 2026-10-18. [^sof-2026]: [State of FinOps 2026](https://data.finops.org/) — FinOps Foundation, 2026 (self-selected practitioner survey). Checked 2026-10-17. [^lf-tokenomics]: [Linux Foundation Announces the Intent to Launch the Tokenomics Foundation to Establish Open Standards for AI Cost Management](https://www.linuxfoundation.org/press/linux-foundation-announces-the-intent-to-launch-the-tokenomics-foundation-to-establish-open-standards-for-ai-cost-management) — The Linux Foundation, 2026-06-03. Checked 2026-10-17. [^tok-about]: [About the Tokenomics Foundation](https://www.tokeneconomics.com/about/) — Tokenomics Foundation, as of 2026-10-17. Checked 2026-10-17. [^tok-what]: [What Is AI Tokenomics?](https://www.tokeneconomics.com/what-is-ai-tokenomics/) — Tokenomics Foundation, draft definition v0.5.2 dated 2026-09-01. Checked 2026-10-17. [^tok-sot]: [State of Tokenomics](https://www.tokeneconomics.com/state-of-tokenomics/) — Tokenomics Foundation, release candidate 1.0, 2026-09-23 (472 responses; sampling not described). Checked 2026-10-17. [^tok-bigt]: [Big-T Notation](https://www.tokeneconomics.com/projects/big-t-notation/) — Dan Neff (Adobe), published by the Tokenomics Foundation, 2026. Checked 2026-10-17. [^tok-stack]: [The Seven-Layer Tokenomics Stack](https://www.tokeneconomics.com/projects/the-seven-layer-tokenomics-stack/) — Tokenomics Foundation, 2026. Checked 2026-10-17. [^focus-ai]: FOCUS [releases](https://github.com/FinOps-Open-Cost-and-Usage-Spec/FOCUS_Spec/releases) (v1.4, latest, 2026-06-04); working-draft pull requests [#2442, model identity](https://github.com/FinOps-Open-Cost-and-Usage-Spec/FOCUS_Spec/pull/2442) (merged 2026-07-20) and [#2606, token costs](https://github.com/FinOps-Open-Cost-and-Usage-Spec/FOCUS_Spec/pull/2606) (merged 2026-10-02); [request #2487, reasoning tokens](https://github.com/FinOps-Open-Cost-and-Usage-Spec/FOCUS_Spec/issues/2487) (open) — FOCUS project on GitHub. Checked 2026-10-17. [^otel-genai]: [OpenTelemetry GenAI semantic conventions repository](https://github.com/open-telemetry/semantic-conventions-genai) (status Development) — OpenTelemetry, main branch as of 2026-10-17; the [former page](https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-metrics/) records the move. Checked 2026-10-17. [^owasp]: [LLM10:2025 Unbounded Consumption](https://genai.owasp.org/llmrisk/llm102025-unbounded-consumption/) — OWASP GenAI Security Project, 2025 edition. Checked 2026-10-17. [^anth-pricing]: [Pricing](https://platform.claude.com/docs/en/about-claude/pricing) — Anthropic, Claude API documentation, as of 2026-10-18. Checked 2026-10-18. [^anth-cache]: [Prompt caching](https://platform.claude.com/docs/en/build-with-claude/prompt-caching) — Anthropic, Claude API documentation. Checked 2026-10-17. [^anth-admin]: [Usage and Cost API](https://platform.claude.com/docs/en/manage-claude/usage-cost-api) and [Workspaces](https://platform.claude.com/docs/en/manage-claude/workspaces) — Anthropic, Claude API documentation. Checked 2026-10-18. [^oai-cache]: [Prompt caching](https://developers.openai.com/api/docs/guides/prompt-caching) — OpenAI API documentation. Checked 2026-10-17. [^oai-tiers]: [Batch API](https://developers.openai.com/api/docs/guides/batch), [Flex processing](https://developers.openai.com/api/docs/guides/flex-processing) and [Pricing](https://developers.openai.com/api/docs/pricing) — OpenAI API documentation, as of 2026-10-18. Checked 2026-10-18. [^oai-reasoning]: [Reasoning models](https://developers.openai.com/api/docs/guides/reasoning) — OpenAI API documentation. Checked 2026-10-17. [^anth-thinking]: [Extended thinking](https://platform.claude.com/docs/en/build-with-claude/extended-thinking) — Anthropic, Claude API documentation. Checked 2026-10-17. [^gem-thinking]: [Gemini thinking](https://ai.google.dev/gemini-api/docs/thinking) — Google, Gemini API documentation. Checked 2026-10-17. [^oai-costs]: [Costs (GET /organization/costs)](https://developers.openai.com/api/reference/resources/admin/subresources/organization/subresources/usage/methods/costs) — OpenAI API reference. Checked 2026-10-17. [^br-tiers]: [Service tiers for optimizing performance and cost](https://docs.aws.amazon.com/bedrock/latest/userguide/service-tiers-inference.html) — Amazon Bedrock User Guide, AWS. Checked 2026-10-17. [^br-pricing]: [Amazon Bedrock pricing](https://aws.amazon.com/bedrock/pricing/) — Amazon Web Services, as of 2026-10-18. Checked 2026-10-18. [^br-pt]: [Increase model invocation capacity with Provisioned Throughput](https://docs.aws.amazon.com/bedrock/latest/userguide/prov-throughput.html), [Set up inference for a custom model](https://docs.aws.amazon.com/bedrock/latest/userguide/model-customization-use.html) and [Deploy a custom model for on-demand inference](https://docs.aws.amazon.com/bedrock/latest/userguide/deploy-custom-model-on-demand.html) — Amazon Bedrock User Guide, AWS. Checked 2026-10-17. [^br-cache]: [Prompt caching for faster model inference](https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-caching.html) — Amazon Bedrock User Guide, AWS. Checked 2026-10-17. [^br-batch]: [Process multiple prompts with batch inference](https://docs.aws.amazon.com/bedrock/latest/userguide/batch-inference.html) — Amazon Bedrock User Guide, AWS. Checked 2026-10-17. [^br-aip]: [Create an application inference profile](https://docs.aws.amazon.com/bedrock/latest/userguide/inference-profiles-create.html) and [Prerequisites for inference profiles](https://docs.aws.amazon.com/bedrock/latest/userguide/inference-profiles-prereq.html) — Amazon Bedrock User Guide; [create-inference-profile](https://docs.aws.amazon.com/cli/latest/reference/bedrock/create-inference-profile.html) and [get-inference-profile](https://docs.aws.amazon.com/cli/latest/reference/bedrock/get-inference-profile.html) — AWS CLI Command Reference. Checked 2026-10-17. [^az-ptu]: [Provisioned throughput billing](https://learn.microsoft.com/en-us/azure/foundry/openai/concepts/provisioned-throughput-billing) (ms.date 2026-05-22) and [What is provisioned throughput](https://learn.microsoft.com/en-us/azure/foundry/openai/concepts/provisioned-throughput) (ms.date 2026-07-15) — Microsoft Learn. Checked 2026-10-17. [^az-spill]: [Manage traffic with spillover for provisioned deployments](https://learn.microsoft.com/en-us/azure/foundry/openai/how-to/spillover-traffic-management) — Microsoft Learn, ms.date 2026-06-18. Checked 2026-10-17. [^gcp-genai]: [Provisioned Throughput overview](https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/provisioned-throughput) and [Batch inference](https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/capabilities/batch-inference) — Google Cloud documentation (Gemini Enterprise Agent Platform), last updated 2026-10-15. Checked 2026-10-18. [^aws-gpu]: [Capacity Blocks for ML](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-capacity-blocks.html) and [Savings Plans types](https://docs.aws.amazon.com/savingsplans/latest/userguide/plan-types.html) — AWS documentation. Checked 2026-10-17. [^gh-copilot]: [What changed with billing](https://docs.github.com/en/copilot/reference/copilot-billing/request-based-billing-legacy/what-changed-with-billing) — GitHub Docs; [GitHub Copilot is moving to usage-based billing](https://github.blog/changelog/2026/04/27/github-copilot-is-moving-to-usage-based-billing/) — GitHub Changelog, 2026-04-27. Checked 2026-10-18. [^k8s-gpu]: [Schedule GPUs](https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus/) and [Dynamic Resource Allocation](https://kubernetes.io/docs/concepts/scheduling-eviction/dynamic-resource-allocation/) — Kubernetes documentation. Checked 2026-10-17. [^opencost]: [OpenCost Specification](https://opencost.io/docs/specification/) — OpenCost project. Checked 2026-10-17. [^nv-sharing]: [Time-Slicing GPUs in Kubernetes](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-sharing.html) — NVIDIA GPU Operator documentation. Checked 2026-10-17. [^dcgm]: [DCGM Feature Overview: profiling metrics](https://docs.nvidia.com/datacenter/dcgm/latest/user-guide/feature-overview.html) and [nvidia-smi documentation](https://docs.nvidia.com/deploy/nvidia-smi/index.html) — NVIDIA. Checked 2026-10-17. [^nv-h100]: [NVIDIA H100 Tensor Core GPU (product specifications)](https://www.nvidia.com/en-us/data-center/h100/) — NVIDIA; [NVIDIA Hopper Architecture In-Depth](https://developer.nvidia.com/blog/nvidia-hopper-architecture-in-depth/) — NVIDIA Technical Blog, 2022. Checked 2026-10-17. [^kaplan]: [Scaling Laws for Neural Language Models](https://arxiv.org/abs/2001.08361) — Jared Kaplan et al., arXiv, 2020. Checked 2026-10-17. [^palm]: [PaLM: Scaling Language Modeling with Pathways](https://arxiv.org/abs/2204.02311) — Aakanksha Chowdhery et al., arXiv, 2022. Checked 2026-10-17. [^llama3]: [The Llama 3 Herd of Models](https://arxiv.org/abs/2407.21783) — Llama Team, AI @ Meta, arXiv, 2024. Checked 2026-10-17. [^vllm]: [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://arxiv.org/abs/2309.06180) — Woosuk Kwon et al., arXiv, 2023. Checked 2026-10-17. [^frugal]: [FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance](https://arxiv.org/abs/2305.05176) — Lingjiao Chen, Matei Zaharia and James Zou, arXiv, 2023. Checked 2026-10-17. [^routellm]: [RouteLLM: Learning to Route LLMs with Preference Data](https://arxiv.org/abs/2406.18665) — Isaac Ong et al., arXiv, 2024. Checked 2026-10-17. [^epoch]: [LLM inference prices have fallen rapidly but unequally across tasks](https://epoch.ai/data-insights/llm-inference-price-trends) — Ben Cottier, Ben Snodin, David Owen and Tom Adamczewski, Epoch AI, 2025-03-12. Checked 2026-10-17. [^litellm]: LiteLLM proxy documentation: [Budgets](https://docs.litellm.ai/docs/proxy/users), [Team budgets](https://docs.litellm.ai/docs/proxy/team_budgets), [Spend tracking](https://docs.litellm.ai/docs/proxy/cost_tracking) and [Config settings](https://docs.litellm.ai/docs/proxy/config_settings) — BerriAI/LiteLLM; version 1.105.1 on [PyPI](https://pypi.org/project/litellm/) (2026-10-14). Checked 2026-10-18. ### Changelog - 2026-10-17: first draft (managing-author pass); term status confirmed as emerging; Labs 1–3, the training estimate and the judgement-call model executed with Python 3.8.10. - 2026-10-17: r1 revision: Bedrock custom models, reasoning-token conventions, marginal capacity rule, retrieval costs; Lab 1 rebuilt around outcomes; Labs 2–3 and the judgement call re-run. - 2026-10-18, post-score polish: Anthropic multipliers scoped by model; forecast template; scenario C expanded, D moved to the FAQ; runbook and self-hosting gate clarified; re-check note; code wrapped.