ProductOps

ProductOps (product operations) keeps a product organisation supplied with trustworthy evidence and a few shared ways of working: product data people believe, customer feedback that is captured and coded rather than lost in chat, planning cadences everyone knows, and repeatable processes for experiments and launches. Product managers and leaders still decide what to build; ProductOps makes sure decisions rest on evidence, can be traced to it, and are reported back to the people who asked.

Picture a pipeline: signals (events, tickets, sales notes, interviews) become evidence (clean metrics, sized themes), evidence feeds decisions (roadmap choices, experiment verdicts, launch go/no-go), and shipped changes produce new signals. ProductOps runs the plumbing of that loop. It does not own the water.

Written for beginners, product managers, product operations staff, analysts and leaders. Sources last verified 2026-10-23.

The most cited definition comes from Melissa Perri and Denise Tilles. The publisher’s description of their 2023 book opens with “Product Operations is the discipline of helping your Product Management function scale well”,^[1] and lists three pillars: “Data and Insights”, “Customer and Market Insights” and “Process and Governance”.^[2]

In 2021 Marty Cagan (Silicon Valley Product Group) found at least six models in use, listed from most damaging to most valuable: the Reincarnated PMO, the Two-in-a-Box PM, the Delegated Product Leader, rebranded production operations, rebranded product marketing, and the Force Multiplier. He concluded that “if a company says ‘we have Product Ops,’ that’s a fairly meaningless statement today”.^[3] His recommended Force Multiplier (quantitative insights, qualitative insights, tools and best practices) was, he says, heavily influenced by how Perri defined product ops in her 2018 book Escaping the Build Trap,^[4] so the two best-known definitions are one lineage, not two independent sources. A 2025 community survey still names unclear roles and responsibilities as product ops teams’ top challenge (its method is not published on the report page).^[5]

A naming trap: Cagan’s “product operating model” is not ProductOps. It is for companies that believe they “should be powering their business with technology”, with teams given problems to solve rather than features to build.^[7][8] Neither requires the other.

Once there are more product teams than one leader can hold in their head, predictable problems appear: scattered evidence (usage data nobody trusts; requests and interview notes spread across tools); numbers that do not match (two teams, two “active users”); requests that vanish, so customer-facing teams escalate harder or stop sending feedback; planning reinvented each quarter, so roadmaps cannot be compared; and untrustworthy experiments and surprising launches.

When ProductOps works, the change is observable: one tracking plan and one set of metric definitions per product, one way to submit feedback, a decision log linking roadmap items to evidence, and PMs who spend more time with customers than with slides.

• Not a project management office. Cagan ranks the “Reincarnated PMO”, planning and governance run as command and control, as the most damaging model.^[3] Project control belongs to ProjectOps; investment choices to PortfolioOps.
• Not the product manager’s assistant. Splitting a PM’s job into strategy and operations halves cuts the PM off from the customers and data that make judgement possible (Cagan’s Two-in-a-Box model).^[3]
• Not the owner of the roadmap. ProductOps runs the process and keeps the evidence; product leaders own the choices.
• Not the research team or the BI team: studies belong to ResearchOps, the metric layer to AnalyticsOps (see the neighbour table).
• Not production operations, which belongs to SRE and IncidentOps.

• 2018 Melissa Perri’s Escaping the Build Trap (O’Reilly) defined product ops, as Cagan later reported.^[4][3]
• By 2020 the term was in mainstream practitioner writing: a ProductPlan article built on an interview with Melissa Perri said product ops is where product management becomes “a scalable resource and function for companies”, and is not program management.^[9]
• 2021 Cagan documented six competing uses.^[3]
• 2022 GitLab’s lead, about two and a half years into the role, described “an independent function under the product umbrella”;^[6] a practitioner survey drew “over a hundred people”.^[10]
• 2023 Perri and Tilles published Product Operations; the publisher lists case studies including Stripe, Uber, athenahealth, Oscar Health and Fidelity.^[2]
• 2026 The debate has moved to experimentation and AI: “The hard part is no longer building, but instead the critical skill is figuring out what to build” (Cagan).^[8]

No source opened here names the first company to create the role.

The mental model: ProductOps runs a factory for evidence and a few shared routines, and measures itself by how well other people decide. If nobody can name the decision an artefact makes faster, better or more traceable, it is bureaucracy.

ProductOps sits between functions that produce signals and functions that turn decisions into shipped work. The boundaries are editorial distinctions of this Atlas, agreed with the neighbouring pages.

%%{init: {"flowchart": {"wrappingWidth": 130, "nodeSpacing": 16, "rankSpacing": 30, "padding": 8}}}%%
flowchart TB
  BZ["BizOps: company goals and cadence"]
  PF["PortfolioOps: investment across products"]
  CX["CXOps, SupportOps: coded customer signals"]
  RO["ResearchOps: studies and the research repository"]
  P["ProductOps: insight-to-roadmap pipeline within a product line"]
  AN["AnalyticsOps: governed metric layer and BI"]
  AG["AgileOps: team cadence and flow"]
  RL["ReleaseOps: flags, rollout, release risk"]
  BZ -->|"company goals"| P
  PF -->|"funding and capacity per product"| P
  CX -->|"themes with evidence"| P
  P <-->|"questions out, insights back"| RO
  P <-->|"event design and metric definitions"| AN
  P -->|"prioritised roadmap items"| AG
  P <-->|"launch tiers, experiment plans"| RL
  RO ~~~ AG
  AN ~~~ RL
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  class P core
  class BZ,PF,CX,RO,AN,AG,RL neutral

DesignOps receives product priorities and returns design capacity; DecisionOps owns decision methods across the organisation. The comparisons part covers the hardest splits.

No official ProductOps manifesto exists. This charter is this Atlas’s proposal, built from the sources on this page.

Mission. Make good product decisions cheaper, faster and more traceable, without taking them away from the people accountable for them.

Cagan’s Force Multiplier parts are his paraphrase of Perri’s earlier pillars,^[3] so the columns below are one model in two vocabularies, not two sources that agree.^[2]

Experimentation and launch operations cut across all three. The data block need not sit in Product: GitLab’s Product Data Insights analysts report to its enterprise data organisation while serving Product, through an issue intake with templates.^[15]

Typical roles are a head of product operations (the operating model), product operations managers (cadences, intake, launches, tools), product analysts (tracking plan, metric definitions; they may sit in a data team) and an experimentation lead (policy, review, coaching). Cagan warns that many companies staff these roles with “very junior people”;^[3] they need enough seniority to coach PMs and to refuse process requests.

RACI (R responsible, A accountable, C consulted, I informed):

The pattern: ProductOps is accountable for mechanisms; product managers and leaders stay accountable for decisions.

Each forum needs an owner, inputs and a decision it produces.

When evidence and a stakeholder’s request conflict, ProductOps makes the trade-off visible and the product leader decides. GitLab’s handbook has a useful rule: a “customer” label with a due date marks a date-critical issue but “does not constitute a promise”.^[16]

The architecture is organisational and informational, not a server diagram: five layers, then a learning step that closes the loop.

%%{init: {"flowchart": {"wrappingWidth": 120, "nodeSpacing": 20, "rankSpacing": 30, "padding": 8}}}%%
flowchart TB
  subgraph SRC["1 Sources"]
    EV["Product events"]
    FB["Tickets, sales notes, research"]
  end
  TP["2 Capture: tracking-plan checks"]
  IN["2 Capture: one feedback intake"]
  MT["3 Evidence: product metrics"]
  TH["3 Evidence: coded, sized themes"]
  RV["4 Decisions: product review and planning"]
  DL["4 Decisions: decision log"]
  EX["5 Delivery: experiments"]
  LN["5 Delivery: tiered launches"]
  EV --> TP
  FB --> IN
  TP --> MT
  IN --> TH
  MT --> RV
  TH --> RV
  RV --> DL
  DL --> EX
  DL --> LN
  ML["6 Learn: measure results, tell requesters"]
  EX --> ML
  LN --> ML
  ML -.->|"new signals"| SRC
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  class TP,IN,MT,TH,EX,LN core
  class RV,DL decision
  class EV,FB neutral
  class ML decision

Two design choices matter most: whether tracking-plan checks block unplanned events (protects data, slows teams) or flag them for review, and whether themes are coded by hand (accurate, slow) or clustered by AI (fast, needs human review). Product metrics are defined here and served through the AnalyticsOps metric layer.

Trust boundaries. Product events and session data can contain personal data (see quality, privacy and compliance); feedback from named customers is commercially sensitive, so restrict who sees account names and deal values.

The workflow follows one piece of feedback. The happy path ends with a decision communicated back; exception paths handle missing evidence and urgent commercial pressure.

%%{init: {"flowchart": {"wrappingWidth": 130, "nodeSpacing": 16, "rankSpacing": 26, "padding": 8}}}%%
flowchart TB
  A["Signal: account manager logs a request"] --> B{"Matches an existing theme?"}
  B -->|"yes"| C["Link to theme; count the account once"]
  B -->|"no"| D["Create theme with a problem statement"]
  C --> E{"Theme above review threshold?"}
  D --> E
  E -->|"no"| F["Stays in backlog; requester told it is logged"]
  E -->|"yes"| G["Size it: reach, data, research links"]
  G --> H{"Evidence enough to decide?"}
  H -->|"no"| I["Ask ResearchOps or analytics; set a date"]
  I --> G
  H -->|"yes"| J["PM decides in product review"]
  J --> K["Decision log: build, park or decline"]
  K --> L["Close the loop with requester and account"]
  A -.->|"deal at risk, date-critical"| M["Escalate to product leader with evidence"]
  M --> J
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  class A,C,D,G,K,L core
  class B,E,H,J decision
  class F,I,M risk

Each step leaves a measurable trace: coding lag (capture to theme), insight-to-decision time (threshold to decision), decision traceability (decision with evidence) and closure rate (requester informed). Together they form the scorecard.

Each playbook gives steps and an acceptance test.

A tracking plan states “what events to track, where those events live in the code base, and why those events are necessary from a business perspective” (Segment).^[14] Amplitude marks events that arrive without a plan entry as “Unexpected”, to be added or deleted.^[17]

1. Start from decisions: list the events each product metric (activation, retention, adoption) needs; nothing else is added.
2. One naming convention, for example Object + Action in Title Case (Invoice Sent) and snake_case properties, with no dynamically generated names or keys.^[14]
3. An owner, a purpose and required properties for every event; instrumentation reviewed against the plan in pull requests.
4. Validate after release (Lab 1 automates it). GitLab’s handbook maps features to instrumentation status after launch, starting with the most-used.^[16]
5. Minimise personal data: identifiers, not emails or names (PrivacyOps).

Boundary (editorial distinction): ProductOps owns the product’s event design and product-metric definitions; the marketing and CDP plumbing that carries events (tags, consent signals, the CDP) belongs to MarTechOps.

Acceptance: every product metric is computable from planned events; conformance reported monthly. Rollback: fix broken events forward and annotate the affected dates in the metric layer.

1. One door: one form or tool with required fields (account or segment, source, the problem in the customer’s words, requester). Chat messages are not intake.
2. A short taxonomy of problems, not solutions (“importing data is hard”, not “add CSV import”), pruned monthly.
3. Count accounts, not mentions. Productboard, for example, links feedback to feature ideas as “insights” and weights an importance score by how critical the customer said it was;^[18] whatever the tool, state your weighting rule.
4. Size themes by reach, usage, research and revenue context (Lab 2); publish a monthly theme report.
5. Close the loop: build (with an honest date range), park (with what would change the decision) or decline (with a reason).

Acceptance: coding lag within target (for example five working days); every theme above the threshold decided within a quarter; closure rate reported. Where CXOps already codes feedback across the journey, start at step 4.

Prioritisation frameworks support conversation; they are not calculators. RICE scores Reach × Impact × Confidence ÷ Effort (impact 3, 2, 1, 0.5 or 0.25; confidence 100, 80 or 50 per cent; effort in person-months), and its author says it is not “a hard and fast rule”.^[19] GitLab standardises the scales for all teams so that “all prioritization decisions based on RICE are using the same metric”, scoring reach 10.0, 6.0, 3.0, 1.5 or 0.5 by the share of users who benefit in the first quarter after launch.^[16]

The quarterly cadence: at week −6 publish the calendar, templates and the company goals to serve; at week −4 teams draft roadmaps with evidence links (ProductOps checks completeness, not merit); at week −2 the product leader chairs the cross-team trade-off review; at week 0 roadmap and decision log are published, with conflicts across product lines escalated to PortfolioOps. For what to put on the roadmap, Teresa Torres’s opportunity solution tree runs from a desired outcome to opportunities, solutions and assumption tests;^[20] ProductOps makes the inputs easy to find, not the discovery.

An online controlled experiment (A/B test) is the strongest evidence most product teams can get, and the easiest to fool yourself with.

%%{init: {"flowchart": {"wrappingWidth": 130, "nodeSpacing": 16, "rankSpacing": 26, "padding": 8}}}%%
flowchart TB
  A["Brief: hypothesis, OEC, guardrails"] --> B["Power: users and days needed"]
  B --> C{"Design review passed?"}
  C -->|"no"| A
  C -->|"yes"| D["Start; check split daily"]
  D --> E{"Sample ratio as designed?"}
  E -->|"no"| F["Stop reading; investigate (runbook)"]
  E -->|"yes"| G["Run to planned size; no early peeking"]
  G --> H["Analyse OEC and guardrails"]
  H --> I{"OEC up, guardrails safe?"}
  I -->|"yes"| J["Ship decision by PM; log it"]
  I -->|"no"| K["Do not ship; log what was learned"]
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  class A,B,D,G,H,J core
  class C,E,I decision
  class F,K risk

The rules come from published practice:

• Hypothesis and metrics up front. A hypothesis “should be falsifiable or provable by the set of metrics considered”, with guardrail and data-quality metrics in the core set (Microsoft ExP);^[12] choose the OEC in advance.^[11]
• Size before starting: their approximation n = (4rσ/Δ)² (r variants, 95% confidence, 90% power) implies a little under 500,000 users to detect a 5% change in a 5% conversion rate.^[11] Lab 3 reproduces it.
• Check the split, strictly. A sample ratio mismatch “in most cases completely invalidates experiment results”; about 6% of experiments at Microsoft showed one during the study.^[13] Every daily look is another chance of a false alarm: in a simulation run for this page (200,000 healthy 14-day tests checked daily; code in Lab 3), a 0.01 threshold raised at least one alarm in about 5.5–5.6% of tests. Microsoft’s platform uses p < 0.0005 “to reduce the likelihood of false positives”;^[21] this page does the same.
• Enable, do not queue. Cagan advocates a hybrid model in which teams own their experiments “with coaching and assistance from experts in experimentation”.^[8] That is ProductOps’s role: policy, review and coaching. Flags and rollout belong to ReleaseOps.

Not every change deserves the same ceremony; a launch-tier matrix (in the reference implementation) sets the checks per tier. Two practices from GitLab’s handbook are worth copying:^[16] separate announcement from launch for large projects (announce early and roll out in stages, or roll out first and announce once proven), and use a four-phase transition instead of a hard cut-over (old only; both, old as default with gradual rollout; both, new as default with old as fallback; new only).

Launches also have a reliability side. Google’s SRE book describes a Launch Coordination Engineering team that uses a launch checklist to keep launches reliable, at up to 70 launches a week.^[22] For Tier 1–2 launches the ProductOps checklist links to that kind of review rather than replacing it.

There is no formal standard for ProductOps: no ISO standard and no certification body defines it. Separate what exists by strength:

What is mandatory comes from law, not from these frameworks: privacy and electronic-communications rules for analytics and experiments (see quality, privacy and compliance) and sector rules for regulated products.

Choose by job, not by brand. Examples are product descriptions verified as of October 2026; capabilities and plans change often.

PostHog and GrowthBook are “open core” (MIT outside enterprise directories);^[26][28] self-hosting moves cost from licences to your own operations.

Automate what is repetitive and checkable: tracking-plan validation in CI (Lab 1), routing feedback from support and CRM tools into the intake, daily experiment health checks, and launch checklists generated from the tier.

AI (mostly large language models) helps with feedback synthesis: clustering free text into candidate themes and summarising calls. Productboard, for example, sells AI topic detection that categorises feedback automatically.^[29] Three cautions: clusters are hypotheses (a model groups similar wording, not similar problems, so a person codes the final theme against raw text); re-count unique accounts from source data; and check where personal and deal data go when notes are sent to a model (PrivacyOps).

Assumed organisation: a B2B software company with eight product teams in two product lines and a ProductOps function of two (a product operations manager and a product analyst) reporting to the Chief Product Officer. Adapt the artefacts; keep the decisions they serve.

Artefact 1: ProductOps charter (one page).

PRODUCTOPS CHARTER                     Owner: Head of Product | Review: 6 months
Mission   Make product decisions cheaper, faster and traceable; decisions stay with PMs.
Serves    8 product teams (2 lines); sales, support, marketing as sources and receivers.
Owns      Tracking-plan standard; feedback intake + taxonomy; planning calendar and
          templates; decision log format; experiment policy; launch tiers; tool admin.
Does not  Own the roadmap; write PRDs; run research studies (ResearchOps); run the
          BI platform (AnalyticsOps); approve features; own company planning.
Measures  Insight-to-decision time; decision traceability; closure rate; experiment
          validity rate; instrumentation coverage; PM hours on planning (process tax).
Forums    Weekly insight triage; monthly product review; quarterly planning;
          experiment review; quarterly ProductOps health review.
Exit test Each routine names the decision it serves, or it is retired at the review.

Artefact 2: feedback intake form.

FEEDBACK INTAKE (required fields marked *)
* Submitted by / team           ________________
* Source                        [ ] support  [ ] sales  [ ] CS  [ ] research  [ ] other
* Account or segment            ________________   (no personal contact details here)
* Problem in the customer's words  ______________________________________________
  What are they trying to do?   ______________________________________________
  Current workaround            ______________________________________________
  Business context (optional)   [ ] renewal risk  [ ] new deal  [ ] expansion
  Deadline that matters (opt.)  YYYY-MM-DD   reason: ________________
* May we contact the account?   [ ] yes  [ ] ask me first
Coded by ProductOps: theme ______  status: new | linked | reviewed | decided

Artefact 3: experiment brief, checked against the policy, not against taste.

Example status: YAML syntax-checked by the Atlas validator; illustrative values, not executed against an experimentation tool.

experiment: checkout-address-autocomplete
owner: pm-payments
hypothesis: >
  Autocompleting the billing address will raise completed checkouts among users
  who start checkout, because address entry is the step with the most drop-off.
population: users who start checkout, web, all regions
baseline_rate: 0.50  # completed checkouts among starters, last four weeks
randomisation_unit: user
split: {control: 0.5, treatment: 0.5}
oec: completed checkouts per user who started checkout
guardrails:
  - checkout page load time (p75) must not rise by more than 5 per cent
  - payment error rate must not rise
data_quality:
  srm_check: chi-square test on every daily analysis; SRM if p < 0.0005
minimum_detectable_effect: 5 per cent relative
power: 0.8
planned_users_per_variant: 6280  # Lab 3: users_per_variant(0.50, 0.05)
planned_duration_days: 14
stopping_rule: no early stop for success; stop at once for guardrail breach or SRM
decision_owner: pm-payments
review: experimentation-lead

Record the baseline so reviewers can check the size. Among users who start checkout the completion rate is high (Kohavi and colleagues’ worked example assumes half of them complete^[11]), so a 5 per cent relative change is a larger absolute difference and needs 6,280 users per variant, not the 119,303 that Lab 3 prints for a 5 per cent baseline.

Artefact 4: launch-tier matrix.

Artefact 5: decision log entry.

DECISION  2026-Q4-07                       Date: 2026-10-14   Owner: PM, Billing
Question  Build bulk edit for invoices this quarter?
Decision  Build (target: first increment in 6 weeks)
Evidence  Theme T-12: 3 accounts in sample, ~120 accounts/quarter; 2 interviews;
          RICE 48 (Lab 2 scales); support handle time on invoice edits.
Rejected  SSO/SCIM provisioning: one large prospect; escalated to product leader.
Changes   We would revisit if import completion (Lab 2) falls below plan.
Loop      Requesters informed 2026-10-16 via intake tool; account teams briefed.

The scorecard is in metrics and KPIs.

Labs 1–3 use synthetic data, Python 3.8+ and the standard library: no accounts, network or cost. Each was run on 2026-10-23 exactly as printed (Python 3.8.10, Ubuntu Linux); expected outputs are copied from those runs. Lab 4 is a tabletop exercise (not executed).

Objective. Check a tracking plan for naming, ownership and purpose, then compare a sample of events with it. Prerequisites. Python 3.8+, a bash shell. Time. 20 minutes. Risk. None.

1. Create a working folder, the tracking plan and an event sample.

Example status: verified by execution on 2026-10-23 (bash, Linux).

mkdir -p ~/productops-labs/lab1 && cd ~/productops-labs/lab1
cat > tracking_plan.json <<'JSON'
{
  "product": "invoicing-web",
  "version": 3,
  "events": [
    {"name": "Invoice Created", "owner": "billing-team",
     "purpose": "Activation step 2; feeds the activation-rate KPI",
     "properties": {"invoice_id": "string", "line_count": "integer",
                    "currency": "string"},
     "required": ["invoice_id", "currency"]},
    {"name": "Invoice Sent", "owner": "billing-team",
     "purpose": "Activation step 3 (aha moment)",
     "properties": {"invoice_id": "string", "channel": "string"},
     "required": ["invoice_id", "channel"]},
    {"name": "Report Exported", "owner": "",
     "purpose": "",
     "properties": {"report_type": "string", "rowCount": "integer"},
     "required": ["report_type"]}
  ]
}
JSON
cat > events.jsonl <<'JSON'
{"event": "Invoice Created", "properties": {"invoice_id": "i1", "currency": "EUR"}}
{"event": "Invoice Sent", "properties": {"invoice_id": "i1", "channel": "email"}}
{"event": "Invoice Created", "properties": {"invoice_id": "i2", "line_count": 1}}
{"event": "invoice_sent", "properties": {"invoice_id": "i2", "channel": "email"}}
{"event": "Clicked Button 7", "properties": {"email": "jane@example.com"}}
{"event": "Report Exported", "properties": {"report_type": "aging"}}
JSON

2. Save the linter as lint_plan.py.

Example status: verified by execution on 2026-10-23 (Python 3.8.10, standard library).

"""Lint a tracking plan and a sample of events against it (standard library only)."""
import json
import re
import sys

OBJECT_ACTION = re.compile(r"^[A-Z][a-z]+( [A-Z][a-z]+)+$")  # e.g. "Invoice Sent"
SNAKE = re.compile(r"^[a-z][a-z0-9_]*$")
PII_TOKENS = {"email", "phone", "ip", "address"}  # whole snake_case tokens only
PII_KEYS = {"name", "first_name", "last_name", "full_name"}  # exact keys

plan = json.load(open("tracking_plan.json"))
events = {e["name"]: e for e in plan["events"]}
problems = 0

def report(level, msg):
    global problems
    problems += level == "ERROR"
    print(f"{level:5} {msg}")

# 1. The plan itself: naming, ownership, purpose.
for name, e in events.items():
    if not OBJECT_ACTION.match(name):
        report("ERROR", f"plan: '{name}' is not 'Object Action' in Title Case")
    if not e.get("owner"):
        report("ERROR", f"plan: '{name}' has no owner")
    if not e.get("purpose"):
        report("ERROR", f"plan: '{name}' has no purpose (which decision or KPI uses it?)")
    for prop in e["properties"]:
        if not SNAKE.match(prop):
            report("ERROR", f"plan: '{name}'.{prop} is not snake_case")

# 2. The observed events: unexpected names, missing required properties, likely PII.
seen = {}
for line in open("events.jsonl"):
    ev = json.loads(line)
    name, props = ev["event"], ev.get("properties", {})
    seen[name] = seen.get(name, 0) + 1
    for key in props:
        k = key.lower()
        if k in PII_KEYS or PII_TOKENS & set(k.split("_")):
            report("ERROR", f"data: '{name}' carries '{key}' (likely personal data)")
    if name not in events:
        continue
    missing = [r for r in events[name]["required"] if r not in props]
    if missing:
        report("WARN", f"data: '{name}' is missing required {missing}")

for name, n in seen.items():
    if name not in events:
        report("WARN", f"data: unexpected event '{name}' ({n}x) is not in the plan")
for name in events:
    if name not in seen:
        report("INFO", f"data: planned event '{name}' was not seen in this sample")

print(f"{problems} error(s)")
sys.exit(1 if problems else 0)

3. Run python3 lint_plan.py; echo "exit=$?".

Expected output.

ERROR plan: 'Report Exported' has no owner
ERROR plan: 'Report Exported' has no purpose (which decision or KPI uses it?)
ERROR plan: 'Report Exported'.rowCount is not snake_case
WARN  data: 'Invoice Created' is missing required ['currency']
ERROR data: 'Clicked Button 7' carries 'email' (likely personal data)
WARN  data: unexpected event 'invoice_sent' (1x) is not in the plan
WARN  data: unexpected event 'Clicked Button 7' (1x) is not in the plan
4 error(s)
exit=1

Validation. Fix the plan (owner reporting-team, a purpose, row_count), delete the Clicked Button 7 line from the sample and rename invoice_sent to Invoice Sent; the script then prints one warning (the missing currency) and 0 error(s), exit 0. A non-zero exit code lets CI fail a pull request. Troubleshooting. JSONDecodeError: a line in events.jsonl is not valid JSON (one object per line). The personal-data check is a heuristic: whole tokens avoid false hits such as shipping_method or plan_name, but it misses free text and unusual names, so review new properties in pull requests. Teardown. rm -r ~/productops-labs/lab1.

Objective. Deduplicate feedback by account, rank themes with RICE, and test how sensitive the ranking is. Prerequisites. Python 3.8+. Time. 30 minutes. Risk. None.

Example status: verified by execution on 2026-10-23 (bash and Python 3.8.10).

mkdir -p ~/productops-labs/lab2 && cd ~/productops-labs/lab2
cat > feedback.csv <<'CSV'
id,source,account,theme
F01,support,acme,bulk-edit
F02,sales,acme,bulk-edit
F03,support,globex,bulk-edit
F04,survey,initech,bulk-edit
F05,support,umbrella,sso-scim
F06,sales,hooli,sso-scim
F07,sales,hooli,sso-scim
F08,sales,hooli,sso-scim
F09,support,acme,csv-import
F10,support,globex,csv-import
F11,survey,initech,csv-import
F12,support,stark,csv-import
F13,interview,wayne,csv-import
F14,support,umbrella,csv-import
F15,support,soylent,unthemed
CSV
cat > themes.csv <<'CSV'
theme,accounts_per_quarter,impact,confidence,effort_pm,evidence
bulk-edit,120,1,0.8,2,3 accounts in sample; 2 interviews
sso-scim,40,2,0.5,3,sales asks; 1 large prospect
csv-import,300,0.5,0.8,1,6 accounts; funnel drop at import step
CSV

Save as rank.py:

"""Turn raw feedback into themes, then rank themes with RICE (standard library only)."""
import csv
from collections import defaultdict

# 1. Deduplicate: count each account once per theme, keep raw mentions for context.
mentions = defaultdict(int)
accounts = defaultdict(set)
for row in csv.DictReader(open("feedback.csv")):
    mentions[row["theme"]] += 1
    accounts[row["theme"]].add(row["account"])

print("theme        mentions  unique_accounts")
for t in sorted(mentions, key=lambda t: -len(accounts[t])):
    print(f"{t:12} {mentions[t]:8}  {len(accounts[t]):15}")

# 2. RICE = Reach x Impact x Confidence / Effort (Intercom's definition).
def rice(r):
    return (float(r["accounts_per_quarter"]) * float(r["impact"])
            * float(r["confidence"]) / float(r["effort_pm"]))

themes = list(csv.DictReader(open("themes.csv")))
print("\nRICE ranking (reach = accounts per quarter, effort = person-months)")
for r in sorted(themes, key=rice, reverse=True):
    print(f"{r['theme']:12} RICE={rice(r):6.1f}  evidence: {r['evidence']}")

# 3. Sensitivity: would one plausible change of a single estimate flip the order?
print("\nSensitivity: impact of sso-scim raised from 2 to 3 (a large deal depends on it)")
for r in themes:
    if r["theme"] == "sso-scim":
        r["impact"] = "3"
for r in sorted(themes, key=rice, reverse=True):
    print(f"{r['theme']:12} RICE={rice(r):6.1f}")

Run python3 rank.py.

Expected output.

theme        mentions  unique_accounts
csv-import          6                6
bulk-edit           4                3
sso-scim            4                2
unthemed            1                1

RICE ranking (reach = accounts per quarter, effort = person-months)
csv-import   RICE= 120.0  evidence: 6 accounts; funnel drop at import step
bulk-edit    RICE=  48.0  evidence: 3 accounts in sample; 2 interviews
sso-scim     RICE=  13.3  evidence: sales asks; 1 large prospect

Sensitivity: impact of sso-scim raised from 2 to 3 (a large deal depends on it)
csv-import   RICE= 120.0
bulk-edit    RICE=  48.0
sso-scim     RICE=  20.0

Validation. By hand: csv-import = 300 × 0.5 × 0.8 ÷ 1 = 120; sso-scim = 40 × 2 × 0.5 ÷ 3 = 13.3. Deduplication shows sso-scim has four mentions but two accounts. The lesson: raising its impact does not move it up, so if it must be built that is a commercial decision about one deal, not a RICE decision; it goes to the product leader with the revenue at stake, as in the workflow. Troubleshooting. KeyError: the CSV headers differ from the script. An “unthemed” row: code it before the monthly review. Teardown. rm -r ~/productops-labs/lab2.

Objective. Compute the users needed for an A/B test at several effect sizes, compare with a published rule of thumb, and run the chi-square SRM test. Prerequisites. Python 3.8+ (for statistics.NormalDist). Time. 30 minutes. Risk. None.

Example status: verified by execution on 2026-10-23 (Python 3.8.10, standard library).

"""Size an A/B test before it starts; check it for sample ratio mismatch (SRM)."""
import math
from statistics import NormalDist

def users_per_variant(p, relative_mde, alpha=0.05, power=0.80):
    """Two-sided test of two proportions, normal approximation, equal split."""
    delta = p * relative_mde
    z_a = NormalDist().inv_cdf(1 - alpha / 2)
    z_b = NormalDist().inv_cdf(power)
    variance = 2 * p * (1 - p)  # both arms assumed near the baseline rate
    return math.ceil((z_a + z_b) ** 2 * variance / delta ** 2)

SRM_ALPHA = 0.0005  # strict: the check runs on every daily analysis

def srm_p_value(n_control, n_treatment, expected_share=0.5):
    """Chi-square goodness-of-fit test, 1 degree of freedom."""
    total = n_control + n_treatment
    exp_c, exp_t = total * expected_share, total * (1 - expected_share)
    chi2 = (n_control - exp_c) ** 2 / exp_c + (n_treatment - exp_t) ** 2 / exp_t
    return chi2, math.erfc(math.sqrt(chi2 / 2))  # survival function of chi-square(1)

print("Sizing: baseline conversion 5%")
for mde in (0.05, 0.10, 0.20):
    for power in (0.80, 0.90):
        n = users_per_variant(0.05, mde, power=power)
        print(f"  {mde:.0%} relative lift, power {power:.0%}: {n:>9,} per variant")

# Kohavi et al. (KDD 2007) rule of thumb, 95% confidence and 90% power:
# total n = (4 * r * sigma / delta)^2 for r variants.
sigma, delta, r = math.sqrt(0.05 * 0.95), 0.05 * 0.05, 2
rule = (4 * r * sigma / delta) ** 2
print(f"  rule of thumb, 5% lift, 90% power: {rule:,.0f} in total")

print(f"\nSRM check (expected 50/50; flag if p < {SRM_ALPHA})")
for label, c, t in (("healthy", 50_210, 49_790), ("paper example", 821_588, 815_482)):
    chi2, p = srm_p_value(c, t)
    verdict = "SRM: do not read" if p < SRM_ALPHA else "no SRM"
    print(f"  {label:13} {c:>9,} vs {t:>9,}  chi2={chi2:7.2f}  p={p:.2e}  {verdict}")

Run mkdir -p ~/productops-labs/lab3 && cd ~/productops-labs/lab3, save the script there as experiment_check.py, then run python3 experiment_check.py.

Expected output.

Sizing: baseline conversion 5%
  5% relative lift, power 80%:   119,303 per variant
  5% relative lift, power 90%:   159,713 per variant
  10% relative lift, power 80%:    29,826 per variant
  10% relative lift, power 90%:    39,929 per variant
  20% relative lift, power 80%:     7,457 per variant
  20% relative lift, power 90%:     9,983 per variant
  rule of thumb, 5% lift, 90% power: 486,400 in total

SRM check (expected 50/50; flag if p < 0.0005)
  healthy          50,210 vs    49,790  chi2=   1.76  p=1.84e-01  no SRM
  paper example   821,588 vs   815,482  chi2=  22.77  p=1.82e-06  SRM: do not read

Validation. (1) Halving the detectable effect roughly quadruples the sample (29,826 → 119,303).^[11] (2) The rule of thumb gives 486,400 in total, matching the paper’s “less than 500,000”; the normal approximation needs about 319,000 at 90% power. The authors note that the factor of 4 “may overestimate by 25%” for large samples;^[11] the gap here is larger because 4² = 16 replaces (z₀.₉₇₅ + z₀.₉)² ≈ 10.5. (3) The paper’s 821,588 versus 815,482 split gives p ≈ 1.8 × 10⁻⁶, consistent with its “less than 1 in 500k”;^[13] the healthy 50.2/49.8 split of 100,000 users is not an SRM. The 0.0005 threshold follows Microsoft’s platform;^[21] one check at the planned size could use the SRM Checker extension’s 0.01.^[30]

Extension: why 0.0005 for daily checks. The playbook’s simulation: with equal daily traffic and no real mismatch, each day adds one normal step to the cumulative split imbalance. Save as daily_looks.py and run it.

Example status: verified by execution on 2026-10-23 (Python 3.8.10, standard library, about 3 seconds).

"""How often do 14 daily SRM checks flag a healthy test? (simulation)"""
import math, random
from statistics import NormalDist

TESTS, DAYS = 200_000, 14
for alpha in (0.01, 0.0005):
    z = NormalDist().inv_cdf(1 - alpha / 2)
    random.seed(1)
    alarms = 0
    for _ in range(TESTS):
        imbalance = 0.0
        for day in range(1, DAYS + 1):
            imbalance += random.gauss(0, 1)  # one day's split noise
            if abs(imbalance) / math.sqrt(day) > z:  # daily SRM test
                alarms += 1
                break
    share = 100 * alarms / TESTS
    print(f"threshold {alpha}: {share:.2f}% of healthy tests raise an alarm")

Expected output.

threshold 0.01: 5.61% of healthy tests raise an alarm
threshold 0.0005: 0.38% of healthy tests raise an alarm

Seeds 2–4 give 5.5% at 0.01.

Troubleshooting. AttributeError: module 'statistics' has no attribute 'NormalDist': Python older than 3.8. Unequal designed splits (for example 90/10): pass expected_share=0.9. Teardown. rm -r ~/productops-labs/lab3.

Objective. Decide which routines to keep. Prerequisites. A list of your recurring product forms, meetings and templates; two product managers; 60 minutes. Steps. (1) Record each routine’s owner, cadence and hours per PM per quarter. (2) Write the decision it serves and when it last changed one. (3) Mark routines with no named or changed decision in two quarters for retirement. (4) Retire one; automate one manual step (for example Lab 1 in CI). Expected output. A table of routines with cost and decision, and one or two retirements. Validation. At the next health review the process-tax survey shows fewer hours per PM and no loss of decision traceability. Teardown. File the table with the health-review notes.

All four scenarios are illustrative, built from the practices cited on this page, not reports of real companies.

ProductOps is judged by other people’s decisions, so its scorecard measures the pipeline and the cost the function adds. Product outcomes belong to product teams; ProductOps owns their definitions, for which HEART’s Goals–Signals–Metrics process is a good method.^[23]

Worked calculation (illustrative quarter). 31 of 40 committed items had linked evidence: traceability = 77.5%. 165 of 220 reviewed feedback items were answered within 30 days: closure = 75%. Eleven themes took 6, 9, 12, 14, 15, 21, 25, 30, 41, 48 and 77 days to decision: the median is 21 days, while the mean (27.1) is pulled up by one outlier, hence the median. One of twelve experiments showed an SRM: validity = 11 ÷ 12 = 91.7%. Process tax fell from 19 to 14 hours per PM. Next targets: traceability and closure, and why the 77-day theme lacked evidence.

Vanity traps: counts of dashboards, experiments, feedback items or templates, which can rise while decisions get worse. Surveys need the same care: a 2022 practitioner survey found 91% of respondents already had a product ops function,^[10] which says more about who answers (mostly product ops staff) than about the market.

Legal analysis belongs to PrivacyOps and counsel; ProductOps asks the questions.

• Consent for analytics depends on jurisdiction. In the UK, the ICO’s guidance (as of October 2026) exempts storage and access for statistics from consent only where that is the sole purpose, the output is aggregate, any analytics provider acts as a processor, and users are informed and can object simply and free of charge; it “does not apply to purposes related to online advertising”.^[31] Other jurisdictions differ.
• Minimise data in events and tools: identifiers, not names or emails; no free text; retention periods; masking in replay and survey tools.
• Experiments on people that touch prices, eligibility, health or financial information need review beyond statistics.
• Confidentiality: restrict access to account names and deal values in the intake; Tier 1 launch claims get legal review.

Costs are people, tools (often priced by event or user volume; Amplitude advertised 2 million free events a month as of October 2026^[24]) and, largest and least visible, product teams’ time: a planning cycle costing eight PMs ten hours each is 80 hours. No source opened here gives a credible ratio of ProductOps staff to PMs; size the team from the work. Consolidating overlapping tools cuts cost and duplicated truth but deepens dependence on one vendor. Collecting only events that serve a decision also cuts storage and energy (GreenOps covers carbon).

• Quarterly health review of the scorecard, process tax and a two-question PM survey (“which routine helps you decide; which would you remove?”).
• Sample audits: trace five committed roadmap items to their evidence, check five closed feedback items really reached the requester, re-run the SRM check on three experiments.
• Exception monitoring: undecided themes older than a quarter; launches without validated instrumentation.

%%{init: {"flowchart": {"wrappingWidth": 130, "nodeSpacing": 16, "rankSpacing": 26, "padding": 8}}}%%
flowchart TB
  A["SRM alert on a running test"] --> B["Freeze: no decision, no announcement"]
  B --> C{"SRM in all assigned users?"}
  C -->|"yes"| D["Check assignment: bucketing, redirects, start times"]
  C -->|"only in a filtered subset"| E["Check the trigger or filter condition"]
  D --> F{"Root cause found and fixable?"}
  E --> F
  F -->|"yes"| G["Fix, restart with a new seed, re-size"]
  F -->|"no"| H["Discard the test; record the unknown"]
  G --> I["Post-incident note in the experiment log"]
  H --> I
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  class B,D,E,G,I core
  class C,F decision
  class A,H risk

1. Freeze. Nobody ships or announces: an SRM in most cases invalidates the results.^[13]
2. Quantify with the chi-square test (Lab 3) on assigned and analysed counts.
3. Locate. A rule of thumb from the SRM paper, as listed in Vermeer’s FAQ: a mismatch only in a filtered subset points to the filter or trigger condition.^[30] The paper groups causes by stage: assignment, execution, log processing and analysis.^[13] Common culprits: treatment code that crashes or redirects before logging, bots filtered differently, staggered starts.
4. Decide. If fixed, restart with fresh assignment and the full planned size, never stitching old and new data; otherwise discard. Record the cause and, if systemic, automate a check.

In this scenario the new widget fails on an old mobile browser and those users leave before the analysis event fires, so the treatment lost the users most likely to fail checkout and its lift was inflated.

Create a dedicated function when most of these hold: five to ten or more product teams or several product lines; roadmaps that cannot be compared and numbers nobody trusts; customer-facing teams that rarely hear back; visible experiment or launch mistakes; PMs spending much of their time assembling data and status.

Do not, when:

• There are three or four PMs: share the routines in rotation (one tracking-plan document, one feedback channel with a 20-minute weekly triage, a one-page planning calendar, each owned by a PM for a quarter).
• The gap is product leadership: ProductOps cannot take over the coaching and product strategy that leaders owe their teams (Cagan’s Delegated Product Leader).^[3]
• The goal is control of teams (a project office).
• The data problem is a platform problem: fix it with DataOps and AnalyticsOps first.

%%{init: {"flowchart": {"wrappingWidth": 130, "nodeSpacing": 16, "rankSpacing": 26, "padding": 8}}}%%
flowchart TB
  A{"More than about 5 product teams or several product lines?"}
  A -->|"yes"| C{"Do leaders make and own product decisions?"}
  A -->|"no"| B["Shared routines in rotation; no hire"]
  C -->|"yes"| E{"Which pain is largest?"}
  C -->|"no"| D["Fix product leadership first"]
  E -->|"untrusted data"| F["Start with the data pillar: analyst plus tracking plan"]
  E -->|"lost feedback"| G["Start with intake, themes, closing the loop"]
  E -->|"chaotic planning or launches"| H["Start with calendar, decision log, launch tiers"]
  F --> I["Review in 2 quarters: add a pillar or stop"]
  G --> I
  H --> I
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  class F,G,H,I core
  class A,C,E decision
  class B,D risk

The thresholds, and the advice to start with one pillar, are this Atlas’s judgement, not research findings.

This maturity model is this Atlas’s proposal; no official ProductOps maturity model exists.

Level 4 has fewer central routines than level 3.

Dependencies: analytics access, a product leader who decides in the forums, engineering time for instrumentation.

Quick start (first month)

• Charter with mission, owned scope, non-goals and measures
• Baseline: process tax, traceability, closure, conformance
• One intake channel announced; planning calendar for next quarter published

Experiment preflight

• Falsifiable hypothesis; OEC and guardrails named before start
• Sample size, duration and minimum detectable effect agreed
• Randomisation unit and split recorded; daily SRM check scheduled
• Stopping rule written; ethical or legal review where the policy requires it

Launch readiness (Tier 1–2)

• Tier recorded at planning; instrumentation validated in staging
• Support, sales and documentation enablement complete
• Staged rollout with fallback; reliability review for high-traffic changes
• Announcement timing chosen; post-launch review date set

Quarterly ProductOps health review

• Scorecard, sample audits and process-tax survey reviewed
• At least one routine retired or simplified; non-goals respected

Theme report (monthly).

THEME REPORT  <product line>  <month>
Top themes by unique accounts (this month / quarter to date)
  T-12 Importing existing data   accounts 6 / 14   trend up   status: decided (build)
  T-07 Bulk editing              accounts 3 / 9    trend flat status: in review
  T-21 SSO/SCIM provisioning     accounts 2 / 4    trend flat status: escalated
New themes: T-30 ...            Retired themes: T-03 (merged into T-12)
Decisions this month (from decision log): 4 build, 2 park, 3 decline
Loop closure this month: 41 of 52 requesters informed (79%)
Data notes: intake missing account on 6 items; coding lag median 3 days

Product review agenda (monthly, 60 minutes).

1. Scorecard (10 min): outcome metrics vs plan; data-quality notes
2. Experiments (15 min): verdicts, validity issues, what we learned
3. Themes (20 min): top themes above threshold; decisions needed today
4. Launches (10 min): last month's post-launch reviews; next month's Tier 1-2
5. Decisions recorded (5 min): read back the decision log entries

Experiment policy (one page).

EXPERIMENT POLICY                                  Owner: experimentation lead
1. Every experiment has a brief: hypothesis, OEC, guardrails, power, stopping rule.
2. Design review before start for tests above <N> users or touching pricing,
   eligibility, health or financial information.
3. SRM test on every analysis; threshold p < 0.0005 (write yours); SRM = not read.
4. No early stop for success; immediate stop for guardrail breach or SRM.
5. Verdicts are made by the PM and recorded in the decision log with the brief.
6. Quarterly audit: re-check three past winners against holdouts or later data.

Other templates are in the reference implementation.

These tests settle disputed cases; ownership is in the neighbour table.

The hardest split is with ResearchOps: both work on customer insight, often in the same person. Keep the boundary at the participant: recruiting, consenting and paying participants is ResearchOps; turning findings and other signals into roadmap decisions is ProductOps.

Is ProductOps a job title or a function? Both. GitLab describes a function with its own lead;^[6] many companies have one product operations manager.

Should it report to the Chief Product Officer? Usually, because its customers are product teams. GitLab described its product operations in 2022 as “under the product umbrella”,^[6] while its product analysts now sit in the data organisation;^[15] both work if the charter is clear.

Can ProductOps own the roadmap tool? As administrator, yes; not the roadmap’s content. Keep tool rights and decision rights apart.

Do we need an experimentation platform? No. A brief, a power calculation, an SRM check and a decision log work with a spreadsheet and a feature flag; platforms only automate the checks.

Does AI make ProductOps unnecessary? It removes some work (summarising, clustering) and adds some (checking output, governing data sent to models). Cagan argues the bottleneck is now deciding what to build,^[8] the problem the pipeline serves.

• Established: experiment validity checks (SRM, guardrails, a pre-registered OEC) are documented in research and industrial practice.^[13][12]
• Vendor-led: “agentic” product management, a marketing theme across the category as of October 2026.^[32] Treat claims of autonomous prioritisation as unproven; use AI to assemble evidence, with human review.
• Open question: no study opened for this page isolates the effect of a ProductOps function on product outcomes. Measure the pipeline and the process tax, and be honest that the link to outcomes is a judgement.

Option A: mandatory scores, central check. Comparable roadmaps, visible assumptions, a defence against escalation by volume. Option B: shared standard scales (as in GitLab’s handbook^[16]), team-owned scores, a sampled audit (the audit is this Atlas’s proposal): ProductOps calibrates scales quarterly, audits ten items and facilitates the review where scores conflict; small items need no score.

Assumptions. About 135 items a quarter; a 20-minute check each costs about 45 hours plus team rework; scores are uncertain (in Lab 2 one changed estimate moved a score by 50 per cent).

Analysis. Option A makes ProductOps a gate, which the charter’s first principle rules out, and invites gaming as teams learn which numbers pass. Its main benefit, comparability, comes from the standard scales, which Option B keeps at a fraction of the cost while leaving decisions with product leaders.

Recommendation under these assumptions: Option B; an item with large commercial or regulatory stakes gets a full evidence page and a product-leader decision instead of a score. What would change it: audits showing systematic inflation. No study opened here compares the options; this is reasoned judgement.

After this page you should be able to explain ProductOps and its term status, separate it from its neighbours, design the core artefacts, size and check an experiment, and decide whether an organisation needs the function.

ProductOps is a young label for an old need: decisions that rest on evidence, and routines that let many teams act as one product organisation. What survives scrutiny is a narrow, demanding core: trustworthy product data, feedback that is captured, coded and answered, a few cadences that name the decisions they serve, and experiment and launch processes that stop teams fooling themselves.

Judge ProductOps not by the process it creates but by how much better and cheaper decisions become, and by how many routines it is willing to retire. Build the pipeline, measure your own process tax, and leave the decisions with the people accountable for the product.

Every source was opened on 2026-10-23; support strength and unfetchable pages are in research/productops-ledger.md. Vendor capabilities change often.

Further reading

• Ron Kohavi, Diane Tang and Ya Xu, Trustworthy Online Controlled Experiments (Cambridge University Press, 2020), not opened for this page.

Changelog

• 2026-10-23: first draft (ledger, labs 1–3 executed, 6 diagrams); cut to the emerging-term band; pipeline figure redrawn after QA.
• 2026-10-23 (r1 revision): brief sized from a stated baseline; SRM threshold 0.0005; SRM and statistics terms defined; Perri’s 2018 book added; attributions tightened; MarTechOps boundary; Lab 1 token matching; sections condensed; labs re-run.
• 2026-10-23, post-score polish: Artefact 5 RICE label fixed; SRM simulation code in Lab 3; pipeline loop drawn, figure fits phones; repetitions trimmed.