TestOps

TestOps is the practice of running a test suite as if it were a production service. Its users are the developers who wait for it, and its product is a verdict on each change: safe to merge, or broken and where. Its service levels: verdicts within minutes, red builds that mean real defects, a known cost per verdict. A test that lies (passes and fails on the same code) gets an owner and a deadline, not endless retries.

No standards body or neutral reference defines “TestOps”. Several vendors use the word as a product or feature name, each with a different scope.^[1][2][3] Wikipedia describes it as managing “the operational aspects of testing within the software delivery lifecycle”, in a thinly sourced article tagged as an orphan.^[4] The only peer-reviewed study found, an analysis of practitioner talks, reports “an inconsistency in practitioner perspectives on the nature of TestOps”.^[5] The practices themselves are mature, under older names: continuous testing, test automation, test management.^[6][7]

It is for SDETs and QA leads, CI engineers, managers who pay for suites, and architects deciding what to test before a merge.

Suites decay gradually: they grow, slow down, pick up tests that fail at random, and people learn to press “re-run”. At Google in 2016, about 1.5% of all test runs reported a flaky result, almost 16% of tests had some flakiness, and about 84% of observed pass-to-fail transitions in post-submit testing involved a flaky test.^[8] When most red builds are noise, people stop reading them and real regressions slip through. Slack’s mobile teams reported a main-branch pass rate around 20%, and about 28 minutes of manual triage per test failure, before they automated flaky-test handling.^[9]

DORA says not to tolerate flaky tests, and adds: “A test suite of ten tests that is reliable, fast, and trustworthy is much better than a test suite of hundreds of tests that is hard to maintain and that nobody trusts.”^[7] TestOps is the operational work that keeps a large suite in the first condition.

• Not a product category. Vendors define the word by what their product does:

• Not a renamed QA department. Developers write and fix most tests; see roles.
• Not test management. Planning and tracking test cases against requirements is test management; ISO/IEC/IEEE 29119 standardises software testing in general.^[10] TestOps focuses on the running system.
• Not release policy. Deciding that a failing check blocks a release is a ReleaseOps decision. TestOps owns whether the check deserves that power. This is an editorial distinction of this Atlas, not an industry standard.

The practices are older than the word: the test pyramid goes back to Mike Cohn in 2003–04,^[11] Fowler wrote about quarantining non-deterministic tests in 2011,^[12] and Google published its flaky-test mitigations in 2016.^[8] Wikipedia dates the word to March 2019, but its source could not be opened, so treat that date as unverified.^[4] That month Keysight’s white paper “The TestOps Manifesto” used it for “DevOps for design and test” of electronic systems.^[13] By 2020 enough practitioner talks used it for Doležel to study them; he found two meanings: an organisational philosophy (testers as a “bridge” between Dev and Ops) and a technique supported by technology.^[5] In 2021 Joe Colantonio, writing up his TestGuild interview with Testim founder Oren Rubin and Microsoft engineer Maor Frankel, called it “all about automation at scale”.^[14] This page follows the technical, software meaning and leaves the organisational one to DevOps.

Mental model: the suite is a service whose product is a verdict. Its service levels borrow SRE vocabulary (an SLI is “a carefully defined quantitative measure of some aspect of the level of service”; an SLO is a target for it).^[20] It fails in two ways. A false red wastes time and teaches people to ignore red; a false green ships a defect. Retries, quarantine and test selection trade one for the other, so each needs measurement and an owner.

flowchart LR
  DEV["DevOps: CI"]:::neutral
  SRE["SRE: SLOs"]:::neutral
  API["API Ops: contracts"]:::neutral
  MOB["MobileOps: devices"]:::neutral
  T["TestOps: trusted verdicts"]:::core
  REL["ReleaseOps: gates"]:::decision
  DX["DevExOps: friction"]:::neutral
  SEC["DevSecOps: scans"]:::neutral
  DEV -->|"runs tests"| T
  SRE -->|"vocabulary"| T
  API -->|"specs"| T
  MOB -->|"results"| T
  T -->|"verdict"| REL
  T -->|"wait times"| DX
  T -->|"flake and duration data"| SEC
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029

The charter below is this Atlas’s proposal: no neutral TestOps manifesto exists, and Keysight’s covers electronic design and test.^[13]

Mission: every change gets a verdict within the agreed time, a red verdict means a real problem, and the suite costs what it is worth. Non-goals: maximising test count or coverage percentage; owning product quality on behalf of developers; deciding release policy.

1. A red build is a claim. Each red verdict is true or traceable to a ticket with an owner; DORA measures whether test failures represent real defects.^[7]
2. Find each defect with the cheapest test that can catch it. In DORA’s guidance, fast unit tests run before slower acceptance tests, and a bug found in a later stage gets a unit test.^[7]
3. Budgets are explicit. Each stage has a time budget; DORA’s upper limit for CI tests is about 10 minutes.^[21]
4. Determinism beats coverage. Google sizes tests by resource limits because “the most important qualities we want from our test suite are speed and determinism”.^[15]
5. Quarantine is temporary and capped. Fowler suggested limits such as “only allow 8 tests in quarantine” or “no longer than a week”.^[12]
6. Test code is product code, with the same review and design care.^[18]

DORA found teams do better when developers are primarily responsible for automated tests, with testers alongside them; when other groups own the automation, two problems often arise: the pipeline stays broken until that group fixes the tests, and developers write hard-to-test code.^[7] At Google each team has a “Build Cop” who keeps its tests passing, and whose most effective tool is rollback.^[22] The RACI is illustrative (R responsible, A accountable, C consulted, I informed).

The states are this page’s model. Burn-in responds to a finding by Luo and colleagues: “Most flaky tests (78%) are flaky the first time they are written”.^[23] It cannot catch the rest: in a study summarised by Parry and colleagues, 75% of 245 flaky tests were flaky from the commit that added them, and for the others a median of 144 commits passed before a change made them flaky.^[24] Detection keeps running after burn-in.

stateDiagram-v2
  state "Later stage (post-merge or nightly)" as Later
  [*] --> BurnIn: new test merged
  BurnIn --> BurnIn: fails, then fix pushed
  BurnIn --> Gating: clean repeated runs
  Gating --> Later: too slow for its stage
  Gating --> Suspected: pass and fail on one commit
  Suspected --> BurnIn: fix pushed within the deadline
  Suspected --> Quarantined: no fix by the deadline
  Quarantined --> BurnIn: fix pushed
  Quarantined --> Deleted: expiry reached
  Later --> Deleted: duplicates a faster test
  Deleted --> [*]

Escalation. GitLab escalates a top flaky test file still open after 24 hours to a Severity 3 production incident.^[25]

The architecture is tool-neutral. Whether its parts are a vendor product (Allure TestOps, Trunk, Azure DevOps flaky-test management) or scripts and a database matters less than the data it keeps: every result, with commit, attempt number, duration, environment and owner.^[26][27][28] Without commit and attempt, “flaky” cannot be told from “broken”.

flowchart TB
  subgraph EXEC["Execution"]
    ENV["Environments and test data"]
    RUN["CI runners and shards"]
  end
  subgraph SIGNAL["Signal"]
    ING["Ingest JUnit XML and JSON reports"]
    STORE["Results store: commit, attempt, duration"]
    ANA["Analysis: flakes, durations, failure groups"]
  end
  subgraph CONTROL["Control"]
    REG["Quarantine registry: owner, issue, expiry"]
    GATE["Gate: block, note or pass"]
    ROUTE["Issues and alerts to owners"]
  end
  ENV --> RUN
  RUN --> ING
  ING --> STORE
  STORE --> ANA
  ANA --> REG
  ANA --> ROUTE
  REG --> GATE
  STORE -->|"this run's results"| GATE
  REG -->|"which tests block"| RUN
  GATE -->|"verdict"| MQ["Merge queue and release policy"]
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  class GATE decision
  class STORE,ANA core

Design notes:

• Keep attempt-level results. CI systems read JUnit XML (pytest --junit-xml=path),^[29][30] which has no single schema. In Lab 2, Playwright’s JUnit reporter recorded a retried-then-passed test as a plain pass, while its JSON reporter recorded flaky; so did pytest’s for a test re-run with pytest-rerunfailures^[31] (14.0, run for this page). Maven Surefire keeps reruns in its own <flakyFailure> and <rerunFailure> elements.^[32] Check what your producers write, and store the richer format too.
• Identify tests by a stable id (JUnit classname plus name), never by bare name: names repeat across files.

flowchart TB
  L["Laptop: small tests, seconds"] --> PR["Pull request: fast stage, at most 10 min"]
  PR --> MQ["Merge queue: same checks on the merged result"]
  MQ --> PM["Post-merge: full suite, sharded, example 30 min"]
  PM --> NI["Nightly: slow, soak and performance tests"]
  NI --> PD["After deploy: smoke tests, synthetics, canary analysis"]
  PD -->|"escaped defect"| NEW["Add the cheapest test that would catch it"]
  NEW --> L
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  class PR,MQ decision
  class NEW core

Happy path. A developer pushes; the fast stage is green within budget; the merge queue re-runs the required checks on the combined result (GitHub needs the merge_group event);^[33] later stages stay green.

Exception paths.

1. Fails, then passes on retry: the attempts are stored and the test is flagged as suspected, never silently green.
2. Fails on every attempt: red. Re-run the parent commit now, because its stored green result may predate a time- or data-dependent break. If it is green, revert this change quickly;^[21] if it is red, the break is older, so find the culprit first^[22] (an earlier merge, a dependency, a date-dependent test). See the runbook.
3. Over budget: the gate adds a note; repeated overruns go to the weekly review.
4. Infrastructure fails (runner lost, registry timeout): recorded as an infrastructure error, not against the test.

Shift-right signals. After deployment, canary analysis compares the new version with the current one, and synthetic checks run scripted user journeys against production from probe locations.^[34][35] Treat them like any other check (owner, flake handling). Production monitoring belongs to ObservabilityOps.

Playbook 1: a flaky test appears.

1. Confirm: the same test passed and failed on one commit (rerun the job, or use history).^[8]
2. Classify the cause. Luo and colleagues’ top three categories were asynchronous waits, concurrency and test-order dependency;^[23] GitLab’s taxonomy adds state leaks, random input, selectors, date and time, and infrastructure.^[36] GitHub retries a failure three ways (same process; time shifted forward; another host) to separate randomness, time assumptions and shared state.^[37]
3. Fix by the deadline, or quarantine with owner, issue and expiry. Quarantined tests keep running non-blocking, because quarantine “could easily mask a real race condition”.^[8]
4. Release from quarantine only after burn-in (for example Playwright’s --repeat-each).^[38]
5. Delete the test if it expires unfixed; GitLab deletes after three months.^[39]

Playbook 2: the fast stage is over budget.

1. Measure p50 and p90 per stage and the slowest 20 tests.
2. Push down slow broad tests; Microsoft’s guideline is under 60 ms on average per L0 test, none over 2 seconds.^[18]
3. Skip work that cannot have changed: by default Bazel reuses a cached result unless the test or its dependencies changed, it failed, it is marked external, or repeats were requested.^[40]
4. Shard by timing data, not file count;^[41] Playwright balances per test only with fullyParallel: true.^[42]
5. Select tests per change, with a safe fallback and periodic full runs (deep comparisons).^[16]
6. Move the rest post-merge, and make sure a failure there still blocks release through ReleaseOps policy.

Much flakiness and slowness comes from what a test talks to. Choose the cheapest option that still exercises the risk:

Test data playbook (steps 1, 2 and 4 follow DORA;^[46] step 3 is this page’s advice):

1. Each test creates its own data in setup, through the application’s APIs; isolated data is “a prerequisite for tests to run in parallel”.
2. Keep test data minimal (it needs upkeep as APIs change); unit tests need none from outside.
3. Name created records uniquely per run and worker; delete them in teardown, or use a throwaway container.
4. For realistic journeys, export relevant, masked sections of production regularly, never a full copy.

• Portfolio shapes. The pyramid (many unit tests, few broad UI tests)^[11] competes with the honeycomb (mostly integration tests, few “Integrated Tests”, ideally none)^[47] and the trophy; Fowler notes that people mean different things by “unit” and “integration”.^[48] Choose by cost and determinism, not by shape.
• Size over scope. Google’s small, medium and large tests (one process, one machine, anywhere) can be enforced by the test infrastructure;^[15] Bazel turns sizes into default timeouts of 60, 300 and 900 seconds.^[49] Microsoft uses L0–L4, where L4 tests run against production.^[18]
• Contract testing. Consumer-driven contracts let a provider learn what its consumers rely on;^[50] Pact implements this and its can-i-deploy tool checks a matrix of verified versions before a deployment.^[17][51]
• Standards. ISO/IEC/IEEE 29119 is the international software-testing standard series; Part 1 (general concepts) was published in 2022.^[10] TMMi is a staged test-process maturity model from the non-profit TMMi Foundation: organisations start at level 1 and climb through Managed, Defined and Measured to Optimization.^[52] Neither standardises flaky-test handling or CI budgets.
• Professional syllabus. The closest body of knowledge is ISTQB’s Test Automation Engineering syllabus (CTAL-TAE v2.0, 2024-05-03), with chapters on CI/CD pipeline integration, reporting and metrics, verifying the automation solution, and continuous improvement; strategy moved to a separate Test Automation Strategy syllabus.^[53] It is a certification syllabus, not a standard.

Tools named are examples checked in their documentation on 2026-10-16, not recommendations.

Lock-in is mostly about data: prefer tools that export attempt-level results through an API.

Automation is essential; machine learning is optional and must earn its place. Meta’s gradient-boosted model caught “more than 99.9 percent of all regressions” while running “just a third of all tests that transitively depend on modified code”, after flaky results were separated out of the training data.^[61] Atlassian scores flakiness with rerun signals and Bayesian inference.^[62] Vendors advertise AI that will “heal locators when the app changes” and generate test cases;^[2] this page found no independent evidence of the effect on escaped defects. A self-healing test can heal over a real regression, so log and review every heal. Generated tests need burn-in and owners; humans decide on deleting tests or changing gate policy.

Assumptions: GitHub Actions, Python unit tests, Playwright end-to-end tests, quarantine tracked in quarantine.json, results kept as CI artifacts. A merge queue is optional: as of October 2026, GitHub offers it in public repositories owned by an organization and in private repositories of organizations on GitHub Enterprise Cloud.^[33] Elsewhere, require branches to be up to date before merging,^[63] or use GitLab merge trains (Premium and Ultimate; see GitLabOps).^[64]

Example status: syntax-checked, not executed (requires a GitHub repository); the @quarantine filters and stamp.py ran locally. Actions are referenced by their October 2026 major-version tags for readability. GitHub’s strongest measure is a workflow dependency lockfile, which “supersedes SHA pinning for repositories that adopt it”; without one, pin full-length commit SHAs.^[65]

# .github/workflows/tests.yml
name: tests
on:
  pull_request:          # untrusted code: no deployment secrets in this workflow
  merge_group:           # required for merge-queue checks
permissions:
  contents: read
jobs:
  fast:
    runs-on: ubuntu-latest
    timeout-minutes: 12                  # budget 10 min plus setup headroom
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-python@v7
        with: { python-version: "3.12" }
      - run: pip install -r requirements-test.txt
      - run: pytest -m "not quarantine" -n auto --junit-xml=results/fast.xml
      - run: python3 tools/stamp.py results/fast.xml   # commit and run attempt
        if: ${{ !cancelled() }}                        # also after failed tests
      - uses: actions/upload-artifact@v7
        if: ${{ !cancelled() }}
        with: { name: results-fast, path: results/fast.xml }
  quarantined:                           # keeps running and reporting, never blocks
    runs-on: ubuntu-latest
    continue-on-error: true
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-python@v7
        with: { python-version: "3.12" }
      - run: pip install -r requirements-test.txt
      - run: pytest -m quarantine --junit-xml=results/quarantine.xml
      - run: python3 tools/stamp.py results/quarantine.xml
        if: ${{ !cancelled() }}
      - uses: actions/upload-artifact@v7
        if: ${{ !cancelled() }}
        with: { name: results-quarantine, path: results/quarantine.xml }
  e2e:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false                   # finish every shard so results are complete
      matrix: { shard: [1, 2, 3, 4] }
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-node@v7
        with: { node-version: 24 }
      - run: npm ci && npx playwright install --with-deps chromium
      - run: >-
          npx playwright test --shard=${{ matrix.shard }}/4 --grep-invert @quarantine
          --retries=1 --reporter=list,json
        env:
          PLAYWRIGHT_JSON_OUTPUT_NAME: results/e2e-${{ matrix.shard }}.json
      - uses: actions/upload-artifact@v7
        if: ${{ !cancelled() }}
        with:
          name: results-e2e-${{ matrix.shard }}
          path: results/e2e-${{ matrix.shard }}.json
  e2e-quarantined:                       # tests tagged @quarantine: run, never block
    runs-on: ubuntu-latest
    continue-on-error: true
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-node@v7
        with: { node-version: 24 }
      - run: npm ci && npx playwright install --with-deps chromium
      - run: >-
          npx playwright test --grep @quarantine --pass-with-no-tests
          --reporter=list,json
        env:
          PLAYWRIGHT_JSON_OUTPUT_NAME: results/e2e-quarantine.json
      - uses: actions/upload-artifact@v7
        if: ${{ !cancelled() }}
        with: { name: results-e2e-quarantine, path: results/e2e-quarantine.json }
  gate:                                  # the one check branch protection requires
    needs: [fast, quarantined, e2e, e2e-quarantined]
    if: ${{ !cancelled() }}              # run even when a needed job failed ...
    runs-on: ubuntu-latest
    steps:
      - name: Block when a blocking job did not succeed   # ... and fail here
        if: ${{ needs.fast.result != 'success' || needs.e2e.result != 'success' }}
        run: echo "fast=${{ needs.fast.result }} e2e=${{ needs.e2e.result }}" && exit 1
      - uses: actions/checkout@v7
      - uses: actions/download-artifact@v8
        with: { pattern: "results-*", path: results/, merge-multiple: true }
      - run: >-
          python3 tools/gate.py results/*.xml --playwright-json results/e2e-*.json
          --quarantine quarantine.json --budget-min 10

Why the first gate step matters: GitHub skips a job whose needed job failed “unless the jobs use a conditional expression that causes the job to continue”,^[66] and !cancelled() is one. Without that step, gate would pass after a red e2e, and a branch requiring only gate would merge end-to-end failures. With --retries=1, a flaky end-to-end test passes its shard; only the JSON report records it, and the gate (Lab 3) turns it into a NOTE.

Each test runner needs a quarantine route:

• Playwright: tag the test ({ tag: '@quarantine' }) and add its registry entry (file::title). Blocking shards skip it, e2e-quarantined runs it,^[67] and a tagged failure without an entry blocks the gate.
• pytest: the hook below marks registry entries by their JUnit XML id (classname::name), so a namesake in another file stays blocking. pytest does not retry here: its retry plugin’s reruns vanish from JUnit XML (design notes).

Example status: executed with pytest 8.3.5 on Python 3.8.10 on 2026-10-17: only the registry’s two tests moved to the quarantine run; namesakes in another file or a class, and another parametrisation, stayed blocking.

# conftest.py: mark every test listed in quarantine.json as "quarantine"
import json
import pathlib

import pytest

ENTRIES = json.loads(pathlib.Path(__file__).with_name("quarantine.json")
                     .read_text(encoding="utf-8"))["entries"]
QUARANTINE = {e["test"] for e in ENTRIES}  # ids as in JUnit XML: classname::name


def junit_id(nodeid):
    """tests/test_cart.py::TestTax::test_x[2] -> tests.test_cart.TestTax::test_x[2]"""
    path, bracket, params = nodeid.partition("[")
    *scope, name = path.split("::")
    scope[0] = scope[0][:-3] if scope[0].endswith(".py") else scope[0]
    dotted = ".".join(s.replace("/", ".") for s in scope)
    return dotted + "::" + name + bracket + params


def pytest_configure(config):
    config.addinivalue_line("markers", "quarantine: known-flaky, outside the gate")


def pytest_collection_modifyitems(config, items):
    for item in items:
        if junit_id(item.nodeid) in QUARANTINE:
            item.add_marker(pytest.mark.quarantine)

Commit and attempt in every result. The fast job runs under pytest-xdist, where record_testsuite_property does not work,^[68] so both pytest jobs stamp their XML with the script below, from GitHub’s default variables.^[69] Ship the stamped files to your results store; Lab 1’s report reads them unchanged.

Example status: executed with pytest 8.3.5 and pytest-xdist 3.6.1 (Python 3.8.10) on 2026-10-17: under -n 2 the fixture wrote nothing; Lab 1’s report read the stamped files and found a same-commit flake.

"""stamp.py: add commit and run attempt to JUnit XML (works with pytest-xdist)."""
import os
import sys
from xml.etree import ElementTree as ET  # trusted CI files; else use defusedxml

# Default variables in every GitHub Actions step. GITHUB_SHA is the code tested:
# the pull request's merge commit, or the merge group's SHA. A re-run keeps it.
STAMP = {"commit": os.environ["GITHUB_SHA"],
         "attempt": os.environ["GITHUB_RUN_ATTEMPT"]}  # 1, then +1 per re-run

for path in sys.argv[1:]:
    tree = ET.parse(path)
    root = tree.getroot()
    for suite in [root] if root.tag == "testsuite" else root.findall("testsuite"):
        props = suite.find("properties")
        if props is None:
            props = ET.Element("properties")
            suite.insert(0, props)  # <properties> comes first in a testsuite
        for name, value in STAMP.items():
            ET.SubElement(props, "property", name=name, value=value)
    tree.write(path, encoding="utf-8", xml_declaration=True)

For services that call other teams’ services, add a contract stage instead of a shared integration environment. The deploy job follows Pact’s documented order: can-i-deploy (is this version verified against the target environment?), deploy, then record-deployment.^[51]

Keep PACT_BROKER_BASE_URL and the credentials (PACT_BROKER_USERNAME and PACT_BROKER_PASSWORD for the open-source Broker, PACT_BROKER_TOKEN for PactFlow)^[70] as secrets of a production environment that only main may deploy to. GitHub gives environment secrets only to jobs that reference the environment, after any required reviewer approves, so pull_request jobs never see them. Preconditions: administrator bypass disabled (administrators may bypass protection rules by default) and, for private repositories, a plan that offers these controls.^[71]

Example status: syntax-checked only (requires a Pact Broker and a deployment script).

# deploy job only: runs on main in the protected "production" environment
set -euo pipefail
pact-broker can-i-deploy --pacticipant checkout-web --version "${GIT_SHA}" \
  --to-environment production         # exit code 1 = "not safe": the job stops here
./deploy.sh production "${GIT_SHA}"   # your deployment step
pact-broker record-deployment --pacticipant checkout-web --version "${GIT_SHA}" \
  --environment production            # only after the deployment succeeded

Lab 1 (beginner): find the lies in your history. Objective: compute the false-red share, test time and flaky tests from JUnit XML. Prerequisites: Python 3.8 or later, no packages; about 15 minutes; no cost or risk (local files only).

Example status: executed on Python 3.8.10 on 2026-10-17; the output below is verbatim. The report was also run on files written by pytest 8.3.5 and on Surefire-style files with <flakyFailure>.

1. Save the generator: 40 pytest-style runs of 200 tests whose bare names repeat across modules; its comments mark the flaky, skipped and regressed tests.

"""make_history.py: write 40 JUnit XML files shaped like pytest's, for 40 CI runs."""
import random
from pathlib import Path
from xml.sax.saxutils import quoteattr

random.seed(7)
MODULES = ["tests.test_cart", "tests.test_search", "tests.test_upload",
           "tests.test_account"]
FLAKY = {"tests.test_search::test_autocomplete": 0.08,  # chance of a false failure
         "tests.test_upload::test_retry": 0.04,         # fails in setup: <error>
         "tests.test_account::test_session_timeout": 0.02}
REGRESSION = "tests.test_cart::test_case_037"           # really breaks in commit 25
SKIPPED = "tests.test_account::test_case_040"           # always skipped
# Every module has test_case_000 ... so bare names repeat; ids are classname::name.
IDS = list(FLAKY) + [f"{m}::test_case_{n:03d}"
                     for n in range(50) for m in MODULES][:197]

out = Path("history")
out.mkdir(exist_ok=True)
commit = 0
for run in range(1, 41):
    if run == 1 or random.random() > 0.3:   # 30% of runs re-run the previous commit
        commit += 1
    cases, total = [], 0.0
    for tid in IDS:
        classname, name = tid.split("::")
        failed = random.random() < FLAKY.get(tid, 0.0)
        failed = failed or (tid == REGRESSION and commit >= 25)
        secs = round(random.uniform(0.2, 9.0), 2)
        total += secs
        body = ""
        if tid == SKIPPED:
            body = '<skipped message="needs a licence server"/>'
        elif failed and "upload" in tid:
            body = '<error message="fixture setup failed"/>'
        elif failed:
            body = '<failure message="assertion failed"/>'
        attrs = f"classname={quoteattr(classname)} name={quoteattr(name)}"
        cases.append(f'<testcase {attrs} time="{secs}">{body}</testcase>')
    props = f'<properties><property name="commit" value="c{commit:03d}"/></properties>'
    xml = (f'<testsuites name="pytest tests">'
           f'<testsuite name="pytest" tests="{len(IDS)}" time="{total:.2f}">'
           f'{props}{"".join(cases)}</testsuite></testsuites>')
    (out / f"run-{run:03d}.xml").write_text(xml, encoding="utf-8")
print(f"wrote 40 runs covering {commit} commits to {out}/")

2. Save the report script. It keys tests by classname::name and calls a test flaky only when it passed and failed on the same commit (Google’s definition).

"""flake_report.py: report flaky tests, false-red runs and test time from JUnit XML."""
import sys
from collections import defaultdict
from pathlib import Path
from xml.etree import ElementTree as ET  # trusted CI files; else use defusedxml

FAILED = ("failure", "error")           # pytest reports fixture errors as <error>
RERUN = ("flakyFailure", "flakyError")  # Surefire: failed, then passed on a rerun


def read_run(path):
    """One file = one CI run. Return its commit and (test id, passed, seconds) rows."""
    root = ET.parse(path).getroot()  # <testsuites> (pytest, Playwright) or <testsuite>
    prop = root.find(".//properties/property[@name='commit']")
    if prop is None:
        sys.exit(f"{path}: no 'commit' property; see Troubleshooting")
    rows = []
    for c in root.iter("testcase"):
        if c.find("skipped") is not None:  # skipped tests say nothing about flakiness
            continue
        tid = f"{c.get('classname')}::{c.get('name')}"  # names repeat across files
        rows += [(tid, False, 0.0) for t in RERUN for _ in c.findall(t)]
        passed = all(c.find(t) is None for t in FAILED)
        rows.append((tid, passed, float(c.get("time", 0))))
    return prop.get("value"), rows


runs = [(f.stem, *read_run(f)) for f in sorted(Path(sys.argv[1]).glob("*.xml"))]
seen = defaultdict(set)      # (test id, commit) -> outcomes seen on that exact code
history = defaultdict(list)  # test id -> outcomes in run order
for _, commit, rows in runs:
    for tid, passed, _ in rows:
        seen[(tid, commit)].add(passed)
        history[tid].append(passed)

flaky = sorted({t for (t, _), o in seen.items() if o == {True, False}})
broken = sorted(t for t, h in history.items()
                if t not in flaky and not any(h[-5:]))   # last 5 outcomes all failed
red = []  # (run, tests whose final outcome in that run was a failure)
for name, _, rows in runs:
    fails = {t for t, p, _ in rows if not p} - {t for t, p, _ in rows if p}
    if fails:
        red.append((name, fails))
false_red = [r for r, fails in red if fails <= set(flaky)]
minutes = sorted(sum(s for *_, s in rows) / 60 for *_, rows in runs)


def flips(h):  # status changes in the last 10 outcomes (a common flakiness heuristic)
    last = h[-10:]
    return sum(a != b for a, b in zip(last, last[1:]))


print(f"runs={len(runs)} commits={len({c for _, c, _ in runs})} tests={len(history)}")
print(f"red runs={len(red)}  false-red (only flaky tests failed)={len(false_red)} "
      f"-> {len(false_red) / max(len(red), 1):.0%} of red runs were noise")
print(f"summed test minutes per run p50={minutes[len(minutes) // 2]:.1f} "
      f"p90={minutes[int(len(minutes) * 0.9)]:.1f}")
print("flaky tests (passed and failed on the same commit):")
for t in flaky:
    h = history[t]
    print(f"  {t:<42} fails {h.count(False):>2}/{len(h)}  flips(last10)={flips(h)}")
print("consistently failing (last 5 runs red, never flaky):",
      ", ".join(broken) or "none")

3. Run python3 make_history.py && python3 flake_report.py history.

Expected output:

wrote 40 runs covering 28 commits to history/
runs=40 commits=28 tests=199
red runs=11  false-red (only flaky tests failed)=6 -> 55% of red runs were noise
summed test minutes per run p50=15.2 p90=15.8
flaky tests (passed and failed on the same commit):
  tests.test_search::test_autocomplete       fails  5/40  flips(last10)=3
  tests.test_upload::test_retry              fails  2/40  flips(last10)=1
consistently failing (last 5 runs red, never flaky): tests.test_cart::test_case_037

• Validation: 6 of 11 red runs were noise; the regression is reported separately; tests=199 because the skipped test is ignored. The 2% test never failed in 40 runs, so no detector could see it: rare flakes need longer windows. Three flips in the last 10 runs is also Allure TestOps’ flakiness threshold.^[26]
• Why ids matter: change the tid line to tid = c.get("name"). The report then sees 53 tests, lists the regression test_case_037 as flaky, and calls 100% of red runs noise.
• Troubleshooting: “no ‘commit’ property” means the run does not record its commit. With pytest, add it with the record_testsuite_property fixture^[30] or, under pytest-xdist, with stamp.py from the reference implementation.
• Teardown: delete history/.

Lab 2 (intermediate): retries, flaky status and shards in Playwright. Objective: see how a retry changes the verdict and what each report keeps. Prerequisites: Node.js 22 or 24 (LTS; Node.js lists 20 as end-of-life as of October 2026),^[72] an empty folder, npm install --save-exact @playwright/test@1.64.1; no browser download. About 20 minutes. Cost and risk: one pinned npm package and its two dependencies.

Example status: executed with Node 24.22.0 and @playwright/test 1.64.1 on 2026-10-16 (1.63.0 gave the same results).

// playwright.config.js: no browser is needed for this lab
const { defineConfig } = require('@playwright/test');

module.exports = defineConfig({
  testDir: './tests',
  fullyParallel: true,          // shard by test, not by file
  retries: 0,                   // set per run on the command line
  reporter: [['list'], ['junit', { outputFile: 'results/junit.xml' }]],
});

// tests/cart.spec.js: three stable tests and one simulated flaky test
const { test, expect } = require('@playwright/test');

const total = (items) => items.reduce((sum, i) => sum + i.price * i.qty, 0);

test('empty cart totals zero', () => {
  expect(total([])).toBe(0);
});

test('total multiplies price by quantity', () => {
  expect(total([{ price: 5, qty: 3 }, { price: 2, qty: 1 }])).toBe(17);
});

test('cart ignores zero-quantity lines', () => {
  expect(total([{ price: 9, qty: 0 }])).toBe(0);
});

// Simulated flake: the first attempt of every fourth repetition fails,
// standing in for a 1-in-4 race with a slow dependency.
test('discount service answers in time', async ({}, testInfo) => {
  const unlucky = testInfo.repeatEachIndex % 4 === 0 && testInfo.retry === 0;
  expect(unlucky, 'discount service timed out').toBe(false);
});

Validation: after step 2, results/junit.xml shows the flaky test as an ordinary passing <testcase>; only the list and JSON reporters show it was flaky, so a JUnit-only dashboard would call the suite healthy. Troubleshooting: “No tests found” means testDir does not match the folder. Teardown: delete the folder (keep e2e.json for Lab 3).

Lab 3 (advanced): a gate that blocks on quarantine abuse. Objective: enforce owners, expiry and a cap on quarantine, keep same-named tests apart, and turn flaky end-to-end results into notes. Prerequisites: Python 3.8 or later; optionally e2e.json from Lab 2; about 30 minutes. Cost and risk: none locally; in a pipeline, start non-blocking (see Rollback).

Example status: executed on Python 3.8.10 on 2026-10-17; outputs verbatim.

{
  "max_entries": 8,
  "max_days": 14,
  "entries": [
    {"test": "tests.test_search::test_autocomplete", "owner": "team-search",
     "issue": "SHOP-1412", "reason": "async wait on suggestion API",
     "added": "2026-10-09", "expires": "2026-10-23"},
    {"test": "tests.test_upload::test_retry", "owner": "team-media",
     "issue": "SHOP-1398", "reason": "order dependency with test_quota",
     "added": "2026-09-20", "expires": "2026-10-04"}
  ]
}

"""gate.py: block real failures and quarantine abuse; note flaky and slow results."""
import argparse
import json
import sys
from datetime import date
from xml.etree import ElementTree as ET  # trusted CI files; else use defusedxml

ap = argparse.ArgumentParser()
ap.add_argument("junit", nargs="+", help="JUnit XML (blocking and quarantine jobs)")
ap.add_argument("--playwright-json", nargs="*", default=[], help="Playwright JSON")
ap.add_argument("--quarantine", default="quarantine.json")
ap.add_argument("--today", default=date.today().isoformat())
ap.add_argument("--budget-min", type=float, default=10.0)
a = ap.parse_args()

q = json.load(open(a.quarantine, encoding="utf-8"))
today = date.fromisoformat(a.today)
quarantined = {e["test"]: e for e in q["entries"]}  # test id = classname::name
problems, notes = [], []

for e in q["entries"]:
    tid, age = e["test"], (today - date.fromisoformat(e["added"])).days
    if not e.get("owner") or not e.get("issue"):
        problems.append(f"quarantine entry {tid} has no owner or issue")
    if today > date.fromisoformat(e["expires"]) or age > q["max_days"]:
        problems.append(f"quarantine expired: {tid} ({age} days): "
                        "fix, delete or renew")
n, cap = len(q["entries"]), q["max_entries"]
if n > cap:
    problems.append(f"{n} tests in quarantine (cap {cap})")


def judge(tid, failed, retried):
    if failed and tid in quarantined:
        issue = quarantined[tid]["issue"]
        notes.append(f"quarantined (non-blocking): {tid} -> {issue}")
    elif failed:
        problems.append(f"FAILED: {tid}")
    elif retried:
        notes.append(f"flaky, passed on retry: {tid}")


for path in a.junit:
    root = ET.parse(path).getroot()
    suites = [root] if root.tag == "testsuite" else root.findall("testsuite")
    minutes = sum(float(s.get("time", 0)) for s in suites) / 60  # wall clock per suite
    if minutes > a.budget_min:
        notes.append(f"{path} took {minutes:.1f} min (budget {a.budget_min:g})")
    for c in root.iter("testcase"):
        judge(f"{c.get('classname')}::{c.get('name')}",
              c.find("failure") is not None or c.find("error") is not None,
              c.find("flakyFailure") is not None or c.find("flakyError") is not None)


def specs(suite, titles=()):  # Playwright nests describe blocks as suites in each file
    for spec in suite.get("specs", []):
        yield f"{spec['file']}::{' › '.join(titles + (spec['title'],))}", spec["tests"]
    for child in suite.get("suites", []):
        yield from specs(child, titles + (child["title"],))


for path in a.playwright_json:
    for top in json.load(open(path, encoding="utf-8"))["suites"]:
        for tid, tests in specs(top):
            for t in tests:
                judge(tid, t["status"] == "unexpected", t["status"] == "flaky")

for n in notes:
    print("NOTE ", n)
for p in problems:
    print("BLOCK", p)
print("gate:", "FAIL" if problems else "PASS")
sys.exit(1 if problems else 0)

Steps: save the files above as quarantine.json and gate.py and this result file as run.xml, then run python3 gate.py run.xml --playwright-json e2e.json --today 2026-10-16 (without Lab 2, drop --playwright-json e2e.json and the flaky line).

<testsuites name="pytest tests">
  <testsuite name="pytest" tests="5" time="708.4">
    <testcase classname="tests.test_cart" name="test_total" time="61.5"/>
    <testcase classname="tests.test_search" name="test_autocomplete" time="240.2">
      <failure message="timeout"/></testcase>
    <testcase classname="tests.test_admin" name="test_autocomplete" time="2.1">
      <failure message="assert 3 == 4"/></testcase>
    <testcase classname="tests.test_cart" name="test_tax" time="310.0"/>
    <testcase classname="tests.test_upload" name="test_retry" time="95.0"/>
  </testsuite>
</testsuites>

Expected output:

NOTE  run.xml took 11.8 min (budget 10)
NOTE  quarantined (non-blocking): tests.test_search::test_autocomplete -> SHOP-1412
NOTE  flaky, passed on retry: cart.spec.js::discount service answers in time
BLOCK quarantine expired: tests.test_upload::test_retry (26 days): fix, delete or renew
BLOCK FAILED: tests.test_admin::test_autocomplete
gate: FAIL

• Validation: tests.test_admin::test_autocomplete blocks although a namesake is quarantined, because ids include the file. Remove the expired entry and the namesake’s <failure> and the gate prints gate: PASS (exit 0); add a <failure/> to test_tax and it prints BLOCK FAILED: tests.test_cart::test_tax; with --today 2026-10-24 the other entry expires (15 days).
• Budget note: it uses the suite’s wall-clock time, not summed test times, which overstate time under parallel workers; time to verdict comes from the CI API (metrics).
• Troubleshooting: ValueError: Invalid isoformat string means a date is not YYYY-MM-DD.
• Rollback: run the gate with continue-on-error: true while teams clean up, then make it required.
• Teardown: delete the files.

All four cases below are the companies’ own published accounts; none was independently audited.

Worked calculation 1: why “a little flakiness” is not little. Assume 1,000 tests per run, each with an independent 0.1% chance of failing without a real defect. P(at least one false failure) = 1 − 0.999^1000 ≈ 1 − e^−1 ≈ 63% of runs. With one independent retry a false failure needs two misses in a row (p = 0.000001), so 1 − (1 − 0.000001)^1000 ≈ 0.1%. That is why retries are tempting; they also turn real intermittent bugs green, and flakes that depend on time or order are not independent. Google’s book makes the same point: 0.1% across 10,000 daily tests is ten flakes a day.^[15]

Worked calculation 2: a test-signal SLO and its error budget. SLI: share of main-branch runs whose verdict was true. SLO: 98% over 28 days. With 1,200 main runs, the error budget is 1,200 × 2% = 24 false reds. If triage has labelled 40 by day 20, the budget is spent and the agreed policy applies: no new end-to-end tests merge, and the top flaky tests get owners next sprint. As in SRE, a budget without agreed consequences is only a chart.^[20]

Vanity traps: raw test count; coverage percentage alone (see the FAQ); pass rate after retries.

• Untrusted code in CI. Pull-request jobs get read-only tokens and no deployment secrets; never combine pull_request_target or workflow_run with a checkout of untrusted code.^[65]
• Test data. Use masked sections, never full production copies (playbook). Whether personal data may be used for testing is a legal, jurisdiction-specific question; see PrivacyOps.
• Production. Presubmit tests should not talk to production backends: Google warns this “can be dangerous” (security and quota vulnerabilities).^[22] Tests in production run behind safeguards such as deployment tiers and feature flags.^[19]
• Audit evidence. Where regulation or contracts require test evidence, agree what to keep (results, approvals, quarantine decisions) and for how long with ComplianceOps.

Sharding buys wall-clock time with extra machine time. GitHub’s standard hosted runners are free in public repositories, and each plan includes some minutes for private ones.^[73] Beyond those, GitHub rounds each job up to a whole minute and charges $0.006 per minute for a Linux 2-core runner (as of October 2026);^[74] the table applies to such private-repository minutes.

Going from 8 to 16 shards saves 2.5 minutes per run for 50% more spend, because setup does not shrink: cut setup (caching, prebuilt images) and skip unchanged tests before adding shards. People cost more: a triaged false red costs engineer time (Slack’s figure is in the problem), far more than the $0.26 to $0.58 per run above. Fewer wasted runs also use less energy; deleting tests that never catch anything is the cheapest optimisation.

Check the system that checks the code: review gate overrides; compare escaped defects with tests quarantined at the time; audit that every registry entry has an owner, an issue and an unexpired date (Lab 3 automates this). Mutation testing inserts small faults to see whether tests notice; Google’s paper calls it “widely considered the strongest test criterion” and runs a diff-based form in code review.^[75]

Scenario (illustrative): 9 of the last 20 main-branch runs are red, developers keep pressing re-run, and two teams have merged with failing checks “because it’s flaky”.

flowchart TB
  A["Main run is red"] --> I{"Infra error?"}
  I -->|"yes"| INF["Infra incident"]
  I -->|"no"| B{"Red on re-run?"}
  B -->|"no"| C["Flaky: fix or quarantine"]
  B -->|"yes"| D{"Parent re-run green?"}
  D -->|"yes"| F["Revert the culprit change"]
  D -->|"no"| W["Bisect from last green"]
  W --> X{"Code change?"}
  X -->|"yes"| F
  X -->|"no"| E["Environment or dependency"]
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  class I,B,D,X decision
  class INF,E,F risk
  class C core
  class A,W neutral

1. Stop the bleeding (first 15 minutes). Announce that main is red; pause merges that skip checks. If a regression is likely, revert first: DORA advises reverting when the build cannot be fixed in a few minutes.^[21]
2. Classify each failing test with the flow; mixed results on one commit mean flaky by definition.^[8] For a red parent, Google’s CI splits failing batches and offers tools that binary-search for the culprit.^[22]
3. Quarantine with discipline: owner, issue and expiry for each entry; the gate refuses entries without them (Lab 3).
4. Check the two teams’ merges, the first culprit candidates: run the full suite several times on each, quarantined tests included, and revert any merge where a test fails consistently.
5. Verify recovery: ten consecutive green main runs without retries.
6. Review within a week: why did the noise build up unseen (no flake dashboard, retries hiding outcomes, no owner mapping)? Track follow-ups as in IncidentOps.

• “Retry until green.” It turns verdicts into dice rolls and hides real races. Bazel’s own documentation discourages its flaky attribute, which runs a test up to three times.^[57]
• Filtering failures out of results. Slack’s history-based suppression let a new, genuinely broken test into main and was rolled back.^[9]
• Quarantine as a graveyard. Without a cap and expiry the list only grows; Fowler warns tests get “thrown into quarantine and forgotten”.^[12]
• The ice-cream cone. Most checks run through the UI, so the suite is slow and brittle.^[11]
• Myth: more tests mean more safety. DORA prefers ten trustworthy tests (see the problem).

Invest in explicit TestOps practices when two or more of these hold: the fast stage regularly exceeds about 10 minutes;^[21] people re-run failing jobs as a habit; main is red more than once a week for reasons other than regressions; more than one team shares a suite; CI spend is a line item someone asks about.

Do not build a platform when: the suite runs in under five minutes and failures are almost always real; there is one team and one repository; or the real problem is missing tests, not unreliable ones. Then follow DevOps practice and write good tests. Hidden costs: results storage, dashboard upkeep, a triage rota, and time for teams to fix their tests.

The decision that matters most day to day is whether a given test may block merges.

flowchart TB
  Q1{"Clean over many repeated runs?"} -->|"no"| A1["Fix it or run it non-blocking"]
  Q1 -->|"yes"| Q2{"Fits the stage time budget?"}
  Q2 -->|"no"| A2["Move it to post-merge or nightly"]
  Q2 -->|"yes"| Q3{"Has an owner and a clear failure message?"}
  Q3 -->|"no"| A3["Assign an owner first"]
  Q3 -->|"yes"| Q4{"Catches defects no faster test catches?"}
  Q4 -->|"no"| A4["Delete it or push it down"]
  Q4 -->|"yes"| A5["Let it block merges"]
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  class Q1,Q2,Q3,Q4 decision
  class A5 core
  class A1,A4 risk

Assumptions: “many repeated runs” is a team choice (for example 50 repeats in burn-in); budgets come from the stage map; and the release consequence of a post-merge failure is set in ReleaseOps.

This is this Atlas’s model, not an industry standard; unlike TMMi^[52] it covers only operating automated suites.

Dependencies: a CI system with artifact retention, a way to map tests to owners (for example code owners), and management agreement that fixing tests counts as real work.

Quick start

• Every test job writes JUnit XML, plus a richer format where retries are dropped (for example Playwright JSON)
• Tests are identified by a stable id (classname::name), not by bare name
• Results are stored with commit, attempt, duration and environment
• Each test maps to an owning team
• Stage map written down, with a time budget per stage

Preflight: before a test may block merges

• Passes the four questions of the decision framework
• Needs no shared mutable data; its failure message says what broke, not only “timeout”

Weekly review

• False-red share and p90 time to verdict against SLOs
• Quarantine count under the cap; nothing past expiry
• Top three flaky tests have owners and dates
• Slowest 20 tests reviewed for push-down or deletion

Governance audit (quarterly)

• Gate overrides and skipped checks reviewed
• CI tokens least-privilege; no secrets in pull-request jobs
• Test data contains no unmasked personal data
• Escaped defects traced to missing or quarantined tests

Red main branch (incident)

• Follow the worked runbook; close only after ten green runs without retries

One-page suite charter

Suite:            <name>            Owner (accountable): <engineering manager>
Users:            <teams who wait for the verdict>
Stages & budgets: PR fast stage <= 10 min | post-merge <= <n> min | nightly <= <n> h
SLOs (28 days):   true-verdict share >= 98% | p90 time to verdict <= <n> min
Quarantine:       cap <n> tests | max age <n> days | owner + issue required
Error-budget policy: when exhausted -> <freeze new E2E tests / staff fixes / ...>
Review:           weekly scorecard <day>, quarterly prune <month>

Quarantine request

Test id:        <classname::name, exactly as in the JUnit XML>
Evidence:       <run ids where it passed and failed on one commit>
Suspected cause: async wait | concurrency | order dependency | time | data
                 | infrastructure | unknown
Owner team:     <team>             Issue: <id>
Expiry date:    <YYYY-MM-DD> (default 14 days)
Coverage risk:  <what could break unnoticed while quarantined; compensating check>
Approved by:    <test owner> on <date>

Weekly scorecard

Is TestOps a job title or a team? Usually neither: it is a set of responsibilities shared by SDETs, a CI or platform team and feature teams. One team owning everyone’s tests repeats the ownership problem described under roles.

Do we need a “TestOps platform” product? Not to start: a results store plus Labs 1 and 3 cover maturity levels 1–2. Buy when maintaining routing, history and dashboards costs more than a licence.

How many retries are acceptable? It depends on the retry’s job. To unblock a change, one retry is enough if its outcome is stored; GitLab retries a failing test once, in a separate process.^[36] To classify a failure, one retry is weak evidence: GitHub’s earlier approach (a later re-run in the same build, plus comparing builds of the same code) identified only 25% of flaky failures; three targeted retries identified 90%.^[37] Retries cost minutes and delay the verdict: Google notes that a 15-minute test marked flaky, reporting failure only after three failures in a row, takes 45 minutes to reveal a real break.^[8]

Is 100% code coverage a good target? No. Coverage shows which code ran, not whether results were checked. Use it per change, and judge test strength with mutation testing or escaped-defect reviews.^[79][75]

Can AI fix flaky tests for us? It can rank, group and suggest fixes, but a flaky test can hide a product race condition,^[8] so a human confirms the cause before an automated change hides the symptom.

• Detecting flakes without re-running. Practice still re-runs: Parry and colleagues’ survey of 76 papers calls repeated execution “a common baseline approach”. Rerun-free methods are research. Differential coverage (DeFlaker) found 96% of the flaky tests a rerun strategy found in open-source Java projects; machine-learning classifiers reached an average F1 of 0.86 in one study, where another feature set, evaluated the same way without artificially balanced test data, averaged 0.19.^[24] Treat such scores as promising, not as a replacement for history and retries.
• Change-risk signals. Google found code recently modified by more than three developers breaks more often, which could weight test selection.^[80]

• Decide: what each stage must answer, and its time budget (about 10 minutes for the fast stage).^[21]
• Measure: false-red share, p90 time to verdict, quarantine count and age, escaped defects, cost per verdict.
• Practise: same-commit flake detection, stable test ids, quarantine with owner and expiry, burn-in, timing-based shards.
• Watch out for: unrecorded retries, results-based suppression, uncapped quarantine, a gate that ignores failed jobs.
• Next action: export last month’s CI results and run Lab 1’s report on them.

Situation (illustrative). A 4,000-test suite, 30 teams, main green 70% of the time. Option A: one automatic retry everywhere; a pass on retry counts as green. Option B: no retries on main; any flaky failure immediately disables the test and files a ticket for its owner, the model Slack ended up with.^[9]

Assumptions. The triage rota can absorb about ten issues a day; some flaky tests hide real races; some flakes depend on time or test order.

Analysis. Option A looks best on the dashboard: one retry cuts false reds from 63% to 0.1% (worked calculation 1), but only for independent flakes. A same-process retry repeats the failing conditions, so time- and order-dependent flakes stay red while real intermittent bugs can turn green; GitHub’s earlier methods, a single re-run among them, found only a quarter of its flaky failures (see the FAQ). Google noted that marking tests flaky “encourages developers to ignore flakiness in their own tests”,^[8] and JUnit XML can hide the retry (Lab 2). Option B keeps main honest but removes coverage fast; at 30 teams it can disable dozens of tests a week.

Decision. A hybrid. Retry once on pull requests and in the merge queue to unblock people, storing every flaky outcome as an issue with an owner and a deadline. Classify with evidence, not that one retry: on main, a failure gets targeted retries^[37] and the flake detector uses same-commit history. Quarantine is capped and expires (Lab 3). Re-evaluate after a month using the false-red share and escaped defects. Uncertainty: if escaped defects rise in areas covered by quarantined tests, the cap or expiry is too loose.

You should now be able to explain the label’s status, compute the effect of flakiness, design a stage map with budgets, run a capped quarantine, choose between retry, quarantine, push-down and deletion, and set a test-signal SLO.

The word “TestOps” is young, inconsistent and often a product name. The problem it points at is old and costly: a suite that has quietly become slow, noisy and expensive, so people stop believing it. Run the suite like a service: give its verdict service levels, keep attempt-level data, give every lying test an owner and a deadline, and spend compute where it buys trust. A team that does this does not need the label; a team that adopts the label without the discipline has bought a dashboard.

Further reading: Playwright retries; SRE book: Testing for Reliability.

Changelog: 2026-10-16, first version (evidence ledger: research/testops-ledger.md). 2026-10-16, revision after review r1 (reviews/testops-r1-response.md): triage flow, lifecycle, gate job and test ids corrected; environments, caching, shift-right, ISTQB and flaky-test research added; labs re-executed. 2026-10-17, revision after scoring (reviews/testops-r2-response.md): Fundamentals table repaired; end-to-end quarantine wired; parent re-run, citations and preconditions tightened. 2026-10-17, post-score polish (reviews/testops-polish.md): Keysight usage, attribution, commit stamping, test-data playbook, Figures 2 and 3 fixed; Roles: DORA’s hedge restored.