MobileOps

MobileOps is a rarely used, informal name for running the delivery of native mobile apps: building and signing iOS and Android binaries, getting them through app-store review, releasing them in stages, watching crashes per version, and keeping the old versions that people still run working. The practice is usually called mobile DevOps, which Wikipedia describes as applying “the principles of DevOps specifically to the development of mobile applications”^[1], or mobile release engineering.

In one line: on the web you can roll back; on a phone you cannot recall a binary. Once users update to a version with a bug, Gergely Orosz writes, “they are stuck with that bug until a new version is released and this user updates”^[2]. MobileOps is the habit of shipping as if that were true, because it is.

It is for mobile and release engineers, platform teams, backend owners and engineering leaders.

Mobile delivery differs from server delivery in three ways that no tool removes:

1. Someone else controls the last step. Apple says App Review typically reviews “at least 50% of submissions in less than 24 hours and 90% in less than 48 hours”^[7]. Google says processing “can take a few hours or up to seven days (or longer in exceptional cases)”^[8]. A one-line fix cannot reach users in minutes.
2. You cannot take a version back. When you halt a Google Play staged rollout, “Users who already received the app version … will remain on that version”^[9]. Apple lets you pause a phased release, but a pause stops only automatic updates: the version “can be manually downloaded from the App Store by anyone at any time”^[10].
3. Many versions run at once. Old versions remain “up to a number of years”, and “Even a non-breaking backend change can break an older version of the app”^[2].

Ignoring them costs hotfix churn, expired-certificate fire drills, back-end changes that break last year’s app and, on Android, Play may reduce the visibility of apps above its crash and ANR thresholds^[11].

• Not a recognised discipline or job title. Engineers doing this work are called mobile release engineers, mobile platform engineers or “mobile DevOps” engineers.
• Not mobile device management. Enrolling and policing corporate phones is EndpointOps, despite the name collision^[3].
• Not DevOps with a phone icon. The CI/CD and DORA baseline is DevOps; this page adds only what changes on mobile.
• Not “mobile testing”. Device labs and test selection are here; whether a test result can be trusted is TestOps.

Mobile delivery grew up inside the stores’ walled gardens: compiled binaries, mandatory signing and review. Wikipedia dates the rise of the term “mobile DevOps” to 2014–2015, citing Google Trends^[1]. Traceable milestones:

Mental model: two layers and three clocks. The binary layer is slow, gated and cannot be recalled; the server layer (config, flags, APIs, minimum versions) changes in minutes. MobileOps moves risk from the first layer to the second. Three clocks set the pace: the build clock (yours: CI and tests), the store clock (theirs: review) and the adoption clock (your users’ update habits). You can shorten the first, plan around the second (submit early, release when ready) and only influence the third.

flowchart TB
  D["DevOps"]:::neutral -->|"CI/CD baseline"| M
  R["ReleaseOps"]:::neutral -->|"rollout and\nflag strategy"| M
  T["TestOps"]:::neutral -->|"trusted test\nresults"| M
  E["EndpointOps"]:::risk -.-|"name clash only"| M
  M["MobileOps:\nstore-gated, installed binaries"]:::core
  M -->|"support window"| A["API Ops"]:::neutral
  M -->|"new SDK or\ndata type"| P["PrivacyOps"]:::neutral
  M -->|"signing keys"| S["DevSecOps"]:::neutral
  M -->|"per-version gates"| O["ObservabilityOps"]:::neutral
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029

This is this Atlas’s proposed charter; no official MobileOps charter exists.

Mission: every release reaches users in stages, every bad build is contained while few users have it, and every installed version is either supported or deliberately retired.

1. Assume there is no rollback. Before shipping, know how you will turn the feature off, stop the rollout and move users on.
2. Fast levers on the server, slow change in the binary. Flags, config and minimum versions change in minutes.
3. Decide gates before the rollout. Numeric, per-version and written down; anyone on the rotation may halt.
4. Support a window, not every version. Publish which versions you support; retire the rest on purpose.
5. Keep signing and store access boring. Inventoried, least-privilege, renewed on a calendar.
6. Measure with consent, and know what you cannot see.
7. Treat store rules as design constraints.

Non-goals: forcing every user onto the latest version; replacing store review; building a home-grown way to swap native code.

Eight blocks recur: signing identities (loss or expiry blocks every release); a build pipeline with macOS for iOS (Xcode 27.1 needs macOS Tahoe 26.6 or later^[25]); distribution channels; rollout controls, the only brake on the binary layer; per-version signals; server levers that act without review; a support window; and an API contract for every supported version.

R = responsible, A = accountable, C = consulted, I = informed. A small team can merge the captain and platform roles; it should not merge store administration into everyday CI credentials.

• Day 0 — before the first release. Set up store roles; enrol in Play App Signing; keep Apple signing identities in a shared, encrypted store; build in CI; add crash reporting with a consent decision and remote config with in-app defaults; ship a minimum-version check in version 1, since in-app updates benefit “only future versions of your app”^[26].
• Day 1 — first release. Neither store stages a first release (levers): rely on TestFlight and testing tracks; Apple reviews the first build sent to external testers^[27].
• Day 2 — operating. Run the train, gate rollouts, manage the tail, renew credentials, and absorb the stores’ floors: since 2026-04-28 uploads to App Store Connect need Xcode 26 and the iOS 26 SDK, and since 2026-09-09 iOS apps must target iOS 13 or later^[28]; since 2026-08-31 Google Play updates must target Android 16 (API level 36) — extension to 2026-11-01 possible; un-updated existing apps face an API 35 floor^[29].

stateDiagram-v2
  direction TB
  [*] --> Built
  Built --> Beta: testers install
  Beta --> InReview: submitted
  InReview --> Beta: rejected, fix
  InReview --> Approved: approved
  Approved --> Rollout: released when ready
  Rollout --> Halted: gate fails
  Halted --> Rollout: resume
  Halted --> Superseded: fix ships as N+1
  Rollout --> Current: 100 percent
  Current --> Supported: newer version ships
  Supported --> Prompted: below soft minimum
  Prompted --> Blocked: below hard minimum
  Blocked --> [*]: no active users
  Superseded --> Prompted: move users on

• Cadence. A weekly or fortnightly train. Approval is not release: submit early, hold the approved build (Apple’s manual release, Play’s managed publishing^[30][8]) and release when the back end and flags are ready. Squarespace submits on Friday and rolls out on Monday^[14]. A hotfix lane needs written entry criteria (crash or data-loss regressions, security, legal).
• Gates. Agreed before the train starts; halting needs no permission, resuming needs data. Squarespace’s captains “weren’t sure when a problem was serious enough to halt a rollout” until numeric heuristics existed^[14].
• Access. Quarterly role review; CI identities scoped as in the reference implementation; a documented break-glass admin.
• Calendar. Certificate expiry, store deadlines and tool retirements in one place.
• Support window. Monthly tail review with backend owners; minimum versions move by decision.
• Escalation. A harmful release becomes an incident run with IncidentOps roles; the captain keeps the levers.

Four parts: build and sign, distribute, the installed fleet, and a server side that keeps working while binaries wait for review.

flowchart TB
  SRC["Repository and lanes"] --> CI["CI: macOS and Linux"]
  SIG["Signing store"] -->|"read-only"| CI
  CI --> BETA["Beta channels"]
  BETA --> REV["Store review"]
  REV --> ROLL["Staged or phased rollout"]
  ROLL --> FLEET["Installed fleet: N, N-1, older tail"]
  subgraph CTRL["Server side: changes in minutes"]
    CFG["Flags and config"]
    MIN["Minimum versions"]
    API["Versioned APIs"]
  end
  CTRL --> FLEET
  FLEET --> CR["Crash and vitals"]
  ROLL <-.-|"halt or advance"| CR
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  class ROLL,MIN core
  class REV risk
  class CR decision

flowchart TB
  A["Branch cut on train day"] --> B["CI builds, tests, signs"]
  B --> C["Beta: staff and testers"]
  C --> D{"Store review"}
  D -->|"rejected"| B
  D -->|"approved"| R["Held until back end and flags are ready"]
  R --> E["Release a step: 1 percent first"]
  E --> W["Wait, then read per-version metrics"]
  W --> G{"Gate"}
  G -.->|"too little data: hold"| W
  G -->|"clear, below 100 percent"| E
  G -->|"clear at 100 percent"| T["Watch the tail, move minimums"]
  G -->|"regression"| X["Flag off, then halt or pause"]
  X --> Y{"Fixable on the server?"}
  Y -->|"yes: fix, then resume"| W
  Y -->|"no"| K["Hotfix N+1, expedited review"]
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  class D,G,Y decision
  class X,K risk
  class R,T core

Rehearse three exception paths: a rejection on launch day (Apple offers expedited review for critical bugs^[7]); a regression found after 100 percent (Play can halt a fully rolled-out release^[31]; Apple says a live version with a usability issue needs an app update^[32]); and a server or SDK change that crashes every installed version at once.

A soft minimum prompts; a hard minimum blocks until the user updates. Both need code that is already installed, with one exception on Android.

• Google Play. The in-app updates library offers a flexible flow (download in the background) and an immediate full-screen flow, on Android 5.0 and later; the app can choose between them from an update priority (0–5) set at rollout^[33][34]. Play Console’s Prompt users to update recovery tool shows users of a chosen version a full-screen prompt with no code change; dismissed, it returns “after every cold restart”. It needs Play App Signing and app bundles and is unavailable with some features (for example Code Transparency)^[26].
• App Store. In the Apple documentation checked in October 2026 there is no equivalent in-app update API (Orosz made the same observation in 2020^[2]). Teams fetch a minimum version from their own config, compare it with the installed version, and link to the App Store page.
• Both. The version-check response must stay readable by the oldest version ever shipped, or blocked users see a crash instead of a message. Compare versions numerically (Lab 2).
• Block only when the replacement reaches everyone you block. A Play staged rollout reaches “only a percentage of your users”, chosen at random, in the countries you target^[9]; the rest cannot update yet, and devices below its minimum OS never can. Block only after the replacement’s Play rollout completes in every targeted country, and give users who still cannot update a soft prompt or another channel. On iOS anyone can download a phased release manually^[10], so a store link works.
• Why bother. MASVS control MASVS-CODE-2 reads “The app has a mechanism for enforcing app updates”^[35]: a security fix works only once users run it.

Apple’s licence agreement says an app “may not download or install executable code”; interpreted code may be downloaded only if it does not change the app’s “primary purpose”, does not “bypass signing, sandbox, or other security features of the OS”, and does not create a storefront for other apps^[36]; the App Review Guidelines add code “which introduces or changes features or functionality”^[37]. Google Play forbids updating “using any method other than Google Play’s update mechanism” and downloading executable code such as dex, JAR or .so files, but exempts code running in “a virtual machine or an interpreter”^[38].

OTA tools such as Expo’s EAS Update replace JavaScript, styling and images, not native code, permissions or SDK versions, and “your updates need to follow the App Store and Play Store guidelines”^[24]. Each update must match the build’s runtime version; an incompatible update calls “code that does not exist inside the build”^[39]. Microsoft’s CodePush survives App Center only as a self-hosted “special version”^[16]. Use OTA for bug fixes in interpreted code and keep behaviour changes in reviewed releases.

Client flag SDKs fetch flags in bulk for one user, cache them, and evaluate against the cache — OpenFeature calls this the static-context paradigm^[40]. Consequences:

• A kill switch is not instant. Firebase Remote Config’s default minimum fetch interval is 12 hours^[41]; a strategy Firebase describes as effective activates fetched values only at the next start, while real-time updates bypass the interval^[42]. Measure propagation.
• Defaults are policy. Until its first successful fetch the app runs its in-app defaults^[42]; a risky feature should default to off.
• Flags do not hide features from review. Apple requires new features to be “described with specificity in the Notes for Review … and accessible for review” (guideline 2.3.1(a))^[37].
• Flags accumulate. Give each an owner and expiry; ReleaseOps covers flag debt.

As of October 2026; recheck before relying on a limit.

No standard defines MobileOps. Separate three kinds of guidance:

• Binding rules: Apple’s licence agreement and App Review Guidelines^[36][37]; Google Play’s policies^[38]. Breaking them blocks releases or removes apps.
• Standards: OWASP MASVS, which OWASP calls “the industry standard for mobile app security”^[45]; Google’s AIP-180 on backwards compatibility, written for APIs whose consumers update on their own schedule^[46] — exactly a mobile back end’s situation (this page’s inference); the DORA metrics (DevOps).
• Practitioner patterns: release trains and rotating captains^[14], daily beta builds^[12]: single-organisation experience, not norms.

Tool categories first; names are examples verified on 2026-10-18, not endorsements.

Lock-in and churn are real. App Center’s crash reporting ends in March 2027 (Azure Monitor is the migration path)^[16]; Firebase Test Lab stops on 2027-09-30^[17]. Keep release logic in your own lanes and scripts so that a vendor change swaps one step, not the process.

Automate what is deterministic: build numbers, signing (read-only), uploads, symbol files, gate evaluation, advancing and halting. Squarespace’s automation reads Crashlytics and Amplitude data and “can automatically halt the rollout”^[14]. Keep a person on resuming a halted rollout and on moving a hard minimum.

AI may group crash reports and draft release notes; a person checks the notes, since Apple requires significant changes in “What’s New” (guideline 2.3.12)^[37]. Google’s Developer Device Platform FAQ lists “Agent skills” — “Help AI coding agents find, reserve, and interact with devices for you” — with no launch date given (page seen October 2026)^[17].

Assumptions: a consumer app on iOS and Android, about 400,000 daily users, a weekly train, per-version crash data, remote config and GitHub-hosted runners. Four artefacts: a rollout policy, a support policy served to apps, release lanes and a rollout script.

Example status: YAML syntax-checked by the Atlas validator, not executed.

# release-policy.yaml: reviewed by the mobile platform team each quarter
app: com.example.shop
platforms:
  android:
    steps: [0.01, 0.05, 0.20, 0.50, 1.0]   # Play userFraction per step
    min_hours_per_step: 24
  ios:
    phased_release: true                   # Apple's fixed 7-day schedule
    pause_budget_days: 30                  # Apple's total pause limit
gates:                                     # compared with the previous version
  min_users_per_step: 5000                 # delays ADVANCE only, never a HALT
  crashed_session_rate_max_ratio: 1.5
  anr_rate_max_ratio: 1.5                  # Android only
  business_events:
    - name: checkout_completed_per_session
      max_drop_pct: 5
halt_rights: [release-captain, on-call-mobile, feature-owner]
resume_requires: [release-captain, mobile-platform]

The support policy lives at a stable URL that every version ever shipped can read: never remove or rename a field (AIP-180 treats both as breaking^[46]), compare versions numerically, and list known-bad versions.

Example status: JSON syntax-checked; the store URLs are placeholders.

{
  "schema": 1,
  "updated": "2026-10-18",
  "platforms": {
    "ios": {
      "hard_min": "5.9.0", "soft_min": "6.8.0", "blocked": ["6.4.0"],
      "store_url": "https://apps.apple.com/app/id0000000000"
    },
    "android": {
      "hard_min": "5.9.0", "soft_min": "6.8.0", "blocked": [],
      "store_url": "https://play.google.com/store/apps/details?id=com.example.shop"
    }
  },
  "message_key": "update_required_v1"
}

The lanes build, upload crash symbols and ship to testers. CI never creates certificates (readonly, as fastlane recommends for CI^[50]); upload_symbols_to_crashlytics does not fail the build on an upload error unless told to^[55].

Example status: Ruby, not syntax-checked (no Ruby interpreter was available), not executed; needs Apple and Google accounts.

# fastlane/Fastfile
platform :ios do
  lane :beta do
    app_store_connect_api_key(
      key_id: ENV["ASC_KEY_ID"],
      issuer_id: ENV["ASC_ISSUER_ID"],
      key_filepath: ENV["ASC_KEY_PATH"]
    )
    match(type: "appstore", readonly: true)
    increment_build_number(build_number: ENV["BUILD_NUMBER"])
    build_app(scheme: "Shop", export_method: "app-store")
    upload_symbols_to_crashlytics(
      dsym_path: lane_context[SharedValues::DSYM_OUTPUT_PATH],
      gsp_path: ENV["GSP_PATH"],
      fail_on_error: true
    )
    upload_to_testflight(skip_waiting_for_build_processing: true)
  end
end

platform :android do
  lane :internal do
    gradle(task: "bundle", build_type: "Release")
    # Testing-track identity only; Play reads the R8 mapping file from the bundle
    upload_to_play_store(track: "internal", json_key: ENV["PLAY_JSON_KEY_PATH"])
  end
end

The rollout script changes one release through the Publishing API’s edit flow (insert, change, commit), sending only that release, since in an update request the releases field carries only the desired changes^[56][57]. It rejects fractions outside 0–1 (a typed 50 would otherwise complete the rollout) and has a dry run; advance on a halted release resumes it.

Example status: Python syntax-checked; argument checks and the edit sequence executed 2026-10-18 against a stubbed API client; not run against Google Play (needs a Play Console app and an identity with release permission).

"""Advance or halt one Google Play release (Publishing API v3).

Usage: python3 play_rollout.py PACKAGE VERSION_CODE advance FRACTION [--dry-run]
       python3 play_rollout.py PACKAGE VERSION_CODE halt [--dry-run]
Options: --track NAME (default production). Credentials: Application Default
Credentials (GOOGLE_APPLICATION_CREDENTIALS) supplied by CI; never commit keys.
"""
import argparse
import json
import sys

import google.auth
from googleapiclient.discovery import build

SCOPES = ["https://www.googleapis.com/auth/androidpublisher"]


def fraction_arg(text):
    value = float(text)
    if not 0 < value <= 1:
        raise argparse.ArgumentTypeError("use a fraction above 0 and up to 1, e.g. 0.05")
    return value


def parse_args(argv):
    parser = argparse.ArgumentParser(description="Advance or halt a Play release.")
    parser.add_argument("package")
    parser.add_argument("version_code")
    parser.add_argument("action", choices=["advance", "halt"])
    parser.add_argument("fraction", nargs="?", type=fraction_arg)
    parser.add_argument("--track", default="production")
    parser.add_argument("--dry-run", action="store_true")
    args = parser.parse_args(argv)
    if (args.action == "advance") != (args.fraction is not None):
        parser.error("advance needs FRACTION; halt takes none")
    return args


def change(release, args):
    if args.action == "halt":
        release["status"] = "halted"
    elif args.fraction == 1:
        release["status"] = "completed"
        release.pop("userFraction", None)
    else:
        release["status"] = "inProgress"
        release["userFraction"] = args.fraction
    return release


def main(argv):
    args = parse_args(argv)
    creds, _ = google.auth.default(scopes=SCOPES)
    edits = build("androidpublisher", "v3", credentials=creds).edits()
    ids = {"packageName": args.package}
    ids["editId"] = edits.insert(body={}, **ids).execute()["id"]
    track = edits.tracks().get(track=args.track, **ids).execute()
    found = [r for r in track.get("releases", [])
             if args.version_code in r.get("versionCodes", [])]
    if len(found) != 1 or args.dry_run:
        edits.delete(**ids).execute()
    if len(found) != 1:
        sys.exit("version code %s is not on track %s" % (args.version_code, args.track))
    release = change(found[0], args)
    print(json.dumps(release))
    if args.dry_run:
        print("dry run: edit deleted, nothing committed")
        return 0
    # Send only the changed release: in an update, releases are the desired changes.
    body = {"track": args.track, "releases": [release]}
    edits.tracks().update(track=args.track, body=body, **ids).execute()
    edits.commit(**ids).execute()
    print("committed: %s %s %s on %s" % (
        args.package, args.version_code, args.action, args.track))
    return 0


if __name__ == "__main__":
    sys.exit(main(sys.argv[1:]))

Do not edit the app in Play Console while the script runs: Google discards an API edit if anyone changes the app in the console meanwhile^[57].

Who can reach which credential.

• App Store Connect. “Team API keys can access all apps, regardless of their role”^[58]. The tokens fastlane signs last at most 20 minutes^[59]; the key file is the risk, so keep it out of repositories and revoke it if it leaks. Give the beta lane a Developer-role key, which may upload builds^[60], and only the protected release job an App Manager key, which phased-release changes require^[10].
• Google Play. fastlane’s setup recommends “Admin (all permissions)”^[61]; grant app-level permissions instead: “Release apps to testing tracks” for CI, “Release to production, exclude devices, and use Play App Signing” only for the approved release job^[62].
• Signing. readonly stops match creating certificates, but “Every team member with access to the selected storage can use those credentials for code signing”^[50]: give signing jobs read-only storage access and the passphrase, and keep both out of pull-request builds.
• Key files. Inject keys from the CI secret store at run time. The script’s Application Default Credentials can use a Workload Identity Federation configuration instead of a service-account key, which Google calls “a security risk”^[63] (not tested with the Play API here).

Labs 1 and 2 run locally with Python 3.8 or later, need no accounts, cost nothing and change nothing outside their folder.

Objective: turn per-version crash numbers into ADVANCE, HOLD or HALT, with a statistical guard against small samples. Prerequisites: Python 3.8+; 30 minutes.

1. Save the script as rollout_gate.py and the data as rollout.json in an empty folder.
2. Run python3 rollout_gate.py rollout.json; echo "exit=$?".
3. Change the candidate’s crashed_sessions to 70 and users_with_crash to 61; run again.
4. Change the candidate to "sessions": 9000, "crashed_sessions": 17, "users": 3400, "users_with_crash": 15; run again.
5. Change it to "sessions": 18000, "crashed_sessions": 49, "users": 4000, "users_with_crash": 41; run again.
6. Replace the numbers with your crash reporter’s per-version export for the last 24 hours.

Example status: Python and JSON syntax-checked; executed 2026-10-18 with Python 3.8.10; outputs below are copied from the run.

"""Decide ADVANCE, HOLD or HALT for one staged-rollout step.

Compares the candidate version's crashed-session rate with the
baseline version's, using a Wilson score interval. Exit codes:
0 = ADVANCE, 1 = HOLD, 2 = HALT (usable as a CI gate).
"""
import json
import math
import sys


def wilson(events, trials, z):
    """Wilson score interval for a proportion events/trials."""
    if trials == 0:
        return 0.0, 1.0
    p = events / trials
    denom = 1 + z * z / trials
    centre = (p + z * z / (2 * trials)) / denom
    half = z * math.sqrt(p * (1 - p) / trials + z * z / (4 * trials * trials))
    return centre - half / denom, centre + half / denom


def decide(data):
    base, cand, pol = data["baseline"], data["candidate"], data["policy"]
    base_rate = base["crashed_sessions"] / base["sessions"]
    cand_rate = cand["crashed_sessions"] / cand["sessions"]
    lo, hi = wilson(cand["crashed_sessions"], cand["sessions"], pol["z"])
    limit = base_rate * pol["relative_risk_halt"]
    print("baseline %s crash-free sessions %.3f%%  users with crash %.3f%%" % (
        base["version"], 100 * (1 - base_rate),
        100 * base["users_with_crash"] / base["users"]))
    print("candidate %s crash-free sessions %.3f%%  users with crash %.3f%%" % (
        cand["version"], 100 * (1 - cand_rate),
        100 * cand["users_with_crash"] / cand["users"]))
    print("candidate crashed-session rate %.4f%% (95%% CI %.4f%%-%.4f%%)" % (
        100 * cand_rate, 100 * lo, 100 * hi))
    print("halt limit (baseline x %.2f) %.4f%%" % (
        pol["relative_risk_halt"], 100 * limit))
    # HALT first: the interval already allows for a small sample, and a
    # minimum-users rule must never suppress a halt.
    if lo > limit:
        return "HALT", "regression is clear: lower bound above the limit"
    if cand["users"] < pol["min_users"]:
        return "HOLD", "only %d users; policy needs %d to advance" % (
            cand["users"], pol["min_users"])
    if hi <= limit:
        return "ADVANCE", "upper bound within the limit"
    return "HOLD", "inconclusive: keep this step and collect more data"


def main(path):
    with open(path) as f:
        data = json.load(f)
    verdict, reason = decide(data)
    print("decision: %s (%s)" % (verdict, reason))
    return {"ADVANCE": 0, "HOLD": 1, "HALT": 2}[verdict]


if __name__ == "__main__":
    sys.exit(main(sys.argv[1] if len(sys.argv) > 1 else "rollout.json"))

{
  "baseline":  {"version": "6.3.2", "sessions": 1840000, "crashed_sessions": 2208,
                "users": 410000, "users_with_crash": 1640},
  "candidate": {"version": "6.4.0", "sessions": 52000, "crashed_sessions": 140,
                "users": 19800, "users_with_crash": 118},
  "policy":    {"min_users": 5000, "relative_risk_halt": 1.5, "z": 1.96}
}

Expected output (step 2):

baseline 6.3.2 crash-free sessions 99.880%  users with crash 0.400%
candidate 6.4.0 crash-free sessions 99.731%  users with crash 0.596%
candidate crashed-session rate 0.2692% (95% CI 0.2282%-0.3176%)
halt limit (baseline x 1.50) 0.1800%
decision: HALT (regression is clear: lower bound above the limit)
exit=2

Step 3 prints decision: ADVANCE (upper bound within the limit), exit 0; step 4 prints decision: HOLD (only 3400 users; policy needs 5000 to advance), exit 1; step 5 (the worked example’s day-1 cohort) prints candidate crashed-session rate 0.2722% (95% CI 0.2060%-0.3597%) and decision: HALT, exit 2. Validation: 99.73% crash-free looks healthy, yet the version crashes more than twice as often as its predecessor; step 5 halts below min_users because the interval already allows for the small sample. Troubleshooting: KeyError = a field name differs from your export; ZeroDivisionError = an empty baseline. Caveats: one user’s sessions are not independent, so the interval is optimistic; early updaters and first-day sessions differ from a steady-state baseline, so compare the same day and time since update where you can. Teardown: delete the folder.

Objective: see who a soft and a hard minimum would affect, and which old endpoints can be retired. Prerequisites: Lab 1’s setup; 30 minutes.

1. Save the script as version_tail.py and the data as tail.json.
2. Run python3 version_tail.py tail.json; echo "exit=$?".
3. Raise hard_min to 6.5.0 and run again: both endpoints become retirable (exit 0), but 53,000 users (10.6%) are blocked.
4. Replace the data with your analytics’ weekly active users per version for one platform.

Example status: Python and JSON syntax-checked; executed 2026-10-18 with Python 3.8.10 (steps 2 and 3).

"""Report the old-version tail and what a minimum-version policy would do."""
import json
import sys


def vkey(text):
    """'6.10.0' -> (6, 10, 0). Never compare version strings as text."""
    return tuple(int(part) for part in text.split("."))


def main(path):
    with open(path) as f:
        data = json.load(f)
    users = data["weekly_active_users"]
    pol = data["policy"]
    total = sum(users.values())
    hard, soft = vkey(pol["hard_min"]), vkey(pol["soft_min"])
    print("text comparison says 6.10.0 < 6.9.0: %s (wrong)" % ("6.10.0" < "6.9.0"))
    print("tuple comparison says 6.10.0 < 6.9.0: %s" % (vkey("6.10.0") < vkey("6.9.0")))
    blocked = prompted = 0
    for version in sorted(users, key=vkey, reverse=True):
        n = users[version]
        if vkey(version) < hard:
            action, blocked = "BLOCK", blocked + n
        elif vkey(version) < soft:
            action, prompted = "PROMPT", prompted + n
        else:
            action = "ok"
        print("%-8s %8d  %5.1f%%  %s" % (version, n, 100.0 * n / total, action))
    print("blocked by hard minimum %s: %d users (%.1f%%)" % (
        pol["hard_min"], blocked, 100.0 * blocked / total))
    print("prompted by soft minimum %s: %d users (%.1f%%)" % (
        pol["soft_min"], prompted, 100.0 * prompted / total))
    status = 0
    for ep in data["endpoints"]:
        # Blocked versions only reach the version-check endpoint, so they
        # stop counting for other endpoints once the hard block is live.
        still = sum(n for v, n in users.items()
                    if hard <= vkey(v) < vkey(ep["needed_below"]))
        share = 100.0 * still / total
        ok = share <= pol["retire_budget_pct"]
        print("%s still needed by %.1f%% of users: %s" % (
            ep["path"], share, "can retire" if ok else "keep serving"))
        status = status if ok else 1
    return status


if __name__ == "__main__":
    sys.exit(main(sys.argv[1] if len(sys.argv) > 1 else "tail.json"))

{
  "weekly_active_users": {
    "6.10.1": 205000, "6.10.0": 92500, "6.9.2": 71000, "6.8.0": 45500,
    "6.5.3": 33000, "6.2.0": 22000, "5.9.1": 16000, "5.4.0": 9000,
    "4.12.0": 6000
  },
  "policy": {"hard_min": "5.9.0", "soft_min": "6.8.0", "retire_budget_pct": 1.0},
  "endpoints": [
    {"path": "/v1/cart", "needed_below": "6.5.0"},
    {"path": "/v1/login-legacy", "needed_below": "5.0.0"}
  ]
}

Expected output (step 2):

text comparison says 6.10.0 < 6.9.0: True (wrong)
tuple comparison says 6.10.0 < 6.9.0: False
6.10.1     205000   41.0%  ok
6.10.0      92500   18.5%  ok
6.9.2       71000   14.2%  ok
6.8.0       45500    9.1%  ok
6.5.3       33000    6.6%  PROMPT
6.2.0       22000    4.4%  PROMPT
5.9.1       16000    3.2%  PROMPT
5.4.0        9000    1.8%  BLOCK
4.12.0       6000    1.2%  BLOCK
blocked by hard minimum 5.9.0: 15000 users (3.0%)
prompted by soft minimum 6.8.0: 71000 users (14.2%)
/v1/cart still needed by 7.6% of users: keep serving
/v1/login-legacy still needed by 0.0% of users: can retire
exit=1

Validation: the hard minimum, not time, is what lets you retire /v1/login-legacy. Troubleshooting: ValueError on a version such as 6.10.0-beta — strip suffixes before comparing. Teardown: delete the folder.

Objective: wire Lab 1’s exit code to play_rollout.py. Prerequisites: a staged release on a closed testing track (Play allows staged rollouts on test tracks^[9]); a CI identity with “Release apps to testing tracks” for that app only^[62].

1. Schedule a CI job that runs python3 rollout_gate.py export.json.
2. On exit code 2, run python3 play_rollout.py com.example.shop 6400 halt --track <closed-track> --dry-run, then without --dry-run.
3. Feed the job Lab 1’s step-5 numbers.

Expected output: the dry run prints the release with "status": "halted"; the real run prints committed: com.example.shop 6400 halt on <closed-track>. Validation: Play Console shows the test release as halted. Teardown: resume it with advance and its previous fraction, or discard the test release; delete the job.

Four documented cases, each a single organisation’s account:

Worked calculation: how much a rollout step can see. Assume 400,000 users with automatic updates on, 4.5 sessions per user per day, a previous version with a crashed-session rate of 0.12%, and a halt limit of 1.5× (0.18%), evaluated with Lab 1’s 95% interval on one day of data per step (illustrative inputs, computed 2026-10-18):

A large regression shows on day one; a moderate one only after 5% of users have it. A small app may never have enough users at 1%: size steps by users, not percentages. Between steps, Play’s Reporting API offers hourly crash rates by versionCode^[52]. On iOS, MetricKit sends metric reports “at most once per day”, but “Diagnostic reports arrive immediately in iOS 15 and later”, crash diagnostics included^[54]; they arrive on the device, so the app or a crash SDK must upload and aggregate them before a gate can use them. Crashlytics velocity alerts are not a gate: they are not sent by default, and the default threshold, one issue in 1% of sessions and 25 users within 30 minutes^[67], is more than five times this policy’s whole-version limit.

What the numbers cannot see. Android vitals collects data only “When a user allows it”^[11], Xcode’s Crashes organizer only from “customers who share diagnostic and usage information”^[53], and opt-in reporting makes Crashlytics metrics “less reliable”^[51]. Compare versions with each other, not with an absolute target. For DORA measures, define a mobile “deployment” (this Atlas suggests: a version reaching 100% of its rollout); see DevOps.

• Signing keys. On Google Play, a lost upload key can be reset, but a self-managed app signing key “cannot be reset” if lost ^[19]. On Apple, an expired App Store distribution certificate leaves live apps untouched but blocks new uploads; an expired in-house (enterprise) certificate stops apps from running; an expired push certificate stops notifications^[68]. Who may reach each key is set out under the reference implementation.
• Security baseline. OWASP MASVS groups controls from storage to privacy^[45]; MASVS-CODE-1 requires an up-to-date platform version and MASVS-CODE-2 an update-enforcement mechanism^[35]. Security testing belongs to DevSecOps.
• Privacy of telemetry. Crash logs and diagnostics are data types to declare in Google Play’s Data safety form, including data sent by SDKs^[69]; on Apple, apps and SDKs declare collected data in a PrivacyInfo.xcprivacy manifest^[70]. MASVS lists “Insertion of Sensitive Data into Logs” as a weakness (MASWE-0005)^[45]. Consent: PrivacyOps.
• Store compliance. Apple may remove apps that are not compatible with the current OS release (licence section 6.8)^[36].

CI. On 2026-10-18 GitHub listed $0.062 a minute for a standard macOS runner and $0.006 for a 2-core Linux runner^[71]. Thirty iOS builds a day at 20 minutes for 22 working days is 13,200 minutes: about $818 a month on macOS against about $79 on Linux (illustrative arithmetic); Xcode Cloud includes 25 compute hours a month^[48]. Cache builds, build iOS only when iOS code changes, and run device suites on a schedule.

Devices. Firebase Test Lab charges $5 an hour per physical and $1 per virtual device beyond its free allowance^[72]; its successor matches those rates only until 2027-04-30^[17]. Flaky tests multiply the bill; Slack’s mobile teams (16,000+ Android and 11,000+ iOS tests) noted they “increase execution cost”^[73].

People. Squarespace estimated losing “nearly a third” of engineering time to release coordination before automating its train^[14]. Fewer redundant builds and device runs also cut energy use.

• Each release: a release record (template below) with gate values at each step and who advanced, halted or resumed.
• Monthly: review every halt — did a gate fire before support tickets arrived, and were any halts false alarms?
• Quarterly: credential and role audit; a kill-switch drill measuring propagation; a signing-store restore test; the device matrix rebuilt from usage.
• Yearly: platform-floor planning for Apple’s SDK requirement and Google’s target API level.

Trigger (illustrative): a velocity alert for 6.4.0 two hours after the Android 5% step and Apple’s day 3. Roles: release captain (levers), feature owner (flags), backend on-call. First question: which versions crash?

flowchart TB
  A["Alert: crash spike"] --> B{"Which versions crash?"}
  B -->|"new version only"| C["Flag off if feature-linked"]
  C --> D["Halt Play rollout, pause Apple phased release"]
  D --> E{"Fixed by flag or server?"}
  E -->|"yes"| F["Verify, resume at same step"]
  E -->|"no"| G["Hotfix N+1, expedited review"]
  B -->|"all versions"| H["Server, config or SDK change"]
  H --> I["Revert change, contact SDK vendor"]
  I --> J["Verify crash rate back to baseline"]
  G --> K["Block bad version once N+1 reaches all"]
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  class A,H risk
  class B,E decision
  class F,J,K core

1. Scope (5 minutes). Crash groups by version and OS: here 6.4.0 only, in the new checkout screen, on both platforms.
2. Flag off. The feature owner sets new_checkout=false; note the time.
3. Stop the binary. Android: python3 play_rollout.py com.example.shop 6400 halt. iOS: pause the phased release in App Store Connect, or send the body below with PATCH /v1/appStoreVersionPhasedReleases/{id}^[44]; manual downloads continue. Removing the app from sale ends the phased release: reinstated, the version “becomes available to all users immediately, regardless of the percentage reached before removal”, and only a new version restores a phased rollout^[10]. Users who already have 6.4.0 keep it^[56].
4. Verify (30–60 minutes). 6.4.0’s crash rate back near the baseline? Then resume at the same step with the flag off and fix forward in the next train.
5. If not: build 6.4.1 and request expedited review with steps to reproduce^[7]. On Play, a new staged release started before the previous one completes “will use the same group of users as the previous release (depending on the percentage of the rollout)”^[9], so start 6.4.1 at 6.4.0’s percentage, not lower (this page’s reading). Soft-prompt 6.4.0 users; add 6.4.0 to blocked only when every 6.4.0 user can get 6.4.1 — on Android, once 6.4.1 has completed its rollout (forced upgrades).
6. Communicate. Give support a status line; if users were harmed, open an incident (IncidentOps).
7. Record. Alert, flag and halt times and decisions; follow-up on why the beta and the 1% step missed it.

Example status: JSON syntax-checked; fields compared by hand with Apple’s documented request schema; not sent (needs an API key).

{
  "data": {
    "type": "appStoreVersionPhasedReleases",
    "id": "<phased-release-id>",
    "attributes": {"phasedReleaseState": "PAUSED"}
  }
}

• “We’ll roll back if it breaks.” Halting stops new installs only; plan the flag, the fix-forward and the block in advance.
• “99.7% crash-free is fine.” It may be twice as bad as the last version (Lab 1); compare, do not admire.
• “Flags or OTA let us skip review.” New features must be described and accessible to reviewers, and OTA is limited to interpreted code that does not change the app’s purpose^[37][36].
• “Support every version forever” — or its opposite, “force everyone to update weekly”. Both cost more than a published window.
• “Signing lives on one laptop.” Store identities centrally, read-only for CI, with named administrators.
• “Our device-lab vendor is permanent.” App Center retired in 2025 (crash reporting lasts to March 2027); Firebase Test Lab shuts down in 2027^[16][17].

Invest in the full practice when a native app ships through public stores, a bad release costs money, safety or trust, more than a few thousand people use it daily, and you release at least monthly.

Keep it light, or do something else, when:

• It is a web app or PWA. You can roll back; see WebOps.
• It is an internal app on managed devices. Distribution is EndpointOps territory; keep CI, signing and a minimum-version check. With Apple in-house distribution, certificate expiry stops the app from running^[68].
• It is a prototype before launch. Do only Day 0: it cannot be added to versions already installed.

Hidden responsibilities: macOS CI images, store administration, a version-check endpoint that must work forever, API compatibility for every supported version, and yearly platform floors.

flowchart TB
  A{"Native app in\npublic stores?"} -->|"no"| B["Web or managed fleet:\nWebOps or EndpointOps"]
  A -->|"yes"| C{"Launched yet?"}
  C -->|"no"| D["Day 0 kit: CI, signing,\ncrash, config,\nversion check"]
  C -->|"yes"| E{"Costly failures\nor many users?"}
  E -->|"no"| F["Light: monthly train,\nmanual rollouts"]
  E -->|"yes"| G["Full: weekly train,\nnumeric gates,\nkill switches"]
  G --> H{"Interpreted-code\nlayer?"}
  H -->|"yes"| I["Add OTA with\nruntime-version policy"]
  H -->|"no"| J["Stop here:\nno OTA tooling"]
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  class A,C,E,H decision
  class D,F,G,I,J core
  class B neutral

This is this Atlas’s model, not an industry standard.

• Days 1–30: inventory signing identities, store roles and expiry dates; move signing to a shared store; build both platforms in CI with symbol upload; write the rollout policy with provisional thresholds.
• Days 31–60: start a fortnightly train with a captain rotation; stage every release; add a kill switch to the next risky feature; ship the minimum-version check and the support-policy endpoint; measure kill-switch propagation in beta.
• Days 61–90: run Lab 1’s gate on real exports and tune thresholds; publish the support window; add contract tests against the oldest supported version; review the first halts; decide on weekly trains.

Release readiness (before submission)

• Build from CI, signed from the shared store; build number higher than any previous upload; symbols uploaded
• Every risky feature behind a flag that defaults to off, tested on and off in beta
• New features described in the review notes and reachable by the reviewer
• API changes checked against the oldest supported version
• Release notes list significant changes; data disclosures updated if an SDK or data type changed

Rollout watch (each step)

• Gate values recorded per version: crashed sessions, ANRs, key business event, minimum users reached
• Velocity alerts switched on, thresholds tuned, routed to the captain and the on-call engineer
• Halt and pause commands tested this quarter; the captain knows who may resume

Quarterly operations review

• Certificates, keys and push credentials more than 60 days from expiry; store roles reviewed
• Version tail, support window and blocked list reviewed with backend owners
• Platform deadlines and tool retirements on the calendar

RELEASE RECORD  app: ______  version: ______  build/versionCode: ______
Captain: ______  Train date: ______  Review submitted / approved: ____ / ____
Flags introduced (owner, default, expiry): ________________________________
Step | Date-time | Users | Crashed-session ratio | ANR ratio | Decision | By
1%   |           |       |                       |           |          |
...
Halts and resumes (time, reason, who): ___________________________________
Follow-ups (owner, due date): ____________________________________________

SUPPORT-WINDOW POLICY  platform: ______  owner: product + mobile platform
Soft minimum: ______ (prompt; reviewed monthly)
Hard minimum: ______ (block; needs product, platform and backend sign-off)
Blocked versions: ______ (known-bad builds)
Rule: never break an API used by a version at or above the hard minimum
Rule: the version-check response stays readable by every version ever shipped
Rule: block only when the replacement reaches every blocked user
      (Play rollout complete, all targeted countries, their OS supported)
Users unable to update (OS below our minimum): alternative channel ______

Is “MobileOps” a real discipline? The work is; the name is rare. Use “mobile DevOps” or “mobile release engineering” when hiring or searching (term status).

Can I roll back an app release? No. You can limit its spread, turn features off and move users on (levers); even Play’s halt of a fully rolled-out release leaves updated users on the halted version^[31].

How many old versions should we support? Decide from the tail data (Lab 2), the cost of API compatibility and the users you would block. This Atlas suggests a soft minimum about three months back, and moving the hard minimum only for security or cost reasons.

Do we need a device farm? Start with emulators, simulators and a few real devices matching your users’ commonest models and OS versions; add a cloud lab when device-specific crashes appear, and check its future^[17].

Should mobile use DORA metrics? Yes, once you define a mobile “deployment” (metrics).

• Moving fast: device-lab consolidation, with new Developer Device Platform pricing after 2027-04-30^[17]; memory metrics that may affect Play visibility from February 2027^[11]; agent-reserved test devices (automation)^[17].
• To evaluate, not adopt blindly: moving more behaviour to the server so fewer changes need a binary, within the stores’ rules on downloaded code^[36][38].
• Regional distribution rules: Apple’s guidelines refer to alternative distribution, which changes store gating where it applies^[37]; check jurisdiction before relying on it.

• Prerequisites: CI/CD basics (DevOps); deploy versus release (ReleaseOps).
• One day: read the two store pages on phased and staged release^[10][9]; run Labs 1 and 2.
• One week: build a beta lane with fastlane; add crash reporting and a kill switch to a sample app.
• Thirty days: run four trains with written gates; ship a minimum-version check; write the support-window policy.
• Specialise: mobile security (DevSecOps, MASVS), API evolution (API Ops), or test infrastructure (TestOps).
• Portfolio project: an open-source sample app with CI, staged releases, a support-policy endpoint and a gate script.

• Decide: release cadence, gate thresholds, who may halt and resume, the support window.
• Build once, early: shared signing, CI, crash reporting per version, flags with safe defaults, a minimum-version check.
• Watch: relative crash and ANR rates per version, time to contain, kill-switch propagation, tail outside the window, credential headroom.
• Remember: no recall; approval is not release; old versions live for years; block only what users can replace.
• Next action: list every signing identity and its expiry date this week.

Situation (illustrative). Version 6.4.0 crashes in checkout for 3% of its sessions on iOS and already runs on 40% of active iOS users (the phased release reached day 6 before the cause was clear). Fixed 6.4.1 was approved this morning. Option A: set hard_min to 6.4.1 now, so every 6.4.0 user must update. Option B: turn checkout off server-side for 6.4.0, show a soft prompt, and let 6.4.1 spread through automatic updates.

Assumptions. The flag stops the crash (verified within 30 minutes); checkout is a third of revenue; some users have automatic updates off; 6.4.1 has been live for only hours.

Reasoning. Option A feels decisive, but it blocks the whole app, not just checkout, for 40% of users, and puts 6.4.1 on every device before its own rollout has data, making any 6.4.1 defect universal (on Android it would also lock out users whom 6.4.1’s staged rollout has not reached). Option B removes the crash at once and lets 6.4.1 pass its own gates; its cost is lost checkout revenue on 6.4.0 until users update. Choose B, then block 6.4.0 once 6.4.1 has passed its gates. Choose A only if the defect corrupts data or is a security flaw the flag cannot neutralise. Uncertainty: how fast users with automatic updates off move; watch the 6.4.0 share daily.

After this page you should be able to explain why mobile releases cannot be recalled; compare Apple’s and Google’s rollout levers; design gates, kill switches and a support window; run a containment runbook; and evaluate OTA and forced-upgrade choices.

1. Fixed 6.4.1 is at 20% of a Play staged rollout. Product wants hard_min raised to 6.4.1 today. What happens?

2. Version 7.1 shows 99.81% crash-free sessions; 7.0 showed 99.90%. Is 7.1 fine?

3. Your team wants to add a forced-upgrade screen “when we need it”. Why is that too late?

4. Can you use an OTA update to enable a new payment feature that review has not seen?

5. Every app version, old and new, starts crashing at launch at the same minute. What do you check first?

“MobileOps” is a rare name for a real job: shipping software that, once installed, is out of reach. The stores add a gate you do not control, and users decide when to update. Everything useful here follows: release in steps large enough to notice harm, keep the fast levers on the server, put the update check and the kill switch into version one, and support a window of versions on purpose. Teams that treat a phone like a server rediscover the difference during their worst release.

Every source was opened on 2026-10-18 (two re-checked 2026-10-19); dates are the source’s own. Store rules and limits change: recheck before use. Claim-level support is in the Atlas evidence ledger (research/mobileops-ledger.md).

Further reading: Orosz’s article also covers state, deep links, offline and localisation problems this page leaves out.

Changelog:

• 2026-10-18 — first version (research, outline, draft, two executed labs, six diagrams).
• 2026-10-18 — r1 revision after the first review round (reviews/mobileops-r1-superseded-202610191512.md): MetricKit corrected; approval versus release; block-only-when-replaceable; halts precede the min-users check; credential scopes; symbol upload; App Center, velocity-alert and deployment-target facts; rollout script hardened; Lab 3 written out; three diagrams redrawn; repetition cut; ten sources added.
• 2026-10-19 — second-pass r1 revision (reviews/mobileops-r1.md; no critical or major findings): Agent-skills date marked as this Atlas’s observation, not a Google launch date (M1); target-API row and Day 2 note add the 2026-11-01 extension and the API 35 floor (M2).
• 2026-10-19, post-score polish (reviews/mobileops-polish.md): boundary and decision diagrams redrawn for label headroom (same nodes, edges and meanings); three repetition trims; no sourced claim changed.