--- title: "MobileOps" slug: mobileops category: delivery term_status: overlapping-synonym evidence_status: verified summary: "MobileOps is an informal name for mobile DevOps: building, signing, store-releasing and operating installed iOS and Android apps — staged rollouts, crash signals, remote config, old-version support and forced upgrades — when a shipped binary cannot be recalled." audience: [Mobile engineers, Mobile release managers, Mobile platform teams, Backend and API owners, Engineering leaders, Beginners] last_verified: 2026-10-19 key_takeaways: - "\"MobileOps\" is not an established label: the practice is called mobile DevOps or mobile release engineering, and the exact name belongs to several unrelated products." - "An installed binary cannot be recalled. Halting a Play rollout or pausing Apple's phased release limits who gets a bad version (Apple still allows manual downloads); users who already have it keep it until they update." - "Every release passes someone else's gate (Apple typically reviews 90% of submissions within 48 hours; Google says processing can take up to seven days, longer in exceptional cases): submit early, release when ready, and keep the fast levers server-side (flags, remote config, a minimum-version switch)." - "Old versions stay in use for months or years: never break a supported version's API, and raise a hard minimum only when every user it blocks can get the replacement." - "Gate each rollout step on per-version crash and ANR numbers chosen in advance: a clear regression must halt at any step, while a moderate one shows only at a larger step, so size steps by the users they contain." - "A forced-upgrade path works only if it is already inside the installed versions; Google Play's console prompt is the one exception, and over-the-air updates are limited to interpreted code within store rules." - "Signing identities, store deadlines and tool retirements are calendar work: in 2026 Apple required the iOS 26 SDK, Google required API level 36, and Firebase Test Lab is set to shut down in September 2027." related: [releaseops, testops, api-ops, devops, endpointops, privacyops, devsecops, observabilityops] --- ## What MobileOps is ### In 30 seconds **MobileOps** is a rarely used, informal name for running the delivery of native mobile apps: building and signing iOS and Android binaries, getting them through app-store review, releasing them in stages, watching crashes per version, and keeping the old versions that people still run working. The practice is usually called **mobile DevOps**, which Wikipedia describes as applying "the principles of DevOps specifically to the development of mobile applications"[^wiki], or mobile release engineering. **In one line: on the web you can roll back; on a phone you cannot recall a binary.** Once users update to a version with a bug, Gergely Orosz writes, "they are stuck with that bug until a new version is released and this user updates"[^orosz]. MobileOps is the habit of shipping as if that were true, because it is. It is for mobile and release engineers, platform teams, backend owners and engineering leaders. ::: note Term status: overlapping synonym — read this first - No standards body, certification or manifesto defines "MobileOps". In October 2026 searches the exact name appeared only as **product names**: a Trinetri unified endpoint management product[^trinetri], an Abavus field-workforce app for local councils[^abavus] and a maritime operations platform[^mobileopscom]. - The practice is documented as **mobile DevOps**: a Wikipedia article (flagged in January 2025 as reading like a personal essay)[^wiki], two Packt books (Tak and Modi, 2018; Nabil, 2023)[^wiki], and vendors that sell a "Mobile DevOps Platform"[^bitrise]. - This Atlas uses **MobileOps as an editorial label** for the constraints specific to mobile delivery: store gating, signing, staged rollouts, the long tail of old versions, API compatibility and forced upgrades. Its boundaries with neighbouring pages are **editorial distinctions**, not industry standards. ::: ### At a glance | Item | Summary | |---|---| | Goal | Predictable releases despite store gating; bad builds found while few users have them; every installed version either supported or deliberately retired | | Users | Mobile and release engineers, mobile platform teams, QA, backend owners, product and support leads | | In scope | Mobile CI/CD and signing, store review, beta channels, staged rollouts, crash and ANR signals, remote config and kill switches, over-the-air (OTA) limits, device testing, API compatibility, forced upgrades, privacy-aware telemetry | | Out of scope | Release strategy → [ReleaseOps](../releaseops/); test reliability → [TestOps](../testops/); API design and linting → [API Ops](../api-ops/); corporate devices → [EndpointOps](../endpointops/); consent and legal basis → [PrivacyOps](../privacyops/); security testing → [DevSecOps](../devsecops/); telemetry pipelines → [ObservabilityOps](../observabilityops/) | | Prerequisites | CI/CD basics ([DevOps](../devops/)); deploy versus release ([ReleaseOps](../releaseops/)) | | Key outputs | Signing inventory, release-train calendar, rollout policy with numeric gates, support-window (minimum version) policy, API compatibility contract, runbooks | | Term status | Overlapping synonym for mobile DevOps | ### The problem it solves Mobile delivery differs from server delivery in three ways that no tool removes: 1. **Someone else controls the last step.** Apple says App Review typically reviews "at least 50% of submissions in less than 24 hours and 90% in less than 48 hours"[^applereview]. Google says processing "can take a few hours or up to seven days (or longer in exceptional cases)"[^playreview]. A one-line fix cannot reach users in minutes. 2. **You cannot take a version back.** When you halt a Google Play staged rollout, "Users who already received the app version ... will remain on that version"[^playstaged]. Apple lets you pause a phased release, but a pause stops only automatic updates: the version "can be manually downloaded from the App Store by anyone at any time"[^applephased]. 3. **Many versions run at once.** Old versions remain "up to a number of years", and "Even a non-breaking backend change can break an older version of the app"[^orosz]. Ignoring them costs hotfix churn, expired-certificate fire drills, back-end changes that break last year's app and, on Android, Play may reduce the visibility of apps above its crash and ANR thresholds[^vitals]. ### What MobileOps is not - **Not a recognised discipline or job title.** Engineers doing this work are called mobile release engineers, mobile platform engineers or "mobile DevOps" engineers. - **Not mobile device management.** Enrolling and policing corporate phones is [EndpointOps](../endpointops/), despite the name collision[^trinetri]. - **Not DevOps with a phone icon.** The CI/CD and DORA baseline is [DevOps](../devops/); this page adds only what changes on mobile. - **Not "mobile testing".** Device labs and test selection are here; whether a test result can be trusted is [TestOps](../testops/). ## Where MobileOps comes from and where it sits ### Origins and evolution Mobile delivery grew up inside the stores' walled gardens: compiled binaries, mandatory signing and review. Wikipedia dates the rise of the term "mobile DevOps" to 2014–2015, citing Google Trends[^wiki]. Traceable milestones: | Year | Milestone | What it shows | |---|---|---| | 2017 | Facebook reports that its mobile apps had gone "from four-week releases to two-week releases to one-week releases", with daily release candidates for about a million Android beta testers[^fbrapid] | Continuous delivery reaches mobile, but as a weekly train | | 2021 | The Linux Foundation launches the Mobile Native Foundation; Lyft's Keith Smiley says platform vendors "did not solve all of the problems" Lyft met as its mobile team grew[^mnf] | Shared tooling for build and release at scale | | 2024 | Squarespace's Unfold team runs one-week trains with automatic rollout halts[^squarespace]; Sonos replaces its app all at once and later commits to gradual releases[^sonospr] | Staged release becomes a reputational issue, not only an engineering one | | 2025–2027 | Visual Studio App Center retires on 2025-03-31, except Analytics & Diagnostics, extended to the end of March 2027[^appcenter]; Firebase Test Lab is deprecated, with shutdown set for 2027-09-30[^ftlfaq] | Mobile tooling churns; exit plans matter | ### Glossary and mental model | Term | Meaning on this page | |---|---| | Version and build number | User-facing version (`versionName`, `CFBundleShortVersionString`) versus build number (`versionCode`, `CFBundleVersion`); Play's `versionCode` must increase, up to 2,100,000,000[^playversion] | | Signing identity | Apple certificates and provisioning profiles; on Android the upload key and the app signing key[^playsigning] | | Phased release / staged rollout | Apple's and Google's ways of releasing an update gradually ([compared below](#platform-levers-compared-app-store-and-google-play)) | | ANR | "Application Not Responding": the Android UI thread blocked, for example no response to input within 5 seconds[^anr] | | User-perceived crash rate | Share of daily active users with at least one crash while actively using the app (Android vitals)[^vitalscrash] | | Symbol files | dSYM files (iOS) and R8 or ProGuard mapping files (Android) that make crash stack traces readable[^crsymios][^playdeob] | | Remote config / feature flag | Server-delivered values the app reads; a **kill switch** is a flag that turns a feature off | | OTA update | Replacing interpreted code (such as JavaScript) and assets in an installed app without a store release[^easintro] | | Version tail | Active users on versions older than the current one | | Soft / hard minimum | Versions below the soft minimum are prompted to update; below the hard minimum they are blocked | | Release train / captain | Fixed release cadence; the rotating person who runs this week's release[^squarespace] | **Mental model: two layers and three clocks.** The **binary layer** is slow, gated and cannot be recalled; the **server layer** (config, flags, APIs, minimum versions) changes in minutes. MobileOps moves risk from the first layer to the second. Three clocks set the pace: the **build clock** (yours: CI and tests), the **store clock** (theirs: review) and the **adoption clock** (your users' update habits). You can shorten the first, plan around the second (submit early, release when ready) and only influence the third. ### How MobileOps relates to other Ops | Neighbour | It owns | MobileOps adds | |---|---|---| | [DevOps](../devops/) | CI/CD, trunk-based development, DORA metrics | Signing, macOS builds, store submission | | [ReleaseOps](../releaseops/) | Deploy versus release, flags, progressive delivery | Store-imposed schedules, no recall, version tail | | [TestOps](../testops/) | Trustworthy, fast test signal | Device labs, OS and device matrix | | [API Ops](../api-ops/) | API contract, linting, lifecycle | Support windows tied to app versions | | [EndpointOps](../endpointops/) | Managing corporate devices (MDM, MAM) | Nothing on device management | | [PrivacyOps](../privacyops/) | Legal basis, consent, data-subject rights | Store disclosures, telemetry with consent | | [DevSecOps](../devsecops/) | Security testing and provenance | Mobile signing keys, OWASP MASVS update controls | | [ObservabilityOps](../observabilityops/) | Telemetry pipelines | Per-version crash signals as release gates | ::: figure What crosses each boundary between MobileOps and its neighbours ```mermaid flowchart TB D["DevOps"]:::neutral -->|"CI/CD baseline"| M R["ReleaseOps"]:::neutral -->|"rollout and\nflag strategy"| M T["TestOps"]:::neutral -->|"trusted test\nresults"| M E["EndpointOps"]:::risk -.-|"name clash only"| M M["MobileOps:\nstore-gated, installed binaries"]:::core M -->|"support window"| A["API Ops"]:::neutral M -->|"new SDK or\ndata type"| P["PrivacyOps"]:::neutral M -->|"signing keys"| S["DevSecOps"]:::neutral M -->|"per-version gates"| O["ObservabilityOps"]:::neutral classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029 classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029 ``` Text equivalent: MobileOps receives a CI/CD baseline from DevOps, a rollout and flag strategy from ReleaseOps and trusted test results from TestOps. It hands API Ops the support window that decides when an endpoint may retire, PrivacyOps every new SDK or data type, DevSecOps the signing keys to protect, and ObservabilityOps the per-version gates its pipeline must feed. EndpointOps shares only the name (dotted line). ::: ## Principles and operating model ### Charter and principles This is **this Atlas's proposed charter**; no official MobileOps charter exists. **Mission:** every release reaches users in stages, every bad build is contained while few users have it, and every installed version is either supported or deliberately retired. 1. **Assume there is no rollback.** Before shipping, know how you will turn the feature off, stop the rollout and move users on. 2. **Fast levers on the server, slow change in the binary.** Flags, config and minimum versions change in minutes. 3. **Decide gates before the rollout.** Numeric, per-version and written down; anyone on the rotation may halt. 4. **Support a window, not every version.** Publish which versions you support; retire the rest on purpose. 5. **Keep signing and store access boring.** Inventoried, least-privilege, renewed on a calendar. 6. **Measure with consent, and know what you cannot see.** 7. **Treat store rules as design constraints.** **Non-goals:** forcing every user onto the latest version; replacing store review; building a home-grown way to swap native code. ### Building blocks Eight blocks recur: **signing identities** (loss or expiry blocks every release); a **build pipeline** with macOS for iOS (Xcode 27.1 needs macOS Tahoe 26.6 or later[^xcodereq]); **distribution channels**; **rollout controls**, the only brake on the binary layer; per-version **signals**; **server levers** that act without review; a **support window**; and an **API contract** for every supported version. ### Roles, ownership and RACI | Activity | Release captain | Mobile platform team | Feature team | Backend owner | Store admin | |---|---|---|---|---|---| | Cut, build and submit the release | A/R | C (pipeline) | C | I | I | | Set rollout gates and thresholds | C | A/R | C | C | I | | Advance, pause or halt a rollout | A/R | C | C | I | I | | Turn a feature flag off | R | I | A | I | I | | Change minimum supported versions | R | C | C | C | A (with product) | | Keep APIs compatible with supported versions | I | C | C | A/R | I | | Renew certificates, rotate keys, manage roles | I | R | I | I | A | | Track platform deadlines (SDK, target API) | I | A/R | C | I | C | R = responsible, A = accountable, C = consulted, I = informed. A small team can merge the captain and platform roles; it should not merge store administration into everyday CI credentials. ### Lifecycle: Day 0, Day 1, Day 2 - **Day 0 — before the first release.** Set up store roles; enrol in Play App Signing; keep Apple signing identities in a shared, encrypted store; build in CI; add crash reporting with a consent decision and remote config with in-app defaults; **ship a minimum-version check in version 1**, since in-app updates benefit "only future versions of your app"[^playprompt]. - **Day 1 — first release.** Neither store stages a first release ([levers](#platform-levers-compared-app-store-and-google-play)): rely on TestFlight and testing tracks; Apple reviews the first build sent to external testers[^testflight]. - **Day 2 — operating.** Run the train, gate rollouts, manage the tail, renew credentials, and absorb the stores' floors: since 2026-04-28 uploads to App Store Connect need Xcode 26 and the iOS 26 SDK, and since 2026-09-09 iOS apps must target iOS 13 or later[^appleupcoming]; since 2026-08-31 Google Play updates must target Android 16 (API level 36) — extension to 2026-11-01 possible; un-updated existing apps face an API 35 floor[^playtarget]. ::: figure Life of one app version, from build to retirement ```mermaid stateDiagram-v2 direction TB [*] --> Built Built --> Beta: testers install Beta --> InReview: submitted InReview --> Beta: rejected, fix InReview --> Approved: approved Approved --> Rollout: released when ready Rollout --> Halted: gate fails Halted --> Rollout: resume Halted --> Superseded: fix ships as N+1 Rollout --> Current: 100 percent Current --> Supported: newer version ships Supported --> Prompted: below soft minimum Prompted --> Blocked: below hard minimum Blocked --> [*]: no active users Superseded --> Prompted: move users on ``` Text equivalent: a version is built, beta-tested and reviewed (a rejection sends it back). Approved, it waits until the team releases it, then rolls out in stages; a failed gate halts it, after which it resumes or is superseded by N+1, whose users are prompted to move on. At 100 percent it is current, then supported once a newer version ships, then prompted below the soft minimum, blocked below the hard minimum, and retired when no active users remain. ::: ### Operating model and governance - **Cadence.** A weekly or fortnightly train. Approval is not release: submit early, hold the approved build (Apple's manual release, Play's managed publishing[^applerelease][^playreview]) and release when the back end and flags are ready. Squarespace submits on Friday and rolls out on Monday[^squarespace]. A hotfix lane needs written entry criteria (crash or data-loss regressions, security, legal). - **Gates.** Agreed before the train starts; halting needs no permission, resuming needs data. Squarespace's captains "weren't sure when a problem was serious enough to halt a rollout" until numeric heuristics existed[^squarespace]. - **Access.** Quarterly role review; CI identities scoped as in the [reference implementation](#reference-implementation); a documented break-glass admin. - **Calendar.** Certificate expiry, store deadlines and tool retirements in one place. - **Support window.** Monthly tail review with backend owners; minimum versions move by decision. - **Escalation.** A harmful release becomes an incident run with [IncidentOps](../incidentops/) roles; the captain keeps the levers. ## Architecture and workflow ### Reference architecture Four parts: build and sign, distribute, the installed fleet, and a server side that keeps working while binaries wait for review. ::: figure Reference architecture: from repository to the installed fleet ```mermaid flowchart TB SRC["Repository and lanes"] --> CI["CI: macOS and Linux"] SIG["Signing store"] -->|"read-only"| CI CI --> BETA["Beta channels"] BETA --> REV["Store review"] REV --> ROLL["Staged or phased rollout"] ROLL --> FLEET["Installed fleet: N, N-1, older tail"] subgraph CTRL["Server side: changes in minutes"] CFG["Flags and config"] MIN["Minimum versions"] API["Versioned APIs"] end CTRL --> FLEET FLEET --> CR["Crash and vitals"] ROLL <-.-|"halt or advance"| CR classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029 classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029 class ROLL,MIN core class REV risk class CR decision ``` Text equivalent: the repository and a signing store that CI reads but cannot change feed CI on macOS (iOS) and Linux (Android). Builds go to beta channels, store review (not under your control), then a staged or phased rollout to the installed fleet, where version N runs beside N-1 and an older tail. A server side that changes in minutes (flags and config, minimum versions, versioned APIs) serves every installed version. The fleet's crash and vitals signals (dotted line) decide whether the rollout halts or advances. ::: | Component | Responsibility | Trust boundary and failure to plan for | |---|---|---| | Signing store | Certificates, profiles, keys; read-only for CI | Compromise lets anyone ship as you; expiry blocks uploads | | CI | Reproducible builds, tests, symbol upload | macOS capacity and Xcode versions pinned per image | | Store accounts | Submission, rollout control, roles | Few administrators; API keys scoped per job | | Flags and config | Kill switches, gradual exposure, safe defaults | Who may flip what; defaults are what offline users get | | Minimum-version policy | Soft and hard minimum per platform | Must stay readable by the oldest version ever shipped | | Versioned APIs | Serve every supported version | Breaking change → old apps fail silently | | Crash and vitals | Per-version stability, alerts | Sees only users who allow collection | ### End-to-end workflow ::: figure One release, with its exception paths ```mermaid flowchart TB A["Branch cut on train day"] --> B["CI builds, tests, signs"] B --> C["Beta: staff and testers"] C --> D{"Store review"} D -->|"rejected"| B D -->|"approved"| R["Held until back end and flags are ready"] R --> E["Release a step: 1 percent first"] E --> W["Wait, then read per-version metrics"] W --> G{"Gate"} G -.->|"too little data: hold"| W G -->|"clear, below 100 percent"| E G -->|"clear at 100 percent"| T["Watch the tail, move minimums"] G -->|"regression"| X["Flag off, then halt or pause"] X --> Y{"Fixable on the server?"} Y -->|"yes: fix, then resume"| W Y -->|"no"| K["Hotfix N+1, expedited review"] classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029 classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029 class D,G,Y decision class X,K risk class R,T core ``` Text equivalent: the branch is cut on train day; CI builds, tests and signs; staff and testers use the beta; store review rejects (back to CI) or approves, and the approved build is held until the back end and flags are ready. Every step (1 percent first) is followed by a wait and a per-version gate: too little data holds the step; a clear result releases the next step until 100 percent, after which the team watches the tail and moves minimum versions; a regression triggers flag-off and a halt or pause. A server-side fix leads back to the gate; otherwise version N+1 ships with expedited review. ::: **Rehearse three exception paths:** a rejection on launch day (Apple offers expedited review for critical bugs[^applereview]); a regression found after 100 percent (Play can halt a fully rolled-out release[^playhaltfull]; Apple says a live version with a usability issue needs an app update[^appleunavail]); and a server or SDK change that crashes every installed version at once. ## Practices, frameworks and tools ### Core practices and playbooks | Practice | Goal | Trigger | Owner | Verification | |---|---|---|---|---| | Release train | Small, predictable releases | Calendar | Release captain | Train day rarely missed | | Gated staged rollout | Contain bad builds | Every production release | Captain; platform sets gates | Halts precede support tickets | | Kill switch per risky feature | Turn behaviour off in minutes | New feature or risky change | Feature team | Switch tested in beta, with a safe default | | Support window and forced upgrade | Bound the tail | Monthly review; security fix | Product with platform | Share of users below hard minimum | | API compatibility window | Old apps keep working | Every API change | Backend owner | Contract tests against the oldest supported version | | Signing hygiene | No surprise expiry or leak | Quarterly; on staff change | Store admin | Expiry dates more than 60 days away | | Device matrix from usage | Test what users run | Each quarter | QA with platform | Top devices and OS versions covered | | OTA with runtime versions | Fix interpreted code safely | Bug in JavaScript or assets | Feature team | Update targets only compatible builds | #### Forced upgrades: soft prompts, hard blocks and what each platform offers A **soft minimum** prompts; a **hard minimum** blocks until the user updates. Both need code that is already installed, with one exception on Android. - **Google Play.** The in-app updates library offers a *flexible* flow (download in the background) and an *immediate* full-screen flow, on Android 5.0 and later; the app can choose between them from an update priority (0–5) set at rollout[^inapp][^inappkotlin]. Play Console's **Prompt users to update** recovery tool shows users of a chosen version a full-screen prompt with no code change; dismissed, it returns "after every cold restart". It needs Play App Signing and app bundles and is unavailable with some features (for example Code Transparency)[^playprompt]. - **App Store.** In the Apple documentation checked in October 2026 there is no equivalent in-app update API (Orosz made the same observation in 2020[^orosz]). Teams fetch a minimum version from their own config, compare it with the installed version, and link to the App Store page. - **Both.** The version-check response must stay readable by the oldest version ever shipped, or blocked users see a crash instead of a message. Compare versions numerically ([Lab 2](#hands-on-labs)). - **Block only when the replacement reaches everyone you block.** A Play staged rollout reaches "only a percentage of your users", chosen at random, in the countries you target[^playstaged]; the rest cannot update yet, and devices below its minimum OS never can. Block only after the replacement's Play rollout completes in every targeted country, and give users who still cannot update a soft prompt or another channel. On iOS anyone can download a phased release manually[^applephased], so a store link works. - **Why bother.** MASVS control MASVS-CODE-2 reads "The app has a mechanism for enforcing app updates"[^masvscode]: a security fix works only once users run it. #### Over-the-air updates: what the stores allow Apple's licence agreement says an app "may not download or install executable code"; interpreted code may be downloaded only if it does not change the app's "primary purpose", does not "bypass signing, sandbox, or other security features of the OS", and does not create a storefront for other apps[^dpla]; the App Review Guidelines add code "which introduces or changes features or functionality"[^appleguidelines]. Google Play forbids updating "using any method other than Google Play's update mechanism" and downloading executable code such as dex, JAR or .so files, but exempts code running in "a virtual machine or an interpreter"[^playdna]. OTA tools such as Expo's EAS Update replace JavaScript, styling and images, not native code, permissions or SDK versions, and "your updates need to follow the App Store and Play Store guidelines"[^easintro]. Each update must match the build's **runtime version**; an incompatible update calls "code that does not exist inside the build"[^easruntime]. Microsoft's CodePush survives App Center only as a self-hosted "special version"[^appcenter]. Use OTA for bug fixes in interpreted code and keep behaviour changes in reviewed releases. #### Remote config and kill switches on a phone Client flag SDKs fetch flags in bulk for one user, cache them, and evaluate against the cache — OpenFeature calls this the **static-context paradigm**[^openfeature]. Consequences: - **A kill switch is not instant.** Firebase Remote Config's default minimum fetch interval is 12 hours[^rcstart]; a strategy Firebase describes as effective activates fetched values only at the next start, while real-time updates bypass the interval[^rcloading]. Measure propagation. - **Defaults are policy.** Until its first successful fetch the app runs its in-app defaults[^rcloading]; a risky feature should default to off. - **Flags do not hide features from review.** Apple requires new features to be "described with specificity in the Notes for Review ... and accessible for review" (guideline 2.3.1(a))[^appleguidelines]. - **Flags accumulate.** Give each an owner and expiry; [ReleaseOps](../releaseops/) covers flag debt. ### Platform levers compared: App Store and Google Play As of October 2026; recheck before relying on a limit. | Lever | Apple App Store | Google Play | |---|---|---| | Review time | Typically 50% in under 24 h, 90% under 48 h; expedited review for critical bugs[^applereview] | A few hours to seven days, longer in exceptional cases[^playreview] | | Release after approval | Manually, automatically, or automatically no earlier than a date; a manual release can take up to 24 h to appear[^applerelease] | Managed publishing holds most approved changes until you publish; a change sent during review may requeue the app[^playreview] | | Pre-release | TestFlight: 100 internal, 10,000 external testers, 90-day builds[^testflight] | Internal (100 testers), closed and open tracks[^playrollout] | | Gradual release | Phased: fixed 7-day schedule, 1% to 100%, automatic updates only; manual downloads always possible[^applephased] | Staged: any percentage you choose, new and existing users, by country[^playstaged] | | Stop a release | Pause up to 30 days in total; Release to All any time[^applephased] | Halt staged or fully rolled-out releases (not the first release on a track)[^playhaltfull] | | First release | Phased release "not available for the first version of an app"[^ascphased] | Staged rollouts are for updates only[^playstaged] | | Prompt to update | Build your own minimum-version check | In-app updates API; console recovery prompt[^inapp][^playprompt] | | Downloaded code | Interpreted code only, with conditions[^dpla] | Interpreted code only; no self-update[^playdna] | | Platform floor (2026) | Xcode 26 / iOS 26 SDK since 2026-04-28; deployment target iOS 13+ since 2026-09-09[^appleupcoming] | Target API 36 since 2026-08-31 (extension to 2026-11-01 possible; un-updated existing apps: API 35)[^playtarget] | ### Methods, standards and frameworks No standard defines MobileOps. Separate three kinds of guidance: - **Binding rules:** Apple's licence agreement and App Review Guidelines[^dpla][^appleguidelines]; Google Play's policies[^playdna]. Breaking them blocks releases or removes apps. - **Standards:** OWASP MASVS, which OWASP calls "the industry standard for mobile app security"[^masvs]; Google's AIP-180 on backwards compatibility, written for APIs whose consumers update on their own schedule[^aip180] — exactly a mobile back end's situation (this page's inference); the DORA metrics ([DevOps](../devops/)). - **Practitioner patterns:** release trains and rotating captains[^squarespace], daily beta builds[^fbrapid]: single-organisation experience, not norms. ### Tooling landscape Tool categories first; names are examples verified on 2026-10-18, not endorsements. | Job | Representative options | Selection notes | |---|---|---| | Release automation | fastlane, which automates "beta deployments and releases for your iOS and Android apps"[^fastlane] | Open source; lanes become your release runbook | | CI | Xcode Cloud (25 compute hours a month with membership)[^xcodecloud]; GitHub-hosted macOS runners[^gharunners]; mobile CI vendors such as Bitrise[^bitrise] | iOS needs macOS; check Xcode versions per image | | Signing | fastlane match (git, Google Cloud or S3 storage)[^match]; Play App Signing[^playsigning] | Read-only access from CI | | Device testing | Firebase Test Lab (shutting down 2027-09-30; successor Developer Device Platform)[^ftlfaq]; BrowserStack App Automate, which Microsoft recommended to App Center users[^appcenter] | Real versus virtual devices, cost, flakiness | | Stability signals | Crashlytics[^crashfree]; Android vitals and the Reporting API[^vitals][^reportingapi]; Xcode Organizer and MetricKit[^applecrash][^metrickit] | Latency and alert thresholds; per-version breakdown; consent coverage; MetricKit data stays on the device until you upload it | | Config and flags | Firebase Remote Config[^rcloading]; OpenFeature client SDKs[^openfeature] | Fetch interval, real-time updates, safe defaults | | OTA | Expo EAS Update[^easintro]; self-hosted CodePush[^appcenter] | Only for interpreted code; runtime-version policy | **Lock-in and churn are real.** App Center's crash reporting ends in March 2027 (Azure Monitor is the migration path)[^appcenter]; Firebase Test Lab stops on 2027-09-30[^ftlfaq]. Keep release logic in your own lanes and scripts so that a vendor change swaps one step, not the process. ### Automation and AI Automate what is deterministic: build numbers, signing (read-only), uploads, symbol files, gate evaluation, advancing and halting. Squarespace's automation reads Crashlytics and Amplitude data and "can automatically halt the rollout"[^squarespace]. Keep a person on resuming a halted rollout and on moving a hard minimum. AI may group crash reports and draft release notes; a person checks the notes, since Apple requires significant changes in "What's New" (guideline 2.3.12)[^appleguidelines]. Google's Developer Device Platform FAQ lists "Agent skills" — "Help AI coding agents find, reserve, and interact with devices for you" — with no launch date given (page seen October 2026)[^ftlfaq]. ## MobileOps in practice ### Reference implementation **Assumptions:** a consumer app on iOS and Android, about 400,000 daily users, a weekly train, per-version crash data, remote config and GitHub-hosted runners. Four artefacts: a rollout policy, a support policy served to apps, release lanes and a rollout script. _Example status: YAML syntax-checked by the Atlas validator, not executed._ ```yaml # release-policy.yaml: reviewed by the mobile platform team each quarter app: com.example.shop platforms: android: steps: [0.01, 0.05, 0.20, 0.50, 1.0] # Play userFraction per step min_hours_per_step: 24 ios: phased_release: true # Apple's fixed 7-day schedule pause_budget_days: 30 # Apple's total pause limit gates: # compared with the previous version min_users_per_step: 5000 # delays ADVANCE only, never a HALT crashed_session_rate_max_ratio: 1.5 anr_rate_max_ratio: 1.5 # Android only business_events: - name: checkout_completed_per_session max_drop_pct: 5 halt_rights: [release-captain, on-call-mobile, feature-owner] resume_requires: [release-captain, mobile-platform] ``` The support policy lives at a stable URL that every version ever shipped can read: never remove or rename a field (AIP-180 treats both as breaking[^aip180]), compare versions numerically, and list known-bad versions. _Example status: JSON syntax-checked; the store URLs are placeholders._ ```json { "schema": 1, "updated": "2026-10-18", "platforms": { "ios": { "hard_min": "5.9.0", "soft_min": "6.8.0", "blocked": ["6.4.0"], "store_url": "https://apps.apple.com/app/id0000000000" }, "android": { "hard_min": "5.9.0", "soft_min": "6.8.0", "blocked": [], "store_url": "https://play.google.com/store/apps/details?id=com.example.shop" } }, "message_key": "update_required_v1" } ``` The lanes build, upload crash symbols and ship to testers. CI never creates certificates (`readonly`, as fastlane recommends for CI[^match]); `upload_symbols_to_crashlytics` does not fail the build on an upload error unless told to[^flsymbols]. _Example status: Ruby, not syntax-checked (no Ruby interpreter was available), not executed; needs Apple and Google accounts._ ```ruby # fastlane/Fastfile platform :ios do lane :beta do app_store_connect_api_key( key_id: ENV["ASC_KEY_ID"], issuer_id: ENV["ASC_ISSUER_ID"], key_filepath: ENV["ASC_KEY_PATH"] ) match(type: "appstore", readonly: true) increment_build_number(build_number: ENV["BUILD_NUMBER"]) build_app(scheme: "Shop", export_method: "app-store") upload_symbols_to_crashlytics( dsym_path: lane_context[SharedValues::DSYM_OUTPUT_PATH], gsp_path: ENV["GSP_PATH"], fail_on_error: true ) upload_to_testflight(skip_waiting_for_build_processing: true) end end platform :android do lane :internal do gradle(task: "bundle", build_type: "Release") # Testing-track identity only; Play reads the R8 mapping file from the bundle upload_to_play_store(track: "internal", json_key: ENV["PLAY_JSON_KEY_PATH"]) end end ``` The rollout script changes one release through the Publishing API's edit flow (insert, change, commit), sending only that release, since in an update request the releases field carries only the desired changes[^playtracks][^playedits]. It rejects fractions outside 0–1 (a typed `50` would otherwise complete the rollout) and has a dry run; `advance` on a halted release resumes it. _Example status: Python syntax-checked; argument checks and the edit sequence executed 2026-10-18 against a stubbed API client; not run against Google Play (needs a Play Console app and an identity with release permission)._ ```python """Advance or halt one Google Play release (Publishing API v3). Usage: python3 play_rollout.py PACKAGE VERSION_CODE advance FRACTION [--dry-run] python3 play_rollout.py PACKAGE VERSION_CODE halt [--dry-run] Options: --track NAME (default production). Credentials: Application Default Credentials (GOOGLE_APPLICATION_CREDENTIALS) supplied by CI; never commit keys. """ import argparse import json import sys import google.auth from googleapiclient.discovery import build SCOPES = ["https://www.googleapis.com/auth/androidpublisher"] def fraction_arg(text): value = float(text) if not 0 < value <= 1: raise argparse.ArgumentTypeError("use a fraction above 0 and up to 1, e.g. 0.05") return value def parse_args(argv): parser = argparse.ArgumentParser(description="Advance or halt a Play release.") parser.add_argument("package") parser.add_argument("version_code") parser.add_argument("action", choices=["advance", "halt"]) parser.add_argument("fraction", nargs="?", type=fraction_arg) parser.add_argument("--track", default="production") parser.add_argument("--dry-run", action="store_true") args = parser.parse_args(argv) if (args.action == "advance") != (args.fraction is not None): parser.error("advance needs FRACTION; halt takes none") return args def change(release, args): if args.action == "halt": release["status"] = "halted" elif args.fraction == 1: release["status"] = "completed" release.pop("userFraction", None) else: release["status"] = "inProgress" release["userFraction"] = args.fraction return release def main(argv): args = parse_args(argv) creds, _ = google.auth.default(scopes=SCOPES) edits = build("androidpublisher", "v3", credentials=creds).edits() ids = {"packageName": args.package} ids["editId"] = edits.insert(body={}, **ids).execute()["id"] track = edits.tracks().get(track=args.track, **ids).execute() found = [r for r in track.get("releases", []) if args.version_code in r.get("versionCodes", [])] if len(found) != 1 or args.dry_run: edits.delete(**ids).execute() if len(found) != 1: sys.exit("version code %s is not on track %s" % (args.version_code, args.track)) release = change(found[0], args) print(json.dumps(release)) if args.dry_run: print("dry run: edit deleted, nothing committed") return 0 # Send only the changed release: in an update, releases are the desired changes. body = {"track": args.track, "releases": [release]} edits.tracks().update(track=args.track, body=body, **ids).execute() edits.commit(**ids).execute() print("committed: %s %s %s on %s" % ( args.package, args.version_code, args.action, args.track)) return 0 if __name__ == "__main__": sys.exit(main(sys.argv[1:])) ``` Do not edit the app in Play Console while the script runs: Google discards an API edit if anyone changes the app in the console meanwhile[^playedits]. **Who can reach which credential.** - **App Store Connect.** "Team API keys can access all apps, regardless of their role"[^ascapikeys]. The tokens fastlane signs last at most 20 minutes[^fastlaneasc]; the key file is the risk, so keep it out of repositories and revoke it if it leaks. Give the beta lane a Developer-role key, which may upload builds[^uploadbuilds], and only the protected release job an App Manager key, which phased-release changes require[^applephased]. - **Google Play.** fastlane's setup recommends "Admin (all permissions)"[^flsupply]; grant app-level permissions instead: "Release apps to testing tracks" for CI, "Release to production, exclude devices, and use Play App Signing" only for the approved release job[^playperms]. - **Signing.** `readonly` stops match creating certificates, but "Every team member with access to the selected storage can use those credentials for code signing"[^match]: give signing jobs read-only storage access and the passphrase, and keep both out of pull-request builds. - **Key files.** Inject keys from the CI secret store at run time. The script's Application Default Credentials can use a Workload Identity Federation configuration instead of a service-account key, which Google calls "a security risk"[^gcpadc] (not tested with the Play API here). ### Hands-on labs Labs 1 and 2 run locally with Python 3.8 or later, need no accounts, cost nothing and change nothing outside their folder. #### Lab 1 (beginner): decide whether a rollout step may advance **Objective:** turn per-version crash numbers into ADVANCE, HOLD or HALT, with a statistical guard against small samples. **Prerequisites:** Python 3.8+; 30 minutes. 1. Save the script as `rollout_gate.py` and the data as `rollout.json` in an empty folder. 2. Run `python3 rollout_gate.py rollout.json; echo "exit=$?"`. 3. Change the candidate's `crashed_sessions` to `70` and `users_with_crash` to `61`; run again. 4. Change the candidate to `"sessions": 9000, "crashed_sessions": 17, "users": 3400, "users_with_crash": 15`; run again. 5. Change it to `"sessions": 18000, "crashed_sessions": 49, "users": 4000, "users_with_crash": 41`; run again. 6. Replace the numbers with your crash reporter's per-version export for the last 24 hours. _Example status: Python and JSON syntax-checked; executed 2026-10-18 with Python 3.8.10; outputs below are copied from the run._ ```python """Decide ADVANCE, HOLD or HALT for one staged-rollout step. Compares the candidate version's crashed-session rate with the baseline version's, using a Wilson score interval. Exit codes: 0 = ADVANCE, 1 = HOLD, 2 = HALT (usable as a CI gate). """ import json import math import sys def wilson(events, trials, z): """Wilson score interval for a proportion events/trials.""" if trials == 0: return 0.0, 1.0 p = events / trials denom = 1 + z * z / trials centre = (p + z * z / (2 * trials)) / denom half = z * math.sqrt(p * (1 - p) / trials + z * z / (4 * trials * trials)) return centre - half / denom, centre + half / denom def decide(data): base, cand, pol = data["baseline"], data["candidate"], data["policy"] base_rate = base["crashed_sessions"] / base["sessions"] cand_rate = cand["crashed_sessions"] / cand["sessions"] lo, hi = wilson(cand["crashed_sessions"], cand["sessions"], pol["z"]) limit = base_rate * pol["relative_risk_halt"] print("baseline %s crash-free sessions %.3f%% users with crash %.3f%%" % ( base["version"], 100 * (1 - base_rate), 100 * base["users_with_crash"] / base["users"])) print("candidate %s crash-free sessions %.3f%% users with crash %.3f%%" % ( cand["version"], 100 * (1 - cand_rate), 100 * cand["users_with_crash"] / cand["users"])) print("candidate crashed-session rate %.4f%% (95%% CI %.4f%%-%.4f%%)" % ( 100 * cand_rate, 100 * lo, 100 * hi)) print("halt limit (baseline x %.2f) %.4f%%" % ( pol["relative_risk_halt"], 100 * limit)) # HALT first: the interval already allows for a small sample, and a # minimum-users rule must never suppress a halt. if lo > limit: return "HALT", "regression is clear: lower bound above the limit" if cand["users"] < pol["min_users"]: return "HOLD", "only %d users; policy needs %d to advance" % ( cand["users"], pol["min_users"]) if hi <= limit: return "ADVANCE", "upper bound within the limit" return "HOLD", "inconclusive: keep this step and collect more data" def main(path): with open(path) as f: data = json.load(f) verdict, reason = decide(data) print("decision: %s (%s)" % (verdict, reason)) return {"ADVANCE": 0, "HOLD": 1, "HALT": 2}[verdict] if __name__ == "__main__": sys.exit(main(sys.argv[1] if len(sys.argv) > 1 else "rollout.json")) ``` ```json { "baseline": {"version": "6.3.2", "sessions": 1840000, "crashed_sessions": 2208, "users": 410000, "users_with_crash": 1640}, "candidate": {"version": "6.4.0", "sessions": 52000, "crashed_sessions": 140, "users": 19800, "users_with_crash": 118}, "policy": {"min_users": 5000, "relative_risk_halt": 1.5, "z": 1.96} } ``` **Expected output (step 2):** ```text baseline 6.3.2 crash-free sessions 99.880% users with crash 0.400% candidate 6.4.0 crash-free sessions 99.731% users with crash 0.596% candidate crashed-session rate 0.2692% (95% CI 0.2282%-0.3176%) halt limit (baseline x 1.50) 0.1800% decision: HALT (regression is clear: lower bound above the limit) exit=2 ``` Step 3 prints `decision: ADVANCE (upper bound within the limit)`, exit 0; step 4 prints `decision: HOLD (only 3400 users; policy needs 5000 to advance)`, exit 1; step 5 (the worked example's day-1 cohort) prints `candidate crashed-session rate 0.2722% (95% CI 0.2060%-0.3597%)` and `decision: HALT`, exit 2. **Validation:** 99.73% crash-free looks healthy, yet the version crashes more than twice as often as its predecessor; step 5 halts below `min_users` because the interval already allows for the small sample. **Troubleshooting:** `KeyError` = a field name differs from your export; `ZeroDivisionError` = an empty baseline. **Caveats:** one user's sessions are not independent, so the interval is optimistic; early updaters and first-day sessions differ from a steady-state baseline, so compare the same day and time since update where you can. **Teardown:** delete the folder. #### Lab 2 (intermediate): size the version tail and test a minimum-version policy **Objective:** see who a soft and a hard minimum would affect, and which old endpoints can be retired. **Prerequisites:** Lab 1's setup; 30 minutes. 1. Save the script as `version_tail.py` and the data as `tail.json`. 2. Run `python3 version_tail.py tail.json; echo "exit=$?"`. 3. Raise `hard_min` to `6.5.0` and run again: both endpoints become retirable (exit 0), but 53,000 users (10.6%) are blocked. 4. Replace the data with your analytics' weekly active users per version for one platform. _Example status: Python and JSON syntax-checked; executed 2026-10-18 with Python 3.8.10 (steps 2 and 3)._ ```python """Report the old-version tail and what a minimum-version policy would do.""" import json import sys def vkey(text): """'6.10.0' -> (6, 10, 0). Never compare version strings as text.""" return tuple(int(part) for part in text.split(".")) def main(path): with open(path) as f: data = json.load(f) users = data["weekly_active_users"] pol = data["policy"] total = sum(users.values()) hard, soft = vkey(pol["hard_min"]), vkey(pol["soft_min"]) print("text comparison says 6.10.0 < 6.9.0: %s (wrong)" % ("6.10.0" < "6.9.0")) print("tuple comparison says 6.10.0 < 6.9.0: %s" % (vkey("6.10.0") < vkey("6.9.0"))) blocked = prompted = 0 for version in sorted(users, key=vkey, reverse=True): n = users[version] if vkey(version) < hard: action, blocked = "BLOCK", blocked + n elif vkey(version) < soft: action, prompted = "PROMPT", prompted + n else: action = "ok" print("%-8s %8d %5.1f%% %s" % (version, n, 100.0 * n / total, action)) print("blocked by hard minimum %s: %d users (%.1f%%)" % ( pol["hard_min"], blocked, 100.0 * blocked / total)) print("prompted by soft minimum %s: %d users (%.1f%%)" % ( pol["soft_min"], prompted, 100.0 * prompted / total)) status = 0 for ep in data["endpoints"]: # Blocked versions only reach the version-check endpoint, so they # stop counting for other endpoints once the hard block is live. still = sum(n for v, n in users.items() if hard <= vkey(v) < vkey(ep["needed_below"])) share = 100.0 * still / total ok = share <= pol["retire_budget_pct"] print("%s still needed by %.1f%% of users: %s" % ( ep["path"], share, "can retire" if ok else "keep serving")) status = status if ok else 1 return status if __name__ == "__main__": sys.exit(main(sys.argv[1] if len(sys.argv) > 1 else "tail.json")) ``` ```json { "weekly_active_users": { "6.10.1": 205000, "6.10.0": 92500, "6.9.2": 71000, "6.8.0": 45500, "6.5.3": 33000, "6.2.0": 22000, "5.9.1": 16000, "5.4.0": 9000, "4.12.0": 6000 }, "policy": {"hard_min": "5.9.0", "soft_min": "6.8.0", "retire_budget_pct": 1.0}, "endpoints": [ {"path": "/v1/cart", "needed_below": "6.5.0"}, {"path": "/v1/login-legacy", "needed_below": "5.0.0"} ] } ``` **Expected output (step 2):** ```text text comparison says 6.10.0 < 6.9.0: True (wrong) tuple comparison says 6.10.0 < 6.9.0: False 6.10.1 205000 41.0% ok 6.10.0 92500 18.5% ok 6.9.2 71000 14.2% ok 6.8.0 45500 9.1% ok 6.5.3 33000 6.6% PROMPT 6.2.0 22000 4.4% PROMPT 5.9.1 16000 3.2% PROMPT 5.4.0 9000 1.8% BLOCK 4.12.0 6000 1.2% BLOCK blocked by hard minimum 5.9.0: 15000 users (3.0%) prompted by soft minimum 6.8.0: 71000 users (14.2%) /v1/cart still needed by 7.6% of users: keep serving /v1/login-legacy still needed by 0.0% of users: can retire exit=1 ``` **Validation:** the hard minimum, not time, is what lets you retire `/v1/login-legacy`. **Troubleshooting:** `ValueError` on a version such as `6.10.0-beta` — strip suffixes before comparing. **Teardown:** delete the folder. #### Lab 3 (advanced, not executed): halt automatically on a test track **Objective:** wire Lab 1's exit code to `play_rollout.py`. **Prerequisites:** a staged release on a closed testing track (Play allows staged rollouts on test tracks[^playstaged]); a CI identity with "Release apps to testing tracks" for that app only[^playperms]. 1. Schedule a CI job that runs `python3 rollout_gate.py export.json`. 2. On exit code 2, run `python3 play_rollout.py com.example.shop 6400 halt --track --dry-run`, then without `--dry-run`. 3. Feed the job Lab 1's step-5 numbers. **Expected output:** the dry run prints the release with `"status": "halted"`; the real run prints `committed: com.example.shop 6400 halt on `. **Validation:** Play Console shows the test release as halted. **Teardown:** resume it with `advance` and its previous fraction, or discard the test release; delete the job. ### Use cases and scenarios Four documented cases, each a single organisation's account: | Case | Context and method | Outcome | Lesson | |---|---|---|---| | Facebook, 2017[^fbrapid] | Web moved to quasi-continuous pushes; mobile kept a weekly train, with daily release candidates for beta testers and Gatekeeper flags separating code from features | Mobile teams grew 15-fold; critical issues "almost constant regardless of the number of deployments" | Mobile gets a faster train, not continuous deployment | | Squarespace Unfold, 2024[^squarespace] | One-week trains, rotating captains, submission on Friday and release on Monday if approved, staged rollouts with automatic halts from Crashlytics and Amplitude data | Hotfixes fell "from many a month to 5 times a year" | Numeric halt rules made rotation work | | Facebook iOS SDK, 2020[^fbsdk1373][^siliconangle] | Apps crashed in SDK code reading server-supplied settings; SiliconANGLE, citing The Verge, reported that Facebook disabled "the server configuration update" | Spotify, TikTok, Pinterest and others crashed until the fix; a similar crash recurred in July 2020 | A remote change can break every installed version at once | | Sonos, 2024[^sonosblog][^sonospr] | A rebuilt app replaced the old one on 7 May in an "all-at-once automated app release" | Fixes every two weeks for months; Sonos promised that "any major change to the Sonos app will be released gradually" | Staging is a customer-trust decision | ::: example Illustrative scenario: a security fix that must reach everyone A banking app fixes a session-handling flaw in 7.2.1. After two weeks of soft prompts 4% remain on older versions; with 7.2.1 at 100% on both stores, the team sets a hard minimum of 7.2.1 and shows users whose phones cannot run it web-banking instructions. Success measure: under 0.5% of sessions on vulnerable versions within 30 days. ::: ## Measure, govern, pay ### Metrics and KPIs | KPI | Definition or formula | Data source | Pitfall | |---|---|---|---| | Crash-free sessions, per version | 1 − crashed sessions ÷ all sessions[^crashfree] | Crash reporter | An aggregate over the period, not an average; fatal crashes only | | Crash-free users | Users who engaged without a crash ÷ users who engaged[^crashfree] | Crash reporter | "shouldn't be compared over different time periods"[^crashfree] | | User-perceived crash and ANR rate | Daily active users with at least one crash (or ANR) while actively using the app ÷ daily active users[^vitalscrash] | Android vitals | Above 1.09% crash or 0.47% ANR overall, Play may reduce visibility[^vitals] | | Relative crash risk at a step | Candidate crashed-session rate ÷ previous version's rate | Crash reporter | Small samples ([Lab 1](#hands-on-labs)) | | Time to contain | First alert → flag off or rollout halted | Release record | Long when nobody may halt without approval | | Kill-switch propagation | Time until 90% of active users run the new flag value | Config service logs | Fetch intervals and next-start activation[^rcloading] | | Adoption | Active users on the newest version at 7 and 14 days | Analytics | Phased release throttles only automatic updates | | Tail outside the window | Active users below soft and hard minimum ÷ active users | Analytics | Users who block analytics are invisible | | Store lead time | Submission → approval; rejection rate | Store consoles | Requeueing on Play when you resubmit[^playreview] | **Worked calculation: how much a rollout step can see.** Assume 400,000 users with automatic updates on, 4.5 sessions per user per day, a previous version with a crashed-session rate of 0.12%, and a halt limit of 1.5× (0.18%), evaluated with Lab 1's 95% interval on one day of data per step (illustrative inputs, computed 2026-10-18): | Apple step | Users | Sessions | New version at 0.27% (2.25×) | New version at 0.21% (1.75×) | |---|---|---|---|---| | Day 1, 1% | 4,000 | 18,000 | 49 crashes, lower bound 0.206% → HALT | 38 crashes, 0.154–0.290% → HOLD | | Day 2, 2% | 8,000 | 36,000 | HALT | 0.169–0.264% → HOLD | | Day 3, 5% | 20,000 | 90,000 | HALT | lower bound 0.182% → HALT | A large regression shows on day one; a moderate one only after 5% of users have it. A small app may never have enough users at 1%: size steps by **users**, not percentages. Between steps, Play's Reporting API offers hourly crash rates by `versionCode`[^reportingapi]. On iOS, MetricKit sends metric reports "at most once per day", but "Diagnostic reports arrive immediately in iOS 15 and later", crash diagnostics included[^metrickit]; they arrive on the device, so the app or a crash SDK must upload and aggregate them before a gate can use them. Crashlytics velocity alerts are not a gate: they are not sent by default, and the default threshold, one issue in 1% of sessions and 25 users within 30 minutes[^crashalerts], is more than five times this policy's whole-version limit. **What the numbers cannot see.** Android vitals collects data only "When a user allows it"[^vitals], Xcode's Crashes organizer only from "customers who share diagnostic and usage information"[^applecrash], and opt-in reporting makes Crashlytics metrics "less reliable"[^crashfree]. Compare versions with each other, not with an absolute target. For DORA measures, define a mobile "deployment" (this Atlas suggests: a version reaching 100% of its rollout); see [DevOps](../devops/). ### Quality, security, privacy and compliance - **Signing keys.** On Google Play, a lost upload key can be reset, but a self-managed app signing key "cannot be reset" if lost [^playsigning]. On Apple, an expired App Store distribution certificate leaves live apps untouched but blocks new uploads; an expired in-house (enterprise) certificate stops apps from running; an expired push certificate stops notifications[^applecerts]. Who may reach each key is set out under the [reference implementation](#reference-implementation). - **Security baseline.** OWASP MASVS groups controls from storage to privacy[^masvs]; MASVS-CODE-1 requires an up-to-date platform version and MASVS-CODE-2 an update-enforcement mechanism[^masvscode]. Security testing belongs to [DevSecOps](../devsecops/). - **Privacy of telemetry.** Crash logs and diagnostics are data types to declare in Google Play's Data safety form, including data sent by SDKs[^datasafety]; on Apple, apps and SDKs declare collected data in a `PrivacyInfo.xcprivacy` manifest[^privacymanifest]. MASVS lists "Insertion of Sensitive Data into Logs" as a weakness (MASWE-0005)[^masvs]. Consent: [PrivacyOps](../privacyops/). - **Store compliance.** Apple may remove apps that are not compatible with the current OS release (licence section 6.8)[^dpla]. ### Cost, capacity and sustainability **CI.** On 2026-10-18 GitHub listed $0.062 a minute for a standard macOS runner and $0.006 for a 2-core Linux runner[^ghapricing]. Thirty iOS builds a day at 20 minutes for 22 working days is 13,200 minutes: about $818 a month on macOS against about $79 on Linux (illustrative arithmetic); Xcode Cloud includes 25 compute hours a month[^xcodecloud]. Cache builds, build iOS only when iOS code changes, and run device suites on a schedule. **Devices.** Firebase Test Lab charges $5 an hour per physical and $1 per virtual device beyond its free allowance[^ftlpricing]; its successor matches those rates only until 2027-04-30[^ftlfaq]. Flaky tests multiply the bill; Slack's mobile teams (16,000+ Android and 11,000+ iOS tests) noted they "increase execution cost"[^slack]. **People.** Squarespace estimated losing "nearly a third" of engineering time to release coordination before automating its train[^squarespace]. Fewer redundant builds and device runs also cut energy use. ### Assurance of the operating model - **Each release:** a release record (template below) with gate values at each step and who advanced, halted or resumed. - **Monthly:** review every halt — did a gate fire before support tickets arrived, and were any halts false alarms? - **Quarterly:** credential and role audit; a kill-switch drill measuring propagation; a signing-store restore test; the device matrix rebuilt from usage. - **Yearly:** platform-floor planning for Apple's SDK requirement and Google's target API level. ## When things go wrong ### Common failures and troubleshooting | Symptom | Likely cause | Diagnosis | Fix and prevention | |---|---|---|---| | iOS upload rejected, or pushes stop | Expired or revoked distribution or push certificate | Certificates in the developer account | Admin renews; CI read-only; alert 60 days ahead[^applecerts] | | Play rejects the upload | `versionCode` not higher than before | Compare with the track's version codes | Derive from a CI counter; watch the 2,100,000,000 cap[^playversion] | | Crash reports unreadable, or one crash split into many groups | No dSYM or mapping file for this version | "Missing dSYM" alert in Crashlytics; obfuscated stacks in Play Console | Upload symbols in the release lane (Play reads mapping files from bundles built with Android Gradle plugin 4.1+); earlier crashes stay unreadable[^crsymios][^playdeob] | | Blocked users see an update-required screen but no update | Block raised before the replacement reached them (staged rollout, countries, OS floor) | Replacement's rollout percentage, countries, minimum OS | Back to a soft prompt; block after the rollout completes[^playstaged] | | Crashes rise on **every** version at once | Server, config or SDK change | Crash timeline versus config and deploy logs | Revert the change; parse remote data defensively[^fbsdk1373] | | Crashes rise on the **new** version only | Client regression | Per-version crash groups | Flag off, halt or pause, hotfix | | An OTA update crashes some builds | Runtime version mismatch | Crashes by build and update ID | Republish the previous update; use an automatic runtime policy[^easruntime] | | A kill switch "does not work" | Fetch interval or next-start activation | Config fetch logs | Real-time updates; measured propagation[^rcloading][^rcstart] | | Old versions fail after an API deploy | Breaking change: removed field, changed default | Errors by app version | Contract tests against the oldest supported version[^aip180] | | Production users get immediate-update prompts | High priority on an internal-track build: priority counts "irrespective of the release track" and cannot be changed later | Release priorities | Set priorities only on production[^inappkotlin] | | Updates refused after a deadline | SDK or target-API floor | Store messages | Track floors on the calendar[^appleupcoming][^playtarget] | ### Worked runbook: crash spike during a staged rollout **Trigger (illustrative):** a velocity alert for 6.4.0 two hours after the Android 5% step and Apple's day 3. Roles: release captain (levers), feature owner (flags), backend on-call. First question: **which versions crash?** ::: figure Triage starts with which versions crash ```mermaid flowchart TB A["Alert: crash spike"] --> B{"Which versions crash?"} B -->|"new version only"| C["Flag off if feature-linked"] C --> D["Halt Play rollout, pause Apple phased release"] D --> E{"Fixed by flag or server?"} E -->|"yes"| F["Verify, resume at same step"] E -->|"no"| G["Hotfix N+1, expedited review"] B -->|"all versions"| H["Server, config or SDK change"] H --> I["Revert change, contact SDK vendor"] I --> J["Verify crash rate back to baseline"] G --> K["Block bad version once N+1 reaches all"] classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029 classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029 classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 class A,H risk class B,E decision class F,J,K core ``` Text equivalent: first ask which versions crash. New version only: flag off if feature-linked, halt on Play and pause on Apple; if the flag or a server fix cured it, verify and resume at the same step, otherwise ship N+1 with expedited review and block the bad version once N+1 can reach all its users. All versions: find and revert the server, config or SDK change, contact the SDK vendor, and verify the crash rate. ::: 1. **Scope (5 minutes).** Crash groups by version and OS: here 6.4.0 only, in the new checkout screen, on both platforms. 2. **Flag off.** The feature owner sets `new_checkout=false`; note the time. 3. **Stop the binary.** Android: `python3 play_rollout.py com.example.shop 6400 halt`. iOS: pause the phased release in App Store Connect, or send the body below with `PATCH /v1/appStoreVersionPhasedReleases/{id}`[^ascphased]; manual downloads continue. Removing the app from sale ends the phased release: reinstated, the version "becomes available to all users immediately, regardless of the percentage reached before removal", and only a new version restores a phased rollout[^applephased]. Users who already have 6.4.0 keep it[^playtracks]. 4. **Verify (30–60 minutes).** 6.4.0's crash rate back near the baseline? Then resume at the same step with the flag off and fix forward in the next train. 5. **If not:** build 6.4.1 and request expedited review with steps to reproduce[^applereview]. On Play, a new staged release started before the previous one completes "will use the same group of users as the previous release (depending on the percentage of the rollout)"[^playstaged], so start 6.4.1 at 6.4.0's percentage, not lower (this page's reading). Soft-prompt 6.4.0 users; add `6.4.0` to `blocked` only when every 6.4.0 user can get 6.4.1 — on Android, once 6.4.1 has completed its rollout ([forced upgrades](#forced-upgrades-soft-prompts-hard-blocks-and-what-each-platform-offers)). 6. **Communicate.** Give support a status line; if users were harmed, open an incident ([IncidentOps](../incidentops/)). 7. **Record.** Alert, flag and halt times and decisions; follow-up on why the beta and the 1% step missed it. _Example status: JSON syntax-checked; fields compared by hand with Apple's documented request schema; not sent (needs an API key)._ ```json { "data": { "type": "appStoreVersionPhasedReleases", "id": "", "attributes": {"phasedReleaseState": "PAUSED"} } } ``` ### Anti-patterns and myths - **"We'll roll back if it breaks."** Halting stops new installs only; plan the flag, the fix-forward and the block in advance. - **"99.7% crash-free is fine."** It may be twice as bad as the last version ([Lab 1](#hands-on-labs)); compare, do not admire. - **"Flags or OTA let us skip review."** New features must be described and accessible to reviewers, and OTA is limited to interpreted code that does not change the app's purpose[^appleguidelines][^dpla]. - **"Support every version forever"** — or its opposite, **"force everyone to update weekly".** Both cost more than a published window. - **"Signing lives on one laptop."** Store identities centrally, read-only for CI, with named administrators. - **"Our device-lab vendor is permanent."** App Center retired in 2025 (crash reporting lasts to March 2027); Firebase Test Lab shuts down in 2027[^appcenter][^ftlfaq]. ## Deciding and adopting ### When to use and when not to use **Invest in the full practice when** a native app ships through public stores, a bad release costs money, safety or trust, more than a few thousand people use it daily, and you release at least monthly. **Keep it light, or do something else, when:** - **It is a web app or PWA.** You can roll back; see [WebOps](../webops/). - **It is an internal app on managed devices.** Distribution is [EndpointOps](../endpointops/) territory; keep CI, signing and a minimum-version check. With Apple in-house distribution, certificate expiry stops the app from running[^applecerts]. - **It is a prototype before launch.** Do only [Day 0](#lifecycle-day-0-day-1-day-2): it cannot be added to versions already installed. **Hidden responsibilities:** macOS CI images, store administration, a version-check endpoint that must work forever, API compatibility for every supported version, and yearly platform floors. ### Decision framework ::: figure How much MobileOps machinery do you need? ```mermaid flowchart TB A{"Native app in\npublic stores?"} -->|"no"| B["Web or managed fleet:\nWebOps or EndpointOps"] A -->|"yes"| C{"Launched yet?"} C -->|"no"| D["Day 0 kit: CI, signing,\ncrash, config,\nversion check"] C -->|"yes"| E{"Costly failures\nor many users?"} E -->|"no"| F["Light: monthly train,\nmanual rollouts"] E -->|"yes"| G["Full: weekly train,\nnumeric gates,\nkill switches"] G --> H{"Interpreted-code\nlayer?"} H -->|"yes"| I["Add OTA with\nruntime-version policy"] H -->|"no"| J["Stop here:\nno OTA tooling"] classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029 classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029 class A,C,E,H decision class D,F,G,I,J core class B neutral ``` Text equivalent: not a native store app → WebOps or EndpointOps. Not launched → the Day 0 kit (CI, signing, crash reporting, config, version check). Launched with cheap failures and few users → a light process (monthly train, manual rollouts). Costly failures or many users → the full practice (weekly train, numeric gates, kill switches), plus OTA with a runtime-version policy if the app has an interpreted-code layer; without one, stop there. ::: ### Maturity model This is **this Atlas's model**, not an industry standard. | Level | Observable behaviour | Exit criterion | |---|---|---| | 1 Ad hoc | Builds from laptops; one person can sign; releases when ready | Every build comes from CI with shared, read-only signing | | 2 Repeatable | CI builds, beta channel, crash reporting | Every production release is staged, with a named captain | | 3 Gated | Numeric per-version gates; anyone on rotation can halt | A halt has happened before support noticed | | 4 Recoverable | Kill switches, published support window, API contract tests against the oldest supported version | A drill turned a feature off and blocked a version within an hour | | 5 Measured | Automatic advance and halt, tail and cost tracked, platform floors planned a year ahead | Hotfix rate and tail outside the window trend down for two quarters | ### Adoption roadmap: 30/60/90 days - **Days 1–30:** inventory signing identities, store roles and expiry dates; move signing to a shared store; build both platforms in CI with symbol upload; write the rollout policy with provisional thresholds. - **Days 31–60:** start a fortnightly train with a captain rotation; stage every release; add a kill switch to the next risky feature; ship the minimum-version check and the support-policy endpoint; measure kill-switch propagation in beta. - **Days 61–90:** run Lab 1's gate on real exports and tune thresholds; publish the support window; add contract tests against the oldest supported version; review the first halts; decide on weekly trains. ## Toolkit ### Checklists **Release readiness (before submission)** - [ ] Build from CI, signed from the shared store; build number higher than any previous upload; symbols uploaded - [ ] Every risky feature behind a flag that defaults to off, tested on and off in beta - [ ] New features described in the review notes and reachable by the reviewer - [ ] API changes checked against the oldest supported version - [ ] Release notes list significant changes; data disclosures updated if an SDK or data type changed **Rollout watch (each step)** - [ ] Gate values recorded per version: crashed sessions, ANRs, key business event, minimum users reached - [ ] Velocity alerts switched on, thresholds tuned, routed to the captain and the on-call engineer - [ ] Halt and pause commands tested this quarter; the captain knows who may resume **Quarterly operations review** - [ ] Certificates, keys and push credentials more than 60 days from expiry; store roles reviewed - [ ] Version tail, support window and blocked list reviewed with backend owners - [ ] Platform deadlines and tool retirements on the calendar ### Templates ```text RELEASE RECORD app: ______ version: ______ build/versionCode: ______ Captain: ______ Train date: ______ Review submitted / approved: ____ / ____ Flags introduced (owner, default, expiry): ________________________________ Step | Date-time | Users | Crashed-session ratio | ANR ratio | Decision | By 1% | | | | | | ... Halts and resumes (time, reason, who): ___________________________________ Follow-ups (owner, due date): ____________________________________________ ``` ```text SUPPORT-WINDOW POLICY platform: ______ owner: product + mobile platform Soft minimum: ______ (prompt; reviewed monthly) Hard minimum: ______ (block; needs product, platform and backend sign-off) Blocked versions: ______ (known-bad builds) Rule: never break an API used by a version at or above the hard minimum Rule: the version-check response stays readable by every version ever shipped Rule: block only when the replacement reaches every blocked user (Play rollout complete, all targeted countries, their OS supported) Users unable to update (OS below our minimum): alternative channel ______ ``` ### Deep comparisons | Distribution model | Containing a bad version | Review | Fits | |---|---|---|---| | Native app via public stores | Flags, halt or pause, hotfix, minimum versions | Every binary | Apps that need device APIs or a store presence | | Cross-platform app with an OTA layer | As above, plus republishing a previous interpreted-code update[^easintro] | Binary yes; OTA within store rules | Teams with much logic in JavaScript | | Web app or PWA | Server rollback | None | Content and transactional apps that need no device APIs ([WebOps](../webops/)) | | Apple in-house distribution | You control distribution; certificate expiry stops the app[^applecerts] | None by the App Store | Internal apps for employees ([EndpointOps](../endpointops/)) | ### FAQ **Is "MobileOps" a real discipline?** The work is; the name is rare. Use "mobile DevOps" or "mobile release engineering" when hiring or searching ([term status](#in-30-seconds)). **Can I roll back an app release?** No. You can limit its spread, turn features off and move users on ([levers](#platform-levers-compared-app-store-and-google-play)); even Play's halt of a fully rolled-out release leaves updated users on the halted version[^playhaltfull]. **How many old versions should we support?** Decide from the tail data ([Lab 2](#hands-on-labs)), the cost of API compatibility and the users you would block. This Atlas suggests a soft minimum about three months back, and moving the hard minimum only for security or cost reasons. **Do we need a device farm?** Start with emulators, simulators and a few real devices matching your users' commonest models and OS versions; add a cloud lab when device-specific crashes appear, and check its future[^ftlfaq]. **Should mobile use DORA metrics?** Yes, once you define a mobile "deployment" ([metrics](#metrics-and-kpis)). ## Going further ### Advanced topics and emerging direction - **Moving fast:** device-lab consolidation, with new Developer Device Platform pricing after 2027-04-30[^ftlfaq]; memory metrics that may affect Play visibility from February 2027[^vitals]; agent-reserved test devices ([automation](#automation-and-ai))[^ftlfaq]. - **To evaluate, not adopt blindly:** moving more behaviour to the server so fewer changes need a binary, within the stores' rules on downloaded code[^dpla][^playdna]. - **Regional distribution rules:** Apple's guidelines refer to alternative distribution, which changes store gating where it applies[^appleguidelines]; check jurisdiction before relying on it. ### Learning path - **Prerequisites:** CI/CD basics ([DevOps](../devops/)); deploy versus release ([ReleaseOps](../releaseops/)). - **One day:** read the two store pages on phased and staged release[^applephased][^playstaged]; run Labs 1 and 2. - **One week:** build a beta lane with fastlane; add crash reporting and a kill switch to a sample app. - **Thirty days:** run four trains with written gates; ship a minimum-version check; write the support-window policy. - **Specialise:** mobile security ([DevSecOps](../devsecops/), MASVS), API evolution ([API Ops](../api-ops/)), or test infrastructure ([TestOps](../testops/)). - **Portfolio project:** an open-source sample app with CI, staged releases, a support-policy endpoint and a gate script. ## Wrap-up ### Executive recap and cheat sheet - **Decide:** release cadence, gate thresholds, who may halt and resume, the support window. - **Build once, early:** shared signing, CI, crash reporting per version, flags with safe defaults, a minimum-version check. - **Watch:** relative crash and ANR rates per version, time to contain, kill-switch propagation, tail outside the window, credential headroom. - **Remember:** no recall; approval is not release; old versions live for years; block only what users can replace. - **Next action:** list every signing identity and its expiry date this week. ### A judgement call: block the bad version now, or wait? **Situation (illustrative).** Version 6.4.0 crashes in checkout for 3% of its sessions on iOS and already runs on 40% of active iOS users (the phased release reached day 6 before the cause was clear). Fixed 6.4.1 was approved this morning. Option A: set `hard_min` to 6.4.1 now, so every 6.4.0 user must update. Option B: turn checkout off server-side for 6.4.0, show a soft prompt, and let 6.4.1 spread through automatic updates. **Assumptions.** The flag stops the crash (verified within 30 minutes); checkout is a third of revenue; some users have automatic updates off; 6.4.1 has been live for only hours. **Reasoning.** Option A feels decisive, but it blocks the whole app, not just checkout, for 40% of users, and puts 6.4.1 on every device before its own rollout has data, making any 6.4.1 defect universal (on Android it would also lock out users whom 6.4.1's staged rollout has not reached). Option B removes the crash at once and lets 6.4.1 pass its own gates; its cost is lost checkout revenue on 6.4.0 until users update. **Choose B**, then block 6.4.0 once 6.4.1 has passed its gates. Choose A only if the defect corrupts data or is a security flaw the flag cannot neutralise. **Uncertainty:** how fast users with automatic updates off move; watch the 6.4.0 share daily. ### Learning objectives and mastery check After this page you should be able to **explain** why mobile releases cannot be recalled; **compare** Apple's and Google's rollout levers; **design** gates, kill switches and a support window; **run** a containment runbook; and **evaluate** OTA and forced-upgrade choices. 1. Fixed 6.4.1 is at 20% of a Play staged rollout. Product wants `hard_min` raised to 6.4.1 today. What happens? ::: answer Show answer for question 1 About 80% of Android users, those outside the rollout, would be blocked with no update available: a staged rollout reaches "only a percentage of your users"[^playstaged]. Soft-prompt now; raise the hard minimum after 6.4.1 completes its rollout. ::: 2. Version 7.1 shows 99.81% crash-free sessions; 7.0 showed 99.90%. Is 7.1 fine? ::: answer Show answer for question 2 Not necessarily: its crashed-session rate (0.19%) is 1.9 times 7.0's (0.10%). With enough users that exceeds a 1.5× gate; with few users, a clear regression must still halt, since a minimum-users rule may delay an advance but never a halt (Lab 1, step 5). ::: 3. Your team wants to add a forced-upgrade screen "when we need it". Why is that too late? ::: answer Show answer for question 3 The check must already be inside the installed versions. Play Console's prompt is the exception on Android[^playprompt]; on iOS the documentation checked shows no store-side equivalent. ::: 4. Can you use an OTA update to enable a new payment feature that review has not seen? ::: answer Show answer for question 4 No. Apple forbids downloaded code that changes the app's primary purpose and requires new features to be accessible for review[^dpla][^appleguidelines]; Google requires interpreted code not to enable policy violations[^playdna]. ::: 5. Every app version, old and new, starts crashing at launch at the same minute. What do you check first? ::: answer Show answer for question 5 A server, config or third-party SDK change, as in the 2020 Facebook SDK incident[^fbsdk1373]. Halting the rollout will not help; revert the change. ::: ### Conclusion "MobileOps" is a rare name for a real job: shipping software that, once installed, is out of reach. The stores add a gate you do not control, and users decide when to update. Everything useful here follows: release in steps large enough to notice harm, keep the fast levers on the server, put the update check and the kill switch into version one, and support a window of versions on purpose. Teams that treat a phone like a server rediscover the difference during their worst release. ## Sources and further reading Every source was opened on 2026-10-18 (two re-checked 2026-10-19); dates are the source's own. Store rules and limits change: recheck before use. Claim-level support is in the Atlas evidence ledger (`research/mobileops-ledger.md`). [^wiki]: [Mobile DevOps](https://en.wikipedia.org/wiki/Mobile_DevOps) — Wikipedia, last edited 2026-09-16 (cites Tak and Modi, *Mobile DevOps*, Packt, 2018; Nabil, *Mobile DevOps Playbook*, Packt, 2023). Checked 2026-10-18. [^orosz]: [10 Engineering Challenges Due to the Nature of Mobile Applications](https://blog.pragmaticengineer.com/10-engineering-challenges-due-to-the-nature-of-mobile-applications/) — Gergely Orosz, The Pragmatic Engineer, 2020-12-09. Checked 2026-10-18. [^trinetri]: [MobileOps: AI-Powered Mobile Device Management](https://trinetriops.com/platform/mobile-device-management-mdm) — Trinetri, undated. Checked 2026-10-18. [^abavus]: [MobileOps: Mobile workforce management software](https://abavus.co.uk/mobile-workforce-management-software/) — Abavus, undated. Checked 2026-10-18. [^mobileopscom]: [MobileOps](https://www.mobileops.com/) — MobileOps (maritime operations platform), undated. Checked 2026-10-18. [^bitrise]: [Mobile DevOps Platform for iOS & Android](https://bitrise.io/) — Bitrise, home page. Checked 2026-10-18. [^applereview]: [App Review](https://developer.apple.com/distribute/app-review/) — Apple Developer. Checked 2026-10-18. [^playreview]: [Control when app changes are reviewed and published](https://support.google.com/googleplay/android-developer/answer/9859654) — Play Console Help. Checked 2026-10-18. [^playstaged]: [Release app updates with staged rollouts](https://support.google.com/googleplay/android-developer/answer/6346149) — Play Console Help. Checked 2026-10-18. [^applephased]: [Release a version update in phases](https://developer.apple.com/help/app-store-connect/update-your-app/release-a-version-update-in-phases) — App Store Connect Help. Checked 2026-10-18. [^applerelease]: [Select an App Store version release option](https://developer.apple.com/help/app-store-connect/manage-your-apps-availability/select-an-app-store-version-release-option) — App Store Connect Help. Checked 2026-10-18. [^appleunavail]: [Make a version unavailable for download](https://developer.apple.com/help/app-store-connect/manage-your-apps-availability/make-a-version-unavailable-for-download) — App Store Connect Help. Checked 2026-10-18. [^ascphased]: [App Store Version Phased Releases](https://developer.apple.com/documentation/appstoreconnectapi/app-store-version-phased-releases) — App Store Connect API documentation (with `PhasedReleaseState` and the update request). Checked 2026-10-18. [^vitals]: [Android vitals](https://developer.android.com/topic/performance/vitals) — Android Developers, updated 2026-10-09. Checked 2026-10-18. [^vitalscrash]: [Crashes (Android vitals)](https://developer.android.com/games/optimize/vitals/crash) — Android Developers, updated 2026-02-26. Checked 2026-10-18. [^anr]: [ANRs](https://developer.android.com/topic/performance/vitals/anr) — Android Developers, updated 2026-09-21. Checked 2026-10-18. [^dpla]: [Apple Developer Program License Agreement](https://developer.apple.com/support/terms/apple-developer-program-license-agreement/) — Apple, sections 3.3.1(B) and 6.8. Checked 2026-10-18. [^appleguidelines]: [App Review Guidelines](https://developer.apple.com/app-store/review/guidelines/) — Apple, last updated 2026-10-16. Checked 2026-10-18. [^playdna]: [Device and Network Abuse](https://support.google.com/googleplay/android-developer/answer/16559646) — Google Play Policy Center. Checked 2026-10-18. [^fbrapid]: [Rapid release at massive scale](https://engineering.fb.com/2017/08/31/web/rapid-release-at-massive-scale/) — Chuck Rossi, Engineering at Meta, 2017-08-31. Checked 2026-10-18. [^mnf]: [New Mobile Native Foundation to Foster Development Collaboration](https://www.linuxfoundation.org/press/press-release/new-mobile-native-foundation-to-foster-development-collaboration) — Linux Foundation, 2021-03-02. Checked 2026-10-18. [^squarespace]: [Unfold's Modern Mobile Release Process and the Subtle Art of Making Them Boring](https://engineering.squarespace.com/blog/2024/unfolds-modern-mobile-release-process-and-the-subtle-art-of-making-them-boring) — Alan Cooke, Squarespace Engineering, 2024-07-12. Checked 2026-10-18. [^sonosblog]: [Update on the Sonos app from Patrick](https://www.sonos.com/en-us/blog/update-on-the-sonos-app) — Patrick Spence, Sonos, 2024-07-25. Checked 2026-10-18. [^sonospr]: [Sonos announces new quality and customer experience commitments](https://www.silicon.co.uk/press-release/sonos-announces-new-quality-and-customer-experience-commitments) — Sonos press release via Business Wire, republished by Silicon UK, 2024-10-01. Checked 2026-10-18. [^appcenter]: [Visual Studio App Center Retirement](https://learn.microsoft.com/en-us/appcenter/retirement) — Microsoft Learn, updated 2026-04-15. Checked 2026-10-18. [^ftlfaq]: [Firebase Test Lab Deprecation FAQ](https://firebase.google.com/docs/test-lab/migrate-faq) — Firebase, updated 2026-10-16. Checked 2026-10-18 and 2026-10-19. [^ftlpricing]: [Usage levels, quotas, and pricing for Test Lab](https://firebase.google.com/docs/test-lab/usage-quotas-pricing) — Firebase, updated 2026-10-16. Checked 2026-10-18. [^playversion]: [Version codes](https://support.google.com/googleplay/android-developer/answer/17367361) — Play Console Help. Checked 2026-10-18. [^playsigning]: [Use Play App Signing](https://support.google.com/googleplay/android-developer/answer/9842756) — Play Console Help. Checked 2026-10-18. [^playrollout]: [Prepare and roll out a release](https://support.google.com/googleplay/android-developer/answer/9859348) — Play Console Help. Checked 2026-10-18. [^playperms]: [Add developer account users and manage permissions](https://support.google.com/googleplay/android-developer/answer/9844686) — Play Console Help. Checked 2026-10-18. [^playdeob]: [Deobfuscate or symbolicate crash stack traces](https://support.google.com/googleplay/android-developer/answer/9848633) — Play Console Help. Checked 2026-10-18. [^playhaltfull]: [Halting a fully rolled-out release](https://support.google.com/googleplay/android-developer/answer/16285429) — Play Console Help. Checked 2026-10-18. [^playprompt]: [Prompt users to update to your latest app version](https://support.google.com/googleplay/android-developer/answer/13812041) — Play Console Help. Checked 2026-10-18. [^playtarget]: [Target API level requirements for Google Play apps](https://support.google.com/googleplay/android-developer/answer/11926878) — Play Console Help. Checked 2026-10-18 and 2026-10-19. [^playtracks]: [REST Resource: edits.tracks](https://developers.google.com/android-publisher/api-ref/rest/v3/edits.tracks) — Google Play Developer API, updated 2026-09-28. Checked 2026-10-18. [^playedits]: [Edits](https://developers.google.com/android-publisher/edits) — Google Play Developer API, updated 2025-12-18. Checked 2026-10-18. [^inapp]: [In-app updates](https://developer.android.com/guide/playcore/in-app-updates) — Android Developers, updated 2026-01-30. Checked 2026-10-18. [^inappkotlin]: [Support in-app updates (Kotlin or Java)](https://developer.android.com/guide/playcore/in-app-updates/kotlin-java) — Android Developers, updated 2026-10-16. Checked 2026-10-18. [^testflight]: [TestFlight overview](https://developer.apple.com/help/app-store-connect/test-a-beta-version/testflight-overview) — App Store Connect Help. Checked 2026-10-18. [^appleupcoming]: [Upcoming Requirements](https://developer.apple.com/news/upcoming-requirements/) — Apple Developer News. Checked 2026-10-18. [^xcodereq]: [Xcode system requirements](https://developer.apple.com/xcode/system-requirements/) — Apple. Checked 2026-10-18. [^xcodecloud]: [Xcode Cloud](https://developer.apple.com/xcode-cloud/) — Apple. Checked 2026-10-18. [^applecerts]: [Certificates overview](https://developer.apple.com/help/account/certificates/certificates-overview) — Apple Developer Account Help. Checked 2026-10-18. [^match]: [match](https://docs.fastlane.tools/actions/match/) — fastlane docs. Checked 2026-10-18. [^fastlane]: [fastlane](https://docs.fastlane.tools/) — fastlane docs home. Checked 2026-10-18. [^fastlaneasc]: [Using App Store Connect API](https://docs.fastlane.tools/app-store-connect-api/) — fastlane docs. Checked 2026-10-18. [^flsupply]: [upload_to_play_store](https://docs.fastlane.tools/actions/upload_to_play_store/) — fastlane docs (setup section). Checked 2026-10-18. [^flsymbols]: [upload_symbols_to_crashlytics](https://docs.fastlane.tools/actions/upload_symbols_to_crashlytics/) — fastlane docs. Checked 2026-10-18. [^ascapikeys]: [Creating API Keys for App Store Connect API](https://developer.apple.com/documentation/appstoreconnectapi/creating-api-keys-for-app-store-connect-api) — Apple developer documentation. Checked 2026-10-18. [^uploadbuilds]: [Upload builds](https://developer.apple.com/help/app-store-connect/manage-builds/upload-builds) — App Store Connect Help. Checked 2026-10-18. [^gcpadc]: [How Application Default Credentials works](https://cloud.google.com/docs/authentication/application-default-credentials) — Google Cloud, updated 2026-10-13. Checked 2026-10-18. [^easintro]: [EAS Update introduction](https://docs.expo.dev/eas-update/introduction/) — Expo documentation. Checked 2026-10-18. [^easruntime]: [Runtime versions and updates](https://docs.expo.dev/eas-update/runtime-versions/) — Expo documentation. Checked 2026-10-18. [^crashfree]: [Understand crash-free metrics](https://firebase.google.com/docs/crashlytics/crash-free-metrics) — Firebase Crashlytics, updated 2026-10-16. Checked 2026-10-18. [^crsymios]: [Get readable crash reports in the Crashlytics dashboard (Apple platforms)](https://firebase.google.com/docs/crashlytics/ios/get-deobfuscated-reports) — Firebase, updated 2026-10-16. Checked 2026-10-18. [^crashalerts]: [Configure and receive Crashlytics alerts](https://firebase.google.com/docs/crashlytics/alerts-default) — Firebase, updated 2026-10-16. Checked 2026-10-18. [^reportingapi]: [vitals.crashrate](https://developers.google.com/play/developer/reporting/reference/rest/v1beta1/vitals.crashrate) — Google Play Developer Reporting API (v1beta1). Checked 2026-10-18. [^applecrash]: [Acquiring crash reports and diagnostic logs](https://developer.apple.com/documentation/xcode/acquiring-crash-reports-and-diagnostic-logs) — Apple developer documentation. Checked 2026-10-18. [^metrickit]: [MetricKit](https://developer.apple.com/documentation/metrickit) — Apple developer documentation. Checked 2026-10-18. [^rcloading]: [Firebase Remote Config loading strategies](https://firebase.google.com/docs/remote-config/loading) — Firebase, updated 2026-10-16. Checked 2026-10-18. [^rcstart]: [Get started with Remote Config on Android](https://firebase.google.com/docs/remote-config/android/get-started) — Firebase, updated 2026-10-16. Checked 2026-10-18. [^openfeature]: [Glossary](https://openfeature.dev/specification/glossary) — OpenFeature specification. Checked 2026-10-18. [^aip180]: [AIP-180: Backwards compatibility](https://google.aip.dev/180) — Google API Improvement Proposals, created 2019-07-23. Checked 2026-10-18. [^masvs]: [OWASP MASVS](https://mas.owasp.org/MASVS/) — OWASP Mobile Application Security project. Checked 2026-10-18. [^masvscode]: [MASVS-CODE: Code Quality](https://mas.owasp.org/MASVS/10-MASVS-CODE/) — OWASP. Checked 2026-10-18. [^privacymanifest]: [Privacy manifest files](https://developer.apple.com/documentation/bundleresources/privacy-manifest-files) — Apple developer documentation. Checked 2026-10-18. [^datasafety]: [Provide information for Google Play's Data safety section](https://support.google.com/googleplay/android-developer/answer/10787469) — Play Console Help. Checked 2026-10-18. [^gharunners]: [GitHub-hosted runners reference](https://docs.github.com/en/actions/reference/runners/github-hosted-runners) — GitHub Docs. Checked 2026-10-18. [^ghapricing]: [Actions runner pricing](https://docs.github.com/en/billing/reference/actions-runner-pricing) — GitHub Docs. Checked 2026-10-18. [^slack]: [Handling Flaky Tests at Scale: Auto Detection & Suppression](https://slack.engineering/handling-flaky-tests-at-scale-auto-detection-suppression/) — Arpita Patel, Slack Engineering, 2022-04-05. Checked 2026-10-18. [^fbsdk1373]: [Suddenly FBSDKRestrictiveDataFilterManager.m - Line 71 started to crash (issue 1373)](https://github.com/facebook/facebook-ios-sdk/issues/1373) — facebook/facebook-ios-sdk on GitHub, opened 2020-05-06, with a statement posted 2020-05-07; recurrence in [issue 1427](https://github.com/facebook/facebook-ios-sdk/issues/1427), 2020-07-10. Checked 2026-10-18. [^siliconangle]: [Facebook software change breaks popular iOS apps such as Spotify and TikTok](https://siliconangle.com/2020/05/07/facebook-sdk-change-breaks-popular-ios-apps-spotify-tiktok/) — SiliconANGLE, 2020-05-07. Checked 2026-10-18. **Further reading:** Orosz's article also covers state, deep links, offline and localisation problems this page leaves out. **Changelog:** - 2026-10-18 — first version (research, outline, draft, two executed labs, six diagrams). - 2026-10-18 — r1 revision after the first review round (`reviews/mobileops-r1-superseded-202610191512.md`): MetricKit corrected; approval versus release; block-only-when-replaceable; halts precede the min-users check; credential scopes; symbol upload; App Center, velocity-alert and deployment-target facts; rollout script hardened; Lab 3 written out; three diagrams redrawn; repetition cut; ten sources added. - 2026-10-19 — second-pass r1 revision (`reviews/mobileops-r1.md`; no critical or major findings): Agent-skills date marked as this Atlas's observation, not a Google launch date (M1); target-API row and Day 2 note add the 2026-11-01 extension and the API 35 floor (M2). - 2026-10-19, post-score polish (`reviews/mobileops-polish.md`): boundary and decision diagrams redrawn for label headroom (same nodes, edges and meanings); three repetition trims; no sourced claim changed.