---
title: "BioOps"
slug: bioops
category: industry
term_status: ambiguous
evidence_status: verified
summary: "BioOps is not a standard term. This Atlas uses it for operating computational biology: running bioinformatics pipelines so that every result can be reproduced, traced back to its samples, code, containers and reference data, and delivered at a known cost."
audience: [Bioinformaticians, Research software engineers, HPC and cloud administrators in life sciences, Core-facility and lab leads, Quality and validation staff in regulated biotech, Beginners]
last_verified: 2026-10-23
key_takeaways:
  - "No published definition of \"BioOps\" was found (searches on 2026-10-23). Genomics England names its pipeline-operations team \"Bio-Ops\"; otherwise the name belongs to an environmental-biotech company and resembles \"BIOps\" (BI operations). The practice is established under other names: bioinformatics workflow management and reproducible computational research."
  - "A result is only as reproducible as its weakest pin: pipeline release with engine version, container digests, reference genome bundle, parameters and input checksums all have to be recorded, not just the code."
  - "Sample identity is the failure that hurts most and is caught least. Check that files match their checksums before a run and that genotypes match the expected person after it."
  - "Workflow managers (Nextflow, Snakemake, CWL and WDL engines) give caching, retries and portability, but their resume logic has rules: a process that edits its inputs, or a non-deterministic channel merge, silently breaks it."
  - "Storage, not CPU, often drives cost: one published pipeline rewrite cut its pre-processing work directory by 65% by switching BAM to CRAM, at the price of 34% more memory."
  - "In clinical or GxP work the pipeline is part of a validated test or computerised system: significant changes need supplemental validation, and you need audit trails, not just Git history. This page flags the obligations; it is not legal or regulatory advice."
related: [labops, researchdataops, workflowops, containerops, healthops, mlops, dataops, complianceops]
---

## What BioOps is

### In 30 seconds

**BioOps** is not an established term. This Atlas uses it for **the operational side of computational biology**: running
bioinformatics pipelines (alignment, variant calling, expression quantification) as a dependable service, so
that every result can be **reproduced**, **traced** back to the samples, code, software and reference data that produced it, and
delivered **on time and at a known cost**.

**In one line: BioOps asks "if we ran this analysis again next year, on different hardware, would we get the same answer, and could
we prove which sample, which code and which genome build it came from?"**

The idea is older than the label: the first of Sandve and colleagues' ten rules for reproducible computational research (2013) is
"For Every Result, Keep Track of How It Was Produced".[^sandve] This page turns that rule into an operating model.

**For** the users listed below; [ContainerOps](../containerops/) is the recommended prerequisite.

::: note The label is ambiguous — read this first
- **No published definition of "BioOps" was found** (web searches on 2026-10-23), nor any standard or certification. The closest
  real use is one organisation's team name: Genomics England, whose service users are mostly NHS clinicians, has a
  "Bioinformatics Operations (Bio-Ops) Team" of Operations Bioinformaticians who keep its pipeline working end to end.[^ge-bioops]
- The name is taken by unrelated things: **bioops Srl**, an Italian company listed as providing "Environmental consulting,
  Environmental biotechnology, Bioremediation and more";[^patsnap] **"BIOps"**, used in an AWS blog post for "business intelligence
  and IT operations";[^biops-aws] and **`biops`**, a "bioprocess simulator" Python package for wastewater reactors.[^biops-py]
  "Bio-OPSEC", biological operations security, is a separate concept from US special-operations publishing (JSOU Press,
  2025).[^bioopsec]
- The practices on this page are **established under their own names**: bioinformatics workflow management, reproducible
  computational research, clinical bioinformatics pipeline validation, and computerised-system validation in GxP. Where this page
  draws a boundary that no standard defines, it says **editorial distinction**.
:::

### At a glance

| Item | Summary |
|---|---|
| Term status | **Ambiguous.** No published definition; one organisation names a team "Bio-Ops"; unrelated companies and products share the name. The practice is established under other names. |
| Goal | Analyses that are reproducible, traceable from sample to result, and cost-controlled. |
| Users | Bioinformaticians, research software engineers (RSEs), research-computing and cloud teams, core facilities, QA and validation staff. |
| In scope | Workflow managers; containers; sequencing formats (FASTQ, BAM/CRAM, VCF); reference data; HPC and cloud execution; provenance; sample identity; testing, benchmarking, validation; compute and storage cost. |
| Out of scope | Wet lab and LIMS, data publishing, clinical care, engine semantics, image building, models: see [neighbours](#how-bioops-relates-to-other-ops). |
| Prerequisites | Linux shell, Git, containers ([ContainerOps](../containerops/)); basic molecular biology vocabulary helps. |
| Key outputs | A versioned pipeline with tests; pinned containers; a reference-data bundle with checksums; a run manifest per batch; identity and QC gates; a cost per sample. |

### The problem it solves

Computational biology fails quietly. A script gives a slightly different answer on the cluster; a spreadsheet renames genes; two
FASTQ files are swapped in a samplesheet; `chr1` meets `1`. Nothing crashes; the numbers are just wrong, or nobody can say how they
were made. The evidence:

- **Notebooks do not rerun.** Of 10,388 Jupyter notebooks from biomedical papers whose dependencies installed, 1,203 ran without
  errors and 879 reproduced the original results.[^jupyter]
- **Spreadsheets corrupt data.** Excel with default settings "is known to convert gene names to dates and floating-point numbers";
  a scan of leading genomics journals found that "approximately one-fifth of papers with supplementary Excel gene lists contain
  erroneous gene name conversions".[^ziemann]
- **Platforms differ.** The Nextflow paper reported that three analyses produced different results on different execution
  platforms.[^nf2017]
- **Formats do not carry identity.** AMP/CAP notes that FASTQ, BAM and VCF do not require sample, patient or run information,
  so without a robust mechanism results "may be assigned to the wrong patient and/or wrong sample".[^ampcap]

Before: a folder of scripts and one person's memory. After: a manifest answers "which samples, which code, which genome build, at
what cost" without asking anyone.

### What BioOps is not

- **Not LabOps.** Wet lab, instruments, LIMS and sample custody belong to [LabOps](../labops/). BioOps starts where **data leaves
  the instrument** and consumes LabOps' sample metadata. *Editorial distinction.*
- **Not ResearchDataOps.** Repositories, DOIs, FAIR metadata and preservation belong to [ResearchDataOps](../researchdataops/).
  BioOps ends where **data is published** and hands over its provenance. *Editorial distinction.*
- **Not bioprocess operations.** Running bioreactors and manufacturing processes is process engineering; the `biops` simulator
  shares a name, not a field.[^biops-py]
- **Not DevOps with a lab coat.** Version control, CI and staged releases transfer and are documented for bioinformatics
  pipelines,[^biogitflow] but correctness here is statistical, inputs are terabytes, and clinical or GxP changes are regulated.
  See [DevOps](../devops/) for the general practice.
- **Not a product.** Managed services can run pipelines; they do not decide what to pin, validate or sign off.

## Where it comes from and where it sits

### Origins and evolution

There is no BioOps movement; the practices grew inside bioinformatics. Shell scripts on shared clusters gave way to **workflow
managers**: Snakemake, and Nextflow, whose 2017 paper showed "three bioinformatics data analyses produce different results when
executed on different execution platforms" and argued that a workflow manager plus containers could control this.[^nf2017]
Engine-independent languages followed: CWL, an open standard,[^cwl] and the openly governed WDL.[^wdl] **Packaged software** made pins practical: Bioconda reported
"over 3,000 Conda software packages" in 2018,[^bioconda] and BioContainers packages tools as containers.[^biocontainers]
**Community pipelines** followed with nf-core (started 2018,[^nfcore-about] described in a 2020 paper[^nfcore-paper]); on 2026-10-23 its `pipelines.json` listed 159 pipeline
repositories, 106 of them released and not archived (our count).[^nfcore-json] **Standard APIs and provenance** came from GA4GH
(WES, TES) and Workflow Run RO-Crate,[^ga4gh-wes][^wrroc] and **regulated use** from the AMP/CAP validation guideline (2018).[^ampcap]

### Glossary and mental model

| Term | Meaning | Why it matters |
|---|---|---|
| FASTQ | Text format for reads plus per-base quality scores | No formal specification; encoding errors are silent. |
| BAM / CRAM | Binary alignment formats; CRAM compresses against a reference sequence | A CRAM is unreadable without the exact reference.[^cram] |
| VCF / BCF | Variant Call Format (text) and binary form; VCFv4.5 finalised 2024, document last revised 25 February 2026[^vcf45] | The usual hand-off to interpretation and publication. |
| Reference bundle | Genome FASTA, indexes, annotation (GTF), known-sites and interval files for one genome build | Mixing builds or contig naming styles breaks results silently. |
| Samplesheet | Table mapping sample IDs to input files and metadata | One wrong row is a sample swap. |
| Executor | The backend that runs tasks: local, Slurm, AWS or Google Batch, Kubernetes | Change the executor, not the pipeline, to move between HPC and cloud.[^nf-executor] |
| Work directory | Per-task scratch area holding staged inputs and intermediate outputs | Often the largest storage cost; needed for resume.[^nf-cache] |
| Container digest | Content hash of an image (`sha256:…`) | Tags move; digests do not. |
| Truth set | Benchmark calls and regions for a reference sample, such as Genome in a Bottle (GIAB) HG002 | Lets you measure accuracy, not just completion.[^giab] |

**Mental model: five pins and two gates.** A result is reproducible when five things are pinned: **pipeline release** (code
revision plus engine version), **software environment** (container digests), **reference bundle**, **parameters** (including any
random seeds, Sandve's Rule 6[^sandve]), and **inputs** (samplesheet plus file checksums). It is
trustworthy when it passes two gates: an **input gate** before the run (files exist, checksums match, samplesheet is consistent)
and an **identity and quality gate** after it (genotype or sex checks, QC thresholds, benchmark accuracy where applicable).

### How BioOps relates to other Ops

::: figure The research and life-science data chain: where BioOps starts and ends
```mermaid
flowchart TB
  L["LabOps: samples, LIMS"]
  subgraph Core["BioOps"]
    B1["Input gate"]
    B2["Pipeline run"]
    B3["Identity, QC, provenance"]
    B1 --> B2 --> B3
  end
  S["WorkflowOps, ContainerOps"]
  R["ResearchDataOps: publish"]
  H["HealthOps: report"]
  L -->|"data leaves instrument"| B1
  S -.->|"support"| B2
  B3 -->|"data is published"| R
  B3 -->|"clinical result"| H
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  class B1,B2,B3 core
  class L,R,H,S neutral
```
Text equivalent: LabOps (samples, LIMS) hands data to BioOps where data leaves the instrument. Inside BioOps, an input gate
(samplesheet and checksums) comes first, then the pipeline run on HPC or cloud, then identity, QC and provenance checks.
WorkflowOps (engine semantics) and ContainerOps (image building) support the pipeline run. BioOps hands results to ResearchDataOps
when data is published, and to HealthOps for clinical reporting.
:::

| Neighbour | Shared scope | This page's focus | Choose the neighbour when |
|---|---|---|---|
| [LabOps](../labops/) | Sample metadata, instrument output | Using sample metadata to keep identity through the pipeline | You manage samples, instruments, LIMS or wet-lab QC |
| [ResearchDataOps](../researchdataops/) | Provenance, checksums, data formats | Producing provenance during analysis | You deposit, cite, license or preserve data |
| [WorkflowOps](../workflowops/) | Retries, resume, DAGs | Bioinformatics-specific use of engines | You need engine semantics for business processes or durable execution |
| [ContainerOps](../containerops/) | Images, registries, digests | Choosing and pinning images per tool, HPC runtimes | You build, scan or run images in general |
| [HealthOps](../healthops/) | Clinical genomics | The pipeline and its validation | You run clinical services, EHR integration, patient flow |
| [ComplianceOps](../complianceops/) | GxP, audit evidence | What the pipeline must produce as evidence | You run the compliance programme itself |

Pipelines that feed business analytics belong to [DataOps](../dataops/); model training on omics data to [MLOps](../mlops/);
organisation-wide cloud cost to [FinOps](../finops/). [ResearchOps](../researchops/) is UX research and unrelated.

## Principles and operating model

### A proposed charter and principles

No published BioOps charter was found. This is **this Atlas's proposal**, built on the ten rules[^sandve] and on the Snakemake authors'
argument that reproducibility "is by no means sufficient" without "automation, scalability, portability, readability, traceability,
and documentation".[^smk-paper]

**Mission.** Every analysis result can be regenerated, explained and attributed to the right sample, at a cost the group can predict.

**Principles (proposed):**

1. **Pin all five.** Pipeline release (with engine version), container digests, reference bundle, parameters and input checksums
   are recorded for every run. A tag such as `latest` is not a pin.
2. **Identity is a first-class check.** Sample identity is verified by data (genotype, sex, expected variants), not only by file names.
3. **No manual steps between raw data and result.** Sandve's Rule 2 is "Avoid Manual Data Manipulation Steps".[^sandve] If a human
   must decide, record the decision as an input.
4. **Tests before trust.** Pipelines have automated tests on small data and are benchmarked on truth sets where they exist.
5. **Change is versioned and, where required, validated.** Every release has notes; in regulated work, significant changes trigger
   supplemental validation.[^ampcap]
6. **Storage is designed, not accumulated.** Every retained file has a reason and an owner; intermediates have a deletion date.
7. **Provenance travels with the data** to whoever publishes or reports it.

**Non-goals.** BioOps does not choose the scientific method, interpret variants clinically, or replace the laboratory's quality system.

### Building blocks

| Block | Typical implementation |
|---|---|
| Pipeline repository (code, configs, tests) | Git; nf-core or Snakemake workflow template |
| Software environments | BioContainers or Bioconda-built images, pinned by digest; Apptainer on HPC |
| Reference bundles | Read-only path or bucket prefix with a checksum manifest |
| Input gate | Samplesheet schema and checksum checks ([Lab 2](#lab-2-intermediate-an-input-gate-and-run-manifest)) |
| Provenance store | Trace and report files, `nextflow log`, lineage, Workflow Run RO-Crate |
| QC and identity gate | MultiQC, sex and genotype concordance, contamination estimate, thresholds |

### Roles, ownership and RACI

Small groups combine roles, but each decision has one accountable person.

| Activity | Pipeline owner (bioinformatician/RSE) | Platform admin (HPC/cloud) | Lab or data provider | Scientific lead / lab director | QA / validation |
|---|---|---|---|---|---|
| Pipeline code, tests, releases | **A/R** | C | I | C | C (regulated) |
| Containers and reference bundles | **A/R** | C | — | I | C |
| Compute, storage, quotas, scheduler | C | **A/R** | — | I | I |
| Samplesheet correctness | C | — | **R** | **A** | I |
| Input and identity gates | **A/R** | — | C | I | C |
| Release to production (research) | **R** | I | I | **A** | — |
| Release to production (clinical/GxP) | R | I | I | C | **A** (with the lab director where required) |
| Cost per sample and retention | R | R | I | **A** | — |
| Incident handling (wrong result) | **R** | C | C | **A** | **C/A** in regulated work |

In clinical laboratories, AMP/CAP expects a qualified medical professional to oversee validation and the laboratory director to
approve each pipeline component.[^ampcap]

### Lifecycle

A pipeline's lifecycle is a software release cycle plus validation against biological truth.

::: figure Pipeline lifecycle from development to production, with the change loop
```mermaid
flowchart TB
  D["Develop on a branch"] --> T["Test: unit and small data"]
  T --> V["Validate: truth sets, real samples"]
  V --> R["Release: version and notes"]
  R --> P["Run in production"]
  P -->|"bug, new tool, new build"| C["Change request"]
  C -->|"back to development"| D
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  class D,T,R,P core
  class V,C decision
```
Text equivalent: develop on a branch, test with small data, validate against truth sets and real samples, release with a
version and notes, run in production. A bug, new tool version or new genome build raises a change request, which returns to
development. Validation and change requests are the decision points.
:::

- **Day 0 — design:** intended use (research, service, clinical test, GxP), data types, volumes, genome builds, acceptance criteria.
- **Day 1 — first release:** tests, pinned images, frozen reference bundle, benchmark, manifest format, a first batch with extra review.
- **Day 2 — operate:** gates, triage, cost review, releases on a cadence. The biogitflow protocol uses development, "validation for
  pre-production" and production environments, with a hotfix path.[^biogitflow]

### Operating model and governance

- **Intake.** Batches arrive with a samplesheet and checksums; gate failures go back to the provider, not into the pipeline.
- **Release cadence.** Service and clinical groups batch changes into planned releases because each costs validation effort.
  Nextflow ships stable releases in April and October and monthly edge releases,[^nf-update] so plan engine upgrades.
- **Change classes (proposed).** *Patch*: no change to outputs (documentation, logging) — tests only. *Minor*: outputs may change
  within tolerance (tool patch, resource settings) — tests plus benchmark comparison. *Major*: method change (new caller, new genome
  build) — full revalidation, and in clinical work the supplemental validation the guideline requires "whenever a significant change
  is made".[^ampcap]
- **Review rhythm.** Weekly: failed runs and gate failures. Monthly: cost per sample, storage growth, open defects. Per release:
  benchmark deltas and sign-off.
- **Escalation.** Any suspected wrong-sample or wrong-result event stops release of affected results and follows the
  [sample-swap runbook](#worked-runbook-a-suspected-sample-swap).
- **Live operations.** Running cases need a named owner. Genomics England's Bio-Ops team, for example, is "the first port of
  call for issues affecting live cases", gives second-line support and maintains pipeline-monitoring tools.[^ge-bioops]

## Architecture and workflow

### Reference architecture

The architecture separates **what** is computed (pipeline code and pinned environments), **where** it runs (executors), and **what
is remembered** (provenance and retained data). That separation is what lets the same release run on a laptop for tests, on a Slurm
cluster for routine batches, and in a cloud batch service for bursts.

::: figure Reference architecture for operating bioinformatics pipelines
```mermaid
flowchart TB
  subgraph Inputs["Inputs, read-only"]
    I1["Raw reads"]
    I2["Samplesheet, checksums"]
    I3["Reference bundle"]
  end
  subgraph Control["Control"]
    G["Input gate"]
    E["Pinned engine"]
    Reg["Images by digest"]
  end
  subgraph Exec["Execution"]
    X1["HPC: Slurm, Apptainer"]
    X2["Cloud batch, spot"]
  end
  subgraph Keep["Keep"]
    WD["Work dir: expires"]
    O["Results"]
    P["Provenance"]
  end
  I1 --> G
  I2 --> G
  G --> E
  I3 --> E
  Reg --> E
  E --> X1
  E --> X2
  X1 --> WD
  X2 --> WD
  WD --> O
  G --> P
  E --> P
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  class G,E,P core
  class I1,I2,I3,Reg,X1,X2,O neutral
  class WD risk
```
Text equivalent: three read-only inputs (raw reads, the samplesheet with checksums, a versioned reference bundle) feed the
control layer. Reads and samplesheet pass an input gate before the engine, which runs at a pinned version and pulls images by
digest. The engine sends tasks to HPC (Slurm with Apptainer) or to cloud batch on spot capacity. Tasks write to an expiring work
directory, marked as a cost risk, from which final results (VCF, CRAM, QC) are published. The input gate starts the provenance record (run manifest) and the engine
adds trace and lineage.
:::

Design choices and their trade-offs:

| Decision | Option A | Option B | What decides it |
|---|---|---|---|
| Where to run | On-premises HPC | Cloud batch | Data location and egress, queue wait, burst needs, data-access rules, existing allocation |
| Container runtime | Docker or Podman | Apptainer on HPC | Shared clusters rarely allow a root daemon; Apptainer runs as "the same user inside a container as outside"[^apptainer] |
| Environment source | BioContainers/Bioconda images | Images you build | Audit needs, patching duties, unusual tools; building means owning updates ([ContainerOps](../containerops/)) |
| Intermediate format | BAM | CRAM | Storage cost versus memory and CPU, and the risk of losing the exact reference ([use case 1](#case-1-nf-coresarek-3-storage-versus-memory-sourced)) |
| Engine | Nextflow | Snakemake, or CWL/WDL engines | Team language, community pipelines available, portability needs ([deep comparison](#workflow-managers-compared)) |
| Managed platform | Self-run engine | Managed service | Operations effort versus lock-in; managed features can close (see [cost](#cost-capacity-and-sustainability)) |

### End-to-end workflow

A routine batch, from delivery to release, with the two exception paths (reject, hold).

::: figure One batch from data delivery to released results
```mermaid
sequenceDiagram
  participant Lab as Lab or provider
  participant Own as Pipeline owner
  participant Eng as Workflow engine
  participant Cmp as Compute
  Lab->>Own: reads, sheet, checksums
  Own->>Own: input gate
  Own-->>Lab: reject if checks fail
  Own->>Eng: start pinned release
  Eng->>Cmp: submit tasks
  Cmp-->>Eng: exit codes, outputs
  Eng->>Eng: retry or resume
  Eng->>Own: QC, identity, manifest
  Own-->>Lab: hold if identity fails
  Own->>Own: release results
```
Text equivalent: the lab delivers reads, samplesheet and checksums to the pipeline owner, who runs the input gate and rejects the
batch to the lab if it fails. Otherwise the owner starts the pinned release; the engine submits tasks to compute, receives exit
codes and outputs, and retries or resumes failed tasks. The engine returns QC, identity checks and the manifest; if identity
fails the owner holds the results and queries the lab, otherwise the owner (or QA in regulated work) releases them.
:::

After release, results and manifest go to publication ([ResearchDataOps](../researchdataops/)) or reporting, and the work
directory expires.

## Practices, standards and tools

### Core practices and playbooks

| Practice | Goal | Verification |
|---|---|---|
| Pin engine, release and images | Same code and software bytes | `NXF_VER` or strict `manifest.nextflowVersion`; `@sha256:` digests in the manifest |
| Freeze reference bundles | Same genome, indexes, annotation | Bundle ID + checksum file; contig names checked |
| Gate inputs | Stop bad data before compute | Checksum and samplesheet checks ([Lab 2](#lab-2-intermediate-an-input-gate-and-run-manifest)) |
| Check identity | Detect swaps and duplicates | Genotype concordance, sex check, controls |
| Estimate contamination | Detect mixed samples | A contamination estimator such as verifyBamID2[^verifybamid] |
| Test and benchmark | Catch code and scientific regressions | CI tests per change; truth-set metrics per release |
| Design retention | Control storage cost | Work directories expire; retention register |

#### Playbook: pinning a pipeline release

1. **Engine.** Require a minimum version (`manifest.nextflowVersion = '!>=25.10.4'`, strict) and pin the exact one per run with
   `NXF_VER`.[^nf-update]
2. **Code.** Run a tagged release (`-r 1.2.0`), never a branch; record the commit ID.
3. **Software.** Resolve each image tag to a digest at release time and write the digest into the config (on 2026-10-23 Quay
   listed `samtools:1.24--h9dcdb79_1` as `sha256:a1304475…`).[^quay-samtools] Tags can be rebuilt; digests cannot change content.
4. **Reference.** Store the bundle read-only, with a checksum file, and a bundle ID such as `GRCh38-noalt-gencode49-v3`.
5. **Parameters.** Keep them in a versioned params file; nf-core asks for parameters via the command line or `-params-file`, not
   config files.[^nfcore-demo]
6. **Record.** Write all five pins (engine version with the release) into the run manifest.

#### Playbook: making resume work, and knowing when it will not

Nextflow reuses a cached task on `-resume` when its hash (inputs, script, container and more) matches and its outputs are still in
the work directory; an input file is hashed by "full path, last modified timestamp, and file size".[^nf-cache] Consequences:

- **Keep both halves.** Resume needs the cache (`.nextflow/cache` in the launch directory) **and** the work directory; on cloud VMs
  use the documented cloud cache in object storage.[^nf-cache]
- **Shared file systems can lie about time.** NFS "may report inconsistent file timestamps"; the `lenient` cache mode ignores the
  timestamp.[^nf-cache]
- **Code can make resume wrong, not just slow.** "If a process modifies its input files, it cannot be resumed", and a process that
  receives a BAM channel and an index channel merged without matching on ID gets "incorrect and non-deterministic" inputs; the fix
  is to `join` on the sample ID.[^nf-cache] That is a correctness bug: a BAM paired with another sample's index.

Snakemake decides re-runs with `--rerun-triggers`; by default code, input, modification time, parameters and software environment
all trigger a re-run, which "guarantees that results are consistent with the workflow code and configuration".[^smk-cli]

#### Playbook: retries that help instead of hiding faults

Retry only failures that a retry can fix. Nextflow's documented pattern raises memory and time on each attempt and retries only
exit codes 137–140, terminating on anything else.[^nf-process] Codes above 128 usually mean a signal kill (128 + signal
number[^bash-exit]): 137 is SIGKILL, for example an out-of-memory kill or a scheduler limit; 139 is a segmentation fault, which
more memory rarely fixes:

```groovy
process ALIGN {
    memory { 8.GB * task.attempt }
    time   { 4.h * task.attempt }
    errorStrategy { task.exitStatus in 137..140 ? 'retry' : 'terminate' }
    maxRetries 2
    // ...
}
```

Blanket retries turn a corrupt input into three failed attempts and a bigger bill. Spot reclaims on AWS Batch are separate:
`aws.batch.maxSpotAttempts` controls them (default 0 since Nextflow 24.10).[^nf-aws] AWS gives two minutes' notice,[^aws-spot] so
short, resumable tasks matter more than clever retry rules.

### Formats and reference data you must operate

| Format | Defined by | Operational gotcha | Check |
|---|---|---|---|
| FASTQ | No formal spec: "several incompatible variants"[^hts-specs][^fastq] | Old quality encodings; truncated gzip files; R1/R2 mismatched | Checksums from provider; `gzip -t`; read counts R1 = R2 |
| SAM/BAM | SAMv1 specification (GA4GH hts-specs)[^hts-specs] | Header `@SQ` contigs must match the reference; unsorted or unindexed files | `samtools quickcheck`; compare `@SQ` with the bundle |
| CRAM | CRAMv3 specification[^cram] | Needs the exact reference; readers check reference MD5s | Keep reference by MD5; test decode after archive |
| VCF/BCF | VCFv4.5 (finalised 2024)[^vcf45] | Representation differences (normalisation, multi-allelics) break comparisons | Normalise before comparing; validate headers |
| Reference bundle | Assembly provider + your manifest | `chr1` (UCSC style) vs `1` (Ensembl style): "Most tools would view those as different"[^galaxy-chr] | Contig-name check in the input gate |

### Methods, standards and frameworks

Separate what is **mandatory** in your context from what is **recommended**:

| Framework | Type | What it asks of a pipeline | Applies to |
|---|---|---|---|
| Ten simple rules (Sandve et al.)[^sandve] | Recommendation | Track how results were produced; avoid manual steps; archive exact program versions; record random seeds; version scripts | All research computing |
| FAIR principles for research software (FAIR4RS)[^fair4rs] | Recommendation | Treat "computational workflows" as research software: identifiers, metadata, licences | Pipelines you publish |
| GA4GH WES and TES[^ga4gh-wes][^ga4gh-tes] | Standard APIs | Submit workflows and tasks to compliant backends the same way | Federated or multi-cloud execution |
| AMP/CAP pipeline validation (2018)[^ampcap] | Professional guideline (US clinical labs) | 17 recommendations: own validation, identity with four identifiers, file integrity, revalidation on significant change | Clinical NGS testing |
| GA4GH benchmarking best practices[^krusche] | Community method | Match variant representations, standard metrics, stratify by type and region | Variant-calling accuracy claims |

Regulations for GxP computing (21 CFR Part 11, EU GMP Annex 11) and GAMP 5 are covered in
[quality, security, privacy and compliance](#quality-security-privacy-and-compliance).

### Tooling landscape

Named tools are examples verified to exist on 2026-10-23, not endorsements.

| Category | Examples | Choose by |
|---|---|---|
| Workflow managers | Nextflow (26.10.3, 2026-10-20),[^nf-releases] Snakemake (9.27.1, 2026-10-21),[^smk-releases] Cromwell or miniwdl (WDL), cwltool or Toil (CWL) | Team language, community pipelines, executors |
| Community pipelines | nf-core (for example sarek 3.10.0, rnaseq 3.27.0),[^nfcore-json] Snakemake workflow catalogue | Release cadence, tests, fit to your assay |
| Software packaging | Bioconda, BioContainers, Apptainer images | Your tools available; digest pinning |
| Testing | nf-test (snapshots, process and pipeline tests),[^nftest] pytest-workflow | Fits the engine and CI |
| Identity and QC | NGSCheckMate,[^ngscheckmate] Picard CrosscheckFingerprints, verifyBamID2 (contamination), MultiQC, FastQC | Data types supported |
| Benchmarking | hap.py, RTG vcfeval, GIAB truth sets[^giab] | Variant types, stratifications |
| Provenance | Nextflow trace, report and lineage; Snakemake reports; Workflow Run RO-Crate | Completeness, queryability |
| Execution | Slurm; AWS, Google or Azure Batch; Kubernetes; managed services (AWS HealthOmics, Seqera Platform, Terra, DNAnexus) | Data residency, cost model, quotas |
| Data transfer | Globus (verifies checksums by default),[^globus] cloud CLIs | Integrity checks, throughput |

### Automation and AI

Automate the deterministic parts fully: gates, pinning, tests, benchmark comparisons, clean-up, cost reports. Keep **humans on
release decisions** for clinical and GxP results and on changes beyond agreed tolerances. Code drafted by language models must pass
the same tests, and models should not choose parameters, genome builds or QC thresholds unreviewed.
For GMP, the EU consulted on a draft **Annex 22 on artificial intelligence** until 7 October 2025.[^annex11-draft]

## BioOps in practice

### Reference implementation

**Assumptions.** A small bioinformatics group runs germline short-read pipelines for a few thousand research samples a year on a
Slurm cluster with Apptainer, bursting to cloud batch. Nextflow is the engine (Snakemake works the same way). Versions as of
October 2026.

**Repository layout.**

```text
germline-pipeline/
  main.nf, modules/, subworkflows/   # pipeline code (or nf-core pipeline + config)
  conf/site.config                   # executors, pinned images, trace, lineage
  conf/refs/GRCh38-noalt-v3.json     # reference bundle manifest (paths + md5)
  params/batch-template.yaml         # parameters, versioned
  tests/                             # nf-test tests and snapshots
  benchmarks/                        # truth-set comparison scripts and past results
  docs/RELEASE_NOTES.md, docs/validation/
  .github/workflows/test.yml         # CI
```

**Site configuration.** One config file holds everything that varies by site, so the pipeline code stays portable.

_Example status: written against the Nextflow 26.10 configuration reference; not executed (Nextflow is not installed on the machine
that built this page). Queue names, paths and the region are placeholders._

```groovy
// conf/site.config
manifest {
    nextflowVersion = '!>=25.10.4'          // refuse older engines
}
process {
    // Retries help only if memory/time scale with task.attempt (nf-core base configs do)
    errorStrategy = { task.exitStatus in 137..140 ? 'retry' : 'terminate' }
    maxRetries    = 2
    withName: 'SAMTOOLS_.*' {
        // samtools:1.24--h9dcdb79_1; digest only (some runtimes reject tag+digest)
        container = 'quay.io/biocontainers/samtools' +
            '@sha256:a130447589651ed09252aa95a5e4f4132942cdb54d835d81a04a9a930d656561'
    }
}
profiles {
    hpc {
        process.executor   = 'slurm'
        process.queue      = 'standard'          // placeholder
        apptainer.enabled  = true
        apptainer.cacheDir = '/shared/apptainer-cache'
    }
    cloud {
        // Needs a Batch compute environment + job queue; every process needs a container
        process.executor = 'awsbatch'
        process.queue    = 'genomics-spot'       // placeholder
        workDir          = 's3://<bucket>/work'  // must be S3, with a subdirectory
        aws.region       = 'eu-west-1'           // placeholder
        aws.batch.maxSpotAttempts = 2            // default is 0 since 24.10
    }
}
trace {
    enabled = true                               // default name trace-<timestamp>.txt
    fields  = 'task_id,hash,name,status,exit,attempt,' +
              'cpus,%cpu,memory,peak_rss,realtime,container'
}
report.enabled   = true
timeline.enabled = true
lineage.enabled  = true                          // experimental since 25.04
```

The digest is the pin. On AWS Batch, Nextflow's setup guide needs a compute environment, a job queue bound to it, an S3 work
directory and a container for every process, and its setup attaches `AmazonS3FullAccess` to the instance role;[^nf-awsbatch]
scope that role to the batch's input, work and output prefixes instead (see [security](#quality-security-privacy-and-compliance)).
The trace fields add `cpus`, `memory`, `attempt` and `container` so efficiency and software can be computed
per task; `%cpu` is "relative to a single CPU" (four busy CPUs report about 400%).[^nf-trace] Lineage records the run (repository,
commit, parameters, resolved config), tasks and output checksums under `lid://` identifiers, but "is an experimental feature added in
Nextflow 25.04",[^nf-lineage] so the run manifest below stays the durable record.

**Parameters file.**

_Example status: YAML syntax-checked; parameter names other than `input` and `outdir` are illustrative._

```yaml
# params/batch-2026-10-B07.yaml
input: /data/batches/2026-10-B07/samplesheet.csv
outdir: /results/2026-10-B07
genome_bundle: GRCh38-noalt-v3        # resolved via conf/refs/GRCh38-noalt-v3.json
batch_id: 2026-10-B07
```

**Continuous integration.** Every change runs the pipeline's tests with the pinned engine.

_Example status: YAML syntax-checked; not executed. Action versions were the latest releases on GitHub on 2026-10-23; pin them to
commit SHAs in production._

```yaml
name: pipeline-tests
on:
  pull_request:
  push:
    branches: [main]
permissions:
  contents: read
jobs:
  nf-test:
    runs-on: ubuntu-24.04
    steps:
      - uses: actions/checkout@v7
      - uses: nf-core/setup-nextflow@v3
        with:
          version: "26.10.3"
      - uses: nf-core/setup-nf-test@v2
        with:
          version: "0.9.5"
      - name: Run tests with the test profile
        run: nf-test test --profile test,docker --verbose
```

**Run manifest.** Written by the input gate ([Lab 2](#lab-2-intermediate-an-input-gate-and-run-manifest)) and completed after the run.

_Example status: JSON syntax-checked; values illustrative._

```json
{
  "batch_id": "2026-10-B07",
  "pipeline": "germline-pipeline",
  "revision": "2.3.1",
  "commit": "<40-character commit id>",
  "engine": "nextflow 26.10.3",
  "reference_bundle": "GRCh38-noalt-v3",
  "params_file_sha256": "<sha256>",
  "samplesheet_sha256": "<sha256>",
  "images": {
    "samtools": "sha256:a130447589651ed09252aa95a5e4f4132942cdb54d835d81a04a9a930d656561"
  },
  "data_access_approval": "<consent or data-access approval id>",
  "identity_check": "passed",
  "released_by": "<role mailbox>",
  "released_at_utc": "2026-10-23T14:00:00Z"
}
```

**The same idea in Snakemake.** Snakemake rules can declare a `container:` or `conda:` environment, and the run chooses how to deploy
them with `--software-deployment-method` (`--sdm`), for example `--sdm apptainer` on a cluster.[^smk-deploy]

_Example status: illustrative Snakefile; not executed and not machine-checked._

```text
rule flagstat:
    input:  "results/{sample}.cram"
    output: "results/{sample}.flagstat.txt"
    # samtools:1.24--h9dcdb79_1, pinned by digest
    container: ("docker://quay.io/biocontainers/samtools"
        "@sha256:a130447589651ed09252aa95a5e4f4132942cdb54d835d81a04a9a930d656561")
    shell:  "samtools flagstat {input} > {output}"
```

### Hands-on labs

Three labs, from running a community pipeline to measuring accuracy. Lab 2, the launcher download and checksum lines of Lab 1 and
the download steps of Lab 3 were executed for this page; the rest is written from the current documentation and says so.

#### Lab 1 (beginner): run, resume and read the trace of a small pipeline

**Objective:** see caching, resume and per-task metrics on a real nf-core pipeline.

**Prerequisites:** Linux or macOS; Bash 3.2+ and Java 17–26;[^nf-install] Docker; about 2 GB free disk; internet access. nf-core/demo
1.2.0 requires Nextflow 25.10.4 or later and runs FastQC, seqtk trimming and MultiQC on small test data.[^nfcore-demo]
**Cost and risk:** none beyond downloads; runs locally.

_Status: not executed for this page (Java and Docker are not available on the machine that built it), except the launcher download
and checksum lines, run on 2026-10-23. Commands follow the Nextflow and nf-core documentation as of October 2026._

```bash
mkdir -p ~/bioops-lab1 && cd ~/bioops-lab1
# pinned launcher (not `curl ... | bash`); SHA-256 from the GitHub release
curl -fsSLo nextflow \
  https://github.com/nextflow-io/nextflow/releases/download/v26.10.3/nextflow
SHA=5f9b4069c4eb62104442e0a64d01a7b6557146008c6b180bf78a995605540896
# macOS has no sha256sum: use `shasum -a 256 -c -` instead
echo "$SHA  nextflow" | sha256sum -c - && chmod +x nextflow  # "nextflow: OK"
export NXF_VER=26.10.3                          # pin the engine for this shell
./nextflow run nf-core/demo -r 1.2.0 -profile test,docker --outdir results
./nextflow run nf-core/demo -r 1.2.0 -profile test,docker --outdir results -resume
./nextflow log                                  # list runs, session IDs, status
./nextflow log last -f name,hash,exit,attempt,realtime,%cpu,peak_rss
ls results/pipeline_info/
```

**Expected output:** the first run completes with status `OK` in `nextflow log`; `results/` contains FastQC and MultiQC output and
`results/pipeline_info/` holds `execution_trace_*.txt`, an execution report, a timeline and a DAG.[^nfcore-demo] The second run reuses
every task from the cache and finishes much faster.[^nf-cache]

**Validation:** the second run reports every process as cached; every task shows `exit` 0. Compare `%cpu` with requested CPUs.

**Troubleshooting:** `nextflow: FAILED` from the checksum → `&&` skips `chmod`, so the launcher cannot run; delete it and download
again. `sha256sum: command not found` on macOS → `shasum -a 256 -c -`. Java errors → install Java 17+. Docker permission errors → `-profile test,apptainer`. Resume re-runs
everything → see [making resume work](#playbook-making-resume-work-and-knowing-when-it-will-not).

**Teardown:** `rm -rf ~/bioops-lab1` (also removes `work/`, `.nextflow/` and results). Images stay in Docker's cache:
`docker image ls` and remove them if wanted.

#### Lab 2 (intermediate): an input gate and run manifest

**Objective:** stop a batch with a corrupted file or a reused FASTQ before any compute runs, and write the run manifest.

**Prerequisites:** Python 3.8+ (standard library only), `gzip`, `md5sum` (GNU coreutils). **Cost and risk:** none; synthetic data.

_Status: executed on 2026-10-23 with Python 3.8.10 on synthetic files (version 1.1.0 after review); output below is abridged
from that run._

The gate allows a repeated sample ID, because nf-core samplesheets repeat it on purpose for each lane or re-sequencing run;[^nfcore-rnaseq]
what it refuses is a file used on two rows, which is how a copy-paste error usually shows.

Save as `prov_check.py`:

```python
"""Pre-run gate: check a samplesheet's FASTQ files and write a run manifest.

Usage: python3 prov_check.py samplesheet.csv --pipeline NAME --revision REV
Samplesheet columns: sample,fastq_1,fastq_2,md5_1,md5_2 (fastq_2/md5_2 may be empty).
Exit code 0 = all checks passed; 2 = at least one problem (nothing should run).
"""
import argparse
import csv
import datetime
import hashlib
import json
import os
import sys

CHECKER_VERSION = "1.1.0"


def digest(path, algo):
    h = hashlib.new(algo)
    with open(path, "rb") as fh:
        for block in iter(lambda: fh.read(1 << 20), b""):
            h.update(block)
    return h.hexdigest()


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("samplesheet")
    ap.add_argument("--pipeline", required=True)
    ap.add_argument("--revision", required=True)
    ap.add_argument("--out", default="run-manifest.json")
    args = ap.parse_args()

    problems, files, seen_samples, seen_paths = [], [], set(), {}
    # A sample ID may repeat (one row per lane or re-sequencing run);
    # a file may not: each path is allowed on one row only.
    with open(args.samplesheet, newline="") as fh:
        rows = list(csv.DictReader(fh))
    for n, row in enumerate(rows, start=2):  # line 1 is the header
        sample = (row.get("sample") or "").strip()
        if not sample:
            problems.append(f"line {n}: empty sample id")
            continue
        seen_samples.add(sample)
        pairs = [("fastq_1", "md5_1"), ("fastq_2", "md5_2")]
        if row.get("fastq_1") and row.get("fastq_1") == row.get("fastq_2"):
            problems.append(f"line {n}: {sample} uses the same file for R1 and R2")
        for col, md5col in pairs:
            path = (row.get(col) or "").strip()
            if not path:
                continue
            real = os.path.realpath(path)
            prev = seen_paths.setdefault(real, (n, sample))
            if prev[0] != n:
                problems.append(
                    f"line {n}: {path} already used on line {prev[0]} ({prev[1]})")
            if not os.path.isfile(path):
                problems.append(f"line {n}: missing file {path}")
                continue
            got = digest(path, "md5")
            want = (row.get(md5col) or "").strip().lower()
            if want and got != want:
                problems.append(f"line {n}: md5 mismatch {path} ({got} != {want})")
            if not want:
                problems.append(f"line {n}: no expected md5 for {path}")
            files.append({"sample": sample, "path": path, "md5": got,
                          "bytes": os.path.getsize(path)})

    manifest = {
        "checker_version": CHECKER_VERSION,
        "checked_at_utc": datetime.datetime.now(datetime.timezone.utc)
        .strftime("%Y-%m-%dT%H:%M:%SZ"),
        "pipeline": args.pipeline,
        "revision": args.revision,
        "samplesheet_sha256": digest(args.samplesheet, "sha256"),
        "samples": len(seen_samples),
        "files": files,
        "problems": problems,
    }
    with open(args.out, "w") as fh:
        json.dump(manifest, fh, indent=2)
    for p in problems:
        print("PROBLEM", p)
    print(f"{len(seen_samples)} samples, {len(files)} files, "
          f"{len(problems)} problems -> {args.out}")
    return 2 if problems else 0


if __name__ == "__main__":
    sys.exit(main())
```

Steps:

```bash
mkdir -p ~/bioops-lab2/fastq && cd ~/bioops-lab2      # save prov_check.py here
for s in S1 S2; do for r in 1 2; do
  printf "@r1\nACGT%s%s\n+\nIIIIII\n" "$s" "$r" | gzip -n > "fastq/${s}_R${r}.fastq.gz"
done; done
m() { md5sum "$1" | cut -d' ' -f1; }
{ echo "sample,fastq_1,fastq_2,md5_1,md5_2"
  for s in S1 S2; do
    r1="fastq/${s}_R1.fastq.gz"; r2="fastq/${s}_R2.fastq.gz"
    echo "$s,$r1,$r2,$(m "$r1"),$(m "$r2")"
  done; } > samplesheet.csv
python3 prov_check.py samplesheet.csv --pipeline nf-core/demo --revision 1.2.0
echo "exit=$?"
# now break it: a copy-paste row that reuses S1's file, and a changed file
cp samplesheet.csv bad.csv
echo "S3,fastq/S1_R1.fastq.gz,,$(m fastq/S1_R1.fastq.gz)," >> bad.csv
printf "@r1\nACGTXX\n+\nIIIIII\n" | gzip -n > fastq/S2_R2.fastq.gz
python3 prov_check.py bad.csv --pipeline nf-core/demo --revision 1.2.0 \
  --out bad-manifest.json; echo "exit=$?"
```

**Expected output** (abridged from the executed run; hashes depend on your files):

```text
2 samples, 4 files, 0 problems -> run-manifest.json
exit=0
PROBLEM line 3: md5 mismatch fastq/S2_R2.fastq.gz (b762e80b… != 530cc53c…)
PROBLEM line 4: fastq/S1_R1.fastq.gz already used on line 2 (S1)
3 samples, 5 files, 2 problems -> bad-manifest.json
exit=2
```

**Validation:** exit code 0 for the clean sheet and 2 for the broken one; `run-manifest.json` contains the samplesheet SHA-256 and one
entry per file. In a real pipeline wrapper, a non-zero exit must stop the launch.

**Troubleshooting:** `md5sum: command not found` on macOS → use `md5 -q` in the helper. Every file reported as mismatched → the
provider's checksums were computed on uncompressed data; ask which. **Teardown:** `rm -rf ~/bioops-lab2`.

**Extend it:** check BAM/CRAM contig names against the bundle and sample IDs against the LIMS export ([LabOps](../labops/)).

#### Lab 3 (advanced): measure variant-calling accuracy against a truth set

**Objective:** replace "the pipeline finished" with measured recall and precision per variant type, as the GA4GH benchmarking
framework recommends (match representations, standard metrics, stratification).[^krusche]

**Prerequisites:** your pipeline's GRCh38 VCF for GIAB sample HG002; the GIAB HG002 v5.0q GRCh38 small-variant benchmark VCF and
BED;[^giab-ftp] the reference FASTA you called against; Apptainer or Docker; about 20 GB disk. **Cost and risk:** public HG002 data
is listed by GIAB.[^giab]

**Which truth set.** As of 2026-10-23 NIST marks the long-used v4.2.1 small-variant benchmark "deprecated for HG002 by v5.0q".[^giab]
v5.0q is assembly-based, covers small variants and SVs, and its README calls it a draft whose results "should be carefully
evaluated"; some regions in v4.2.1 are excluded from it.[^giab-ftp] Numbers from the two versions are therefore not comparable:
record the version, and do not switch mid-study without re-running the earlier comparisons.

_Status: download and checksum steps executed on 2026-10-23; the comparison was not (no query VCF or container runtime here).
File names and `--gender male` follow the v5.0q README; the hap.py image (built 2023-05-23, Python 2.7) was listed on Quay on
2026-10-23.[^quay-samtools] The README recommends hap.py's `vcfeval` engine, which needs RTG Tools in the image (bioconda lists it
only as a build dependency), so the command uses the default `xcmp` engine._

```bash
GIAB=https://ftp-trace.ncbi.nlm.nih.gov/ReferenceSamples/giab/release
GIAB=$GIAB/AshkenazimTrio/HG002_NA24385_son/v5.0q
T=HG002_GRCh38_v5.0q_smvar
# hap.py:0.3.15--py27hcb73b3d_0, pinned by digest
IMG=docker://quay.io/biocontainers/hap.py
IMG=$IMG@sha256:d63b963a6cb01b4830393b22369e7b91d298e4156dde353739e74e4cfa4f96d0
mkdir -p ~/bioops-lab3 && cd ~/bioops-lab3
for f in "$T.vcf.gz" "$T.vcf.gz.tbi" "$T.benchmark.bed" checksum.md5; do
  curl -sfO "$GIAB/$f"
done
grep -E "$T\.(vcf\.gz|benchmark\.bed)$" checksum.md5 | md5sum -c -   # expect 2 x OK
# query.vcf.gz = your HG002 calls; ref.fa (+ .fai) = the reference you called against
apptainer exec "$IMG" hap.py "$T.vcf.gz" query.vcf.gz -f "$T.benchmark.bed" \
  -r ref.fa -o hg002_vs_giab --pass-only --gender male
column -s, -t < hg002_vs_giab.summary.csv | cut -c1-120
```

**Expected output:** `hg002_vs_giab.summary.csv` with rows for SNP and INDEL, and columns including `TRUTH.TP`, `TRUTH.FN`,
`QUERY.FP`, `METRIC.Recall`, `METRIC.Precision` and `METRIC.F1_Score`.[^happy] The [metrics section](#metrics-and-kpis) shows how
to read them.

**Validation:** both downloads report `OK` against GIAB's checksums; the benchmark uses `chr`
names for chromosomes 1–22, X and Y, so your query must too; rerunning gives the same numbers. Record the truth-set version, BED,
reference, image digest and options next to the result.

**Troubleshooting:** almost no matches → contig names or build differ;[^galaxy-chr] fix the query, never the truth set. Low indel
recall → check normalisation and indel filters.

**Teardown:** `rm -rf ~/bioops-lab3`; delete the cached image from Apptainer's cache (`apptainer cache clean`) if wanted.

### Use cases and scenarios

#### Case 1: nf-core/sarek 3, storage versus memory (sourced)

**Problem.** A variant-calling pipeline had a large storage footprint, partly from "Nextflow's requirement for a work directory
with intermediate results to facilitate resuming".[^sarek]
**Method.** Version 3 switched internal files after duplicate marking from BAM to CRAM and increased parallelism within each
sample. **Result reported by the authors**, benchmarking five tumour–normal pairs: the pre-processing work directory dropped by 65%,
from 170.4 TB to 59.7 TB; average CPU hours fell from 3,761.07 to 3,252.37; memory use rose by 34%. Together with the runtime
changes, the abstract reports "a 70% cost reduction in commercial clouds" (3.1.1 against 2.7.x, AWS Batch spot); other
sections give 50–70% and name 2.7.1 or 2.7.2 as the baseline, so treat 70% as the upper end.[^sarek]
**Trade-off.** CRAM saves storage but costs memory and ties every file to its exact reference. **Success criteria:** storage per
sample down, benchmark unchanged, reference archived by MD5. **Pitfall:** a "same" reference with different contigs leaves archived
CRAMs undecodable.[^cram]

#### Case 2: a clinical NGS pipeline under validation (sourced guidance, generic lab)

**Context.** A US clinical laboratory offers an NGS test; the pipeline is part of the test. **Method.** Under the AMP/CAP guideline
the laboratory validates its own pipeline (recommendation 1) after design and optimisation (3), in an environment that emulates
real use (4); preserves sample identity "with a minimum of four unique identifiers" (sample, patient, run, laboratory location)
in each file (10); protects file integrity (13); uses in-silico validation only to supplement end-to-end validation with human
samples (14); and revalidates after significant changes (17).[^ampcap]
**Trade-off.** Every release now costs validation work, so releases become rarer and larger. **Success criteria:** zero results
released without passing identity checks; validation report per release; time from approved change to production. **Pitfall:**
treating a container rebuild as "no change"; the tool version inside may have moved.

#### Case 3: a core facility moves from scripts to managed pipelines (illustrative)

::: example Illustrative scenario
A university sequencing core runs RNA-seq and exome analyses for 40 groups with Bash scripts on Slurm; results differ between
staff and old analyses cannot be rerun. **Method:** nf-core rnaseq and sarek at pinned releases, one site config, a Lab 2-style
gate, manifests stored with results, work directories deleted 30 days after release. **Measures:** reproduction rate, storage per
sample, turnaround. **Pitfall:** community defaults that do not fit a library kit; validate on the facility's own data.
:::

#### Case 4: a sequencing-based assay in a GMP laboratory (illustrative)

::: example Illustrative scenario
A biologics manufacturer uses sequencing to confirm the identity of a cell bank, so the pipeline produces GMP records and is a
computerised system. Annex 11 says: "The application should be validated; IT infrastructure should be qualified."[^annex11]
**Method:** freeze one release for the assay, validate it for its intended use on qualified infrastructure, keep audit trails,
and route changes through change control; research and GMP pipelines then need separate releases and owners. **Pitfall:** treating Git history as the audit trail for runs and approvals (see
[compliance](#quality-security-privacy-and-compliance)). Your quality unit decides; this is not regulatory advice.
:::

## Measure, govern, pay

### Metrics and KPIs

Measure correctness first, then speed and cost.

| KPI | Definition / formula | Data source | Pitfall | Cadence |
|---|---|---|---|---|
| Provenance completeness | Released results with all five pins (engine version included) recorded ÷ released results | Run manifests | Counting a tag as a pin | Monthly; target 100% |
| Reproduction rate | Scheduled re-runs that match the original (checksums, or agreed tolerance) ÷ re-runs attempted | Monthly re-run of one archived batch | Comparing files with embedded timestamps byte-for-byte; compare content | Monthly |
| Identity-check failure rate | Samples failing genotype/sex checks ÷ samples checked | Identity reports | Zero can mean "check not running" | Per batch |
| First-pass run success | Runs finishing without manual intervention ÷ runs started | Engine logs | Hiding failures with blanket retries | Weekly |
| Turnaround time | Median and 90th percentile hours, gate accepted → results released | Manifest timestamps | Averages hide queue spikes | Weekly |
| Benchmark accuracy | Recall = TRUTH.TP ÷ (TRUTH.TP + TRUTH.FN); precision = QUERY.TP ÷ (QUERY.TP + QUERY.FP); F1 = 2PR ÷ (P + R)[^happy] | hap.py or vcfeval per release | One number for all variant types; report SNVs and indels separately[^krusche] | Per release |
| CPU efficiency | Σ(%cpu ÷ 100 × realtime) ÷ Σ(cpus × realtime) per process | Trace file[^nf-trace] | Short tasks give unreliable metrics[^nf-reports] | Monthly |
| Cost per sample | (compute + storage for the period + transfer) ÷ samples released | Billing tags per batch + manifests | Leaving out storage of work directories | Monthly |

::: example Worked calculation (illustrative numbers)
**CPU efficiency.** Three `ALIGN` tasks each request 16 CPUs. Their trace shows %cpu 1,120% for 2.0 h, 1,440% for 1.5 h and 400% for
1.0 h. CPU-hours used = 11.2 × 2.0 + 14.4 × 1.5 + 4.0 × 1.0 = 48. CPU-hours requested = 16 × (2.0 + 1.5 + 1.0) = 72. Efficiency =
48 ÷ 72 = **67%**. The third task is the outlier; look at its input size before cutting the request for everyone.

**Cost per sample.** A 96-sample batch costs 480 in compute, 150 for one month of work-directory storage before clean-up, 40 for
results storage and 30 for transfer (any currency): (480 + 150 + 40 + 30) ÷ 96 = **7.29 per sample**; the work directory is the
easiest line to cut.

**Accuracy.** For SNVs: 3,000,000 truth variants, TRUTH.TP 2,985,000, TRUTH.FN 15,000, QUERY.FP 6,000 (assume QUERY.TP = TRUTH.TP).
Recall = 0.9950, precision = 0.9980, F1 = **0.9965**. For indels: 500,000 truth, 470,000 TP, 9,000 FP gives recall 0.940, precision
0.981, F1 **0.960**. A single combined number would hide that indels are the weak point.
:::

The accuracy KPI applies to variant calling. Other assays need their own reference material, such as ERCC RNA spike-ins, used "to
measure sensitivity, accuracy, and biases in RNA-seq experiments".[^ercc] Set targets from your own baseline. An accuracy figure means nothing without its regions: SNV concordance of two methods was
"99.7% inside versus 76.5% outside high-confidence regions".[^krusche]

### Quality, security, privacy and compliance

**Quality.** Small-data tests catch code regressions, benchmarks scientific ones, identity checks swaps, and a contamination
estimate mixed samples.[^verifybamid] NGSCheckMate compares allele read fractions at known SNPs from FASTQ, BAM or VCF above
0.5×.[^ngscheckmate] Keep thresholds and their rationale in the validation documents.

**Security.** The compute role should read the batch's inputs and reference bundle and write its work and output prefixes,
nothing else: replace setup-guide policies such as `AmazonS3FullAccess` with prefix-scoped ones.[^nf-awsbatch] Update pinned
images on a schedule ([ContainerOps](../containerops/)). On shared clusters, Apptainer users "cannot gain additional privilege on the host system
by default".[^apptainer] Encrypt sensitive files at rest; crypt4gh wraps genomic files in an encryption layer.[^hts-specs]

**Privacy (jurisdiction-specific, not legal advice).** In the EU, genetic data is listed as sensitive personal data "subject to
specific processing conditions" (GDPR Articles 4(13) and 9).[^ec-sensitive] Record in each manifest the consent or data-access
approval the batch was processed under. The privacy programme belongs to [PrivacyOps](../privacyops/); data-platform access
policies to [DataSecOps](../datasecops/).

**Regulated computing (GxP and clinical).** If results feed patient care or regulated manufacturing, the pipeline is part of a
validated test or computerised system:

- **US FDA-regulated records:** 21 CFR Part 11 requires validation "to ensure accuracy, reliability, consistent intended performance,
  and the ability to discern invalid or altered records", and "secure, computer-generated, time-stamped audit trails" (§11.10, eCFR
  current to 2026-10-21).[^part11]
- **EU GMP:** Annex 11 (revision January 2011) applies risk management "throughout the lifecycle of the computerised system", requires
  validation and qualified infrastructure, and controlled change.[^annex11] A revised Annex 11 and a new Annex 22 (AI) were
  consulted on until 7 October 2025; on 2026-10-23 the EudraLex page still listed the 2011 text.[^annex11-draft]
- **Industry guidance:** GAMP 5 (second edition, July 2022) describes a risk-based approach and stresses "critical thinking" by
  subject-matter experts.[^gamp5]
- **US clinical laboratories:** the AMP/CAP recommendations in [case 2](#case-2-a-clinical-ngs-pipeline-under-validation-sourced-guidance-generic-lab).[^ampcap]

What BioOps contributes is evidence: release records, validation reports, run manifests, audit trails of who ran and approved
what. The compliance programme itself belongs to [ComplianceOps](../complianceops/).

### Cost, capacity and sustainability

Drivers are compute, storage, transfer and people time spent on failed runs.

- **Storage first.** Work directories hold every intermediate and are often the biggest line. Expire them after release and
  consider CRAM ([case 1](#case-1-nf-coresarek-3-storage-versus-memory-sourced)).[^sarek]
- **Right-size requests** per process from the trace, with retry-on-memory for outliers.[^nf-process]
- **Spot capacity** suits short, resumable tasks; long tasks lose work on each interruption.[^aws-spot]
- **Data gravity.** Move compute to large or legally constrained data; WES aims to bring "algorithms to genomic data".[^ga4gh-wes]
- **Exit costs.** Managed-service availability changes in both directions. AWS closed HealthOmics variant and annotation stores
  to new customers "starting November 7th, 2025".[^healthomics] Its bioinformatics CI/CD guidance relies on CodeCommit, which AWS
  closed to new customers in July 2024 and returned to general availability on 24 November 2025;[^codecommit-ga] on 2026-10-23 the
  guidance page still said CodeCommit "is no longer available to new customers".[^aws-guidance] Keep pipelines in a portable
  engine and data in open formats.

The sarek authors present their savings as lowering both cost and CO2 emissions;[^sarek] see [GreenOps](../greenops/) for carbon
accounting and [FinOps](../finops/) for organisation-wide cloud cost.

### Assurance of the operating model

Four scheduled checks show whether the operating model works:

1. **Monthly re-run** of one archived batch from its manifest alone, by someone else. Differences are defects.
2. **Manifest audit:** can ten sampled results each be traced to sheet line, checksums, release, images and bundle in 15 minutes?
3. **Gate drill:** feed the gate a known-bad batch (like Lab 2) after every gate change.
4. **Retention audit:** stored bytes without an owner and deletion date are findings.

## When things go wrong

### Common failures and troubleshooting

| Symptom | Likely cause | Diagnosis | Remediation | Prevention |
|---|---|---|---|---|
| Resume re-runs everything | Work directory or `.nextflow/cache` deleted; launched from another directory; NFS timestamps | `nextflow log`; compare hashes with `-dump-hashes`[^nf-cache] | Relaunch from original directory; `lenient` cache on NFS | Keep cache with the run; cloud cache on VMs |
| Resume reuses a wrong result | Process edits its inputs; non-deterministic channel merge | Inspect process code; compare inputs by sample ID | `join` on sample ID; never modify inputs[^nf-cache] | Code review rule; tests with shuffled input order |
| Few or no variants match a truth set | Contig names or genome build differ | Compare VCF header contigs with the bundle | Re-run with the right bundle | Contig check in the input gate[^galaxy-chr] |
| CRAM cannot be read | Reference missing or different | Reader reports MD5 mismatch[^cram] | Restore exact reference by MD5 | Archive reference with CRAMs |
| Task killed with exit 137 | Out of memory or a scheduler limit | Trace `peak_rss` vs `memory` | Retry with more memory on 137–140 only[^nf-process] | Per-process sizing from trace |
| Results change after "no change" | Image tag rebuilt; engine upgraded | Compare digests and engine versions in manifests | Pin digests; re-run | Digest pinning; `NXF_VER` |
| Sample's genotype matches another sample | Swap or duplicate | Identity report; LIMS records | [Runbook below](#worked-runbook-a-suspected-sample-swap) | Identity gate; four identifiers in files[^ampcap] |
| Excess heterozygosity in one sample | Contamination (mixed DNA) | Contamination estimate[^verifybamid] | Exclude or re-sequence above a set threshold (studies often use 1–3%)[^verifybamid] | Contamination check in the QC gate |

### Worked runbook: a suspected sample swap

**Trigger.** The post-run identity check shows that sample P-0142's genotype matches sample P-0147 from the same batch, and P-0142's
reported sex disagrees with the LIMS record. Nothing from the batch has been released. (Variant: a swap found **after** release,
for example when a later sample from the same person disagrees; then step 5 applies to everyone who received affected results.)

::: figure Sample-swap response: contain, locate, correct, prevent
```mermaid
flowchart TB
  A["Identity check fails"] --> B["Hold all results of the batch"]
  B --> C{"Swap inside the pipeline?"}
  C -->|"samplesheet or code"| D["Fix sheet, re-run affected samples"]
  C -->|"before data arrived"| E["Lab investigates tubes and plates"]
  E --> F{"Cause found?"}
  F -->|"yes"| D
  F -->|"no"| G["Withdraw samples, request re-sequencing"]
  D --> H["Identity re-check passes"]
  H --> I["Release; notify if already released"]
  G --> K["Tell requesters; notify if already released"]
  K --> J["Post-incident review"]
  I --> J
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  class A,B,G risk
  class C,F decision
  class D,E,H,I,J,K core
```
Text equivalent: when an identity check fails, hold the whole batch and decide whether the swap happened inside the pipeline
(samplesheet or code) or before the data arrived. Inside: fix the sheet and re-run the affected samples; the identity re-check
must pass before release. Before: the lab investigates; if it finds the cause, the fix path follows, otherwise the samples are
withdrawn, re-sequencing is requested and the requesters are told. On both paths anyone who already received results is
notified, and both end in a post-incident review.
:::

1. **Contain (first hour).** Hold the whole batch, not just two samples: swaps are often shifted rows. Record the hold.
2. **Locate.** Compare the samplesheet with the LIMS export ([LabOps](../labops/)); check the gate manifest for duplicated paths;
   run the identity tool across the batch and against earlier data from the same individuals.
3. **Decide where.** Sheet or pipeline error → correct and re-run from raw data (never rename outputs). Lab error → the lab
   investigates; if unresolved, withdraw the samples.
4. **Verify.** Identity checks must pass for every sample in the batch.
5. **Notify.** Before release: tell the requesters about the delay or withdrawal. After release: notify everyone who received
   affected results; in clinical settings the laboratory director decides reporting.
6. **Prevent.** Review within a week; add the pattern to the input gate; track actions ([IncidentOps](../incidentops/)).

### Anti-patterns and myths

- **"It's in a container, so it's reproducible."** Containers pin software, not reference, parameters, inputs or engine.[^nf2017]
- **`latest` tags and branch runs in production.** Results change with no code change.
- **Notebooks as pipelines.** Fine for exploration, but they rarely rerun;[^jupyter] promote stable analysis to the pipeline.
- **Keeping every work directory "just in case".** Storage grows without an owner; keep manifests and final outputs instead.
- **Myth: "validation is only for clinical labs."** Research groups gain from a light version: a benchmark per release and a
  written intended use.
- **Myth: "a managed platform means no operations."** Someone still owns pins, gates, costs and exit plans ([NoOps](../noops/)).

## Deciding and adopting

### When to use and when not to use

**Use the full practice** when analyses are shared or repeated (a core facility, a consortium, a service), when results inform
publications others build on, when data volumes make re-runs expensive, or when results enter clinical or GxP decisions.

**Use a light version** (pinned environment, Git, a run log) for one-off exploratory analyses by one person.

**Do not over-build** when a project is a short pilot whose results will be regenerated anyway, or when the team has no capacity to
own a pipeline; a well-maintained community pipeline run at a pinned release is often better than an in-house one. Hidden costs of
the full practice: validation effort per release, release discipline that slows experimentation, and storage of provenance.

### Decision framework

::: figure How much BioOps does an analysis need?
```mermaid
flowchart TB
  Q1{"Results used in patient care or GMP?"}
  Q1 -->|"yes"| R1["Validated pipeline, change control, audit trails"]
  Q1 -->|"no"| Q2{"Shared, repeated, or published?"}
  Q2 -->|"no"| R3["Light: pinned env, Git, run log"]
  Q2 -->|"yes"| Q3{"Community pipeline fits?"}
  Q3 -->|"yes"| R2a["Run it pinned, local config, gates"]
  Q3 -->|"no"| R2b["Own pipeline: tests, benchmark, releases"]
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  class Q1,Q2,Q3 decision
  class R1 risk
  class R2a,R2b core
  class R3 neutral
```
Text equivalent: if results are used in patient care or GMP, use a validated pipeline with change control and audit trails. If not,
ask whether the analysis is shared, repeated or published. If it is not, a light approach is enough: pinned environment, Git and a
run log. If it is, ask whether a community pipeline fits: if yes, run it at a pinned release with local configuration and gates; if
no, own a pipeline with tests, benchmarks and releases.
:::

### Maturity model

**This Atlas's model**, not an industry standard. A published, broader five-level model for scientific operations, SciOps, also
derives from DevOps;[^sciops] the levels below are narrower and specific to pipelines.

| Level | Observable behaviour | Exit criteria to move up |
|---|---|---|
| 0 Ad hoc | Scripts on laptops and clusters; tools installed by hand | Code in Git; one documented environment |
| 1 Scripted | Versioned scripts; environments recorded but not pinned | A workflow manager with containers for one main analysis |
| 2 Managed | Workflow manager, pinned releases and images, input gate | Run manifest for 100% of released results; CI tests |
| 3 Assured | Identity gate, truth-set benchmarks per release, retention rules, cost per sample | Monthly re-run passes; benchmark deltas reviewed per release |
| 4 Validated | Intended use documented; formal validation; change control; audit trails | Sustained: audits find no untraceable result |

Level 4 is required only where regulation demands it; level 3 is a sensible target for shared research services.

### Adoption roadmap: 30/60/90 days

| Window | Actions | Milestone | Metric |
|---|---|---|---|
| Days 1–30 | Inventory analyses and where results go; pick one high-volume analysis; choose engine and community pipeline; pin engine, release and images; write the run-manifest format | First pinned run of one analysis | Provenance completeness for that analysis |
| Days 31–60 | Input gate (Lab 2 pattern) and identity check; CI tests; site config for HPC and cloud; work-directory expiry | Gates block a bad batch in a drill | Gate rejection rate; first-pass success |
| Days 61–90 | Truth-set benchmark (Lab 3) and per-release review; cost per sample report; monthly re-run; decide which other analyses to migrate | First release with benchmark sign-off | Reproduction rate; cost per sample |

Quick win: digest pinning and the input gate are cheap and catch silent failures in the first month.

## Toolkit

### Checklists

#### Pipeline release readiness

- [ ] Tagged release with notes; engine version pinned
- [ ] Every image pinned by digest; digests listed in the release record
- [ ] Reference bundle ID and checksum file referenced
- [ ] CI tests pass on the release commit
- [ ] Benchmark run on the release; deltas against the previous release reviewed and signed
- [ ] Change class decided (patch, minor, major); validation done where required

#### Per batch

- [ ] Input gate passed (checksums, duplicates, contig names); manifest started
- [ ] Release and profile recorded; run launched with `-r <tag>` and pinned engine
- [ ] Identity and QC gates passed for every sample
- [ ] Results and manifest stored together; work directory expiry date set

### Templates

#### Pipeline release record

```text
Pipeline / release:        germline-pipeline 2.3.1   (commit: <id>)
Change class:              minor  (tool patch: samtools 1.23 -> 1.24)
Engine:                    nextflow 26.10.3          Reference bundle: GRCh38-noalt-v3
Images (digests):          see release-images.json
Tests:                     CI run <link>, all passed
Benchmark (HG002 v5.0q):   SNV F1 0.9965 (prev 0.9964); indel F1 0.9601 (prev 0.9598)
Intended use unchanged:    yes
Approved by / date:        <scientific lead or QA>, <YYYY-MM-DD>
```

#### Change request

```text
Requested change:   what and why (bug, new tool version, new genome build)
Affected outputs:   files and fields that may change
Risk:               low / medium / high, with reason
Evidence required:  tests only | tests + benchmark | full revalidation
Rollback:           previous release tag; results re-run needed?
Decision:           approved / rejected, by whom, date
```

### Workflow managers compared

Criteria with evidence only; versions as of October 2026.

| Criterion | Nextflow | Snakemake | CWL (with cwltool, Toil, others) | WDL (with Cromwell, miniwdl, others) |
|---|---|---|---|---|
| Model | Dataflow processes and channels | Rules with file targets (Python-based) | Open standard for tools and workflows[^cwl] | Openly governed workflow language[^wdl] |
| Resume | Task cache plus work directory[^nf-cache] | Rerun triggers on code, input, params, environment[^smk-cli] | Engine-dependent | Engine-dependent |
| Software | Per-process container, Conda, Spack[^nf-container] | `container:` / `conda:` with `--sdm`[^smk-deploy] | Docker requirement in the standard | Runtime container attribute |
| Community pipelines | nf-core (106 released, non-archived on 2026-10-23)[^nfcore-json] | Snakemake workflow catalogue | WorkflowHub, Dockstore | Dockstore, Terra |
| Provenance | Trace, report, experimental lineage;[^nf-lineage] Workflow Run RO-Crate via the nf-prov plugin[^wrroc] | Reports; Workflow Run RO-Crate via a report plugin[^wrroc] | CWLProv runs convertible to Workflow Run RO-Crate[^wrroc] | Engine metadata |

Choose by team language and the community pipelines you will use.

### FAQ

**Is BioOps a job title?** Not a common one. Genomics England has a "Bio-Ops" team with the role "Operations
Bioinformatician";[^ge-bioops] elsewhere bioinformaticians, research software engineers and research-computing staff do this work.

**Is a Conda environment file enough?** Only if it pins exact builds; images pinned by digest are easier to prove.

**Why pin the reference bundle when the genome build does not change?** Because indexes, annotation releases, decoys and contig names
do change within a build, and CRAM decoding needs the exact sequence.[^cram]

**Results differ slightly between HPC and cloud. Is that a bug?** Not always; platform differences exist.[^nf2017] Compare against
a tolerance agreed in advance and investigate anything outside it.

**Is our research pipeline "GxP"?** Only if its results support regulated decisions. If they do, your quality unit decides the
validation approach; this page is not regulatory advice.

## Going further

### Advanced topics and emerging direction

- **Queryable provenance (in development):** Nextflow lineage and Workflow Run RO-Crate;[^nf-lineage][^wrroc] expect format changes.
- **Federated analysis (standards exist, adoption uneven):** WES and TES run one workflow where the data is.[^ga4gh-tes]
- **Wider benchmarks (active research):** GIAB's assembly-based HG002 v5.0q covers small variants and SVs together, and its older
  SV benchmark is listed as deprecated.[^giab]
- **Citable pipelines (community principles):** FAIR4RS identifiers and metadata for workflows.[^fair4rs]

### Learning path

- **Prerequisites:** Linux shell, Git, containers ([ContainerOps](../containerops/)).
- **One day:** run Lab 1; read the ten simple rules.[^sandve]
- **One week:** Lab 2 on a real batch; write a site config for your cluster; add one nf-test or Snakemake test.
- **30 days:** Lab 3 with your pipeline; a release record; a cost-per-sample report.
- **Specialise:** clinical validation (AMP/CAP); GxP computerised systems (Annex 11, Part 11, GAMP 5); federated analysis (GA4GH).
- **Portfolio project:** take a published analysis, rebuild it as a pinned pipeline with a manifest, and show it reruns.

## Wrap-up

### Executive recap and cheat sheet

| Area | Remember |
|---|---|
| Decide | How much assurance the use needs: light, assured, or validated ([decision framework](#decision-framework)) |
| Pin | Release with engine version, image digests, reference bundle, parameters (with seeds), input checksums |
| Gate | Inputs before compute; identity and QC after |
| Measure | Provenance completeness, reproduction rate, identity failures, accuracy by variant type, cost per sample |
| Watch | Resume rules, contig naming, CRAM references, blanket retries, work-directory storage, managed-service changes |
| Hand off | Samples and LIMS ← [LabOps](../labops/); publication → [ResearchDataOps](../researchdataops/) |

### A judgement call: upgrade mid-study or freeze?

::: decision Dilemma
**Situation (illustrative).** A 400-sample cohort study has processed 250 samples with pipeline release 2.2. Release 2.3 fixes an
indel-filtering bug: on HG002, indel F1 rises from 0.951 to 0.960; SNVs are unchanged. 150 samples remain. Re-running all 400
costs about two weeks of compute budget.

**Option A — freeze 2.2** for the rest of the study: consistent processing, known (lower) indel accuracy.
**Option B — upgrade now**: better indels for 150 samples, but a processing batch effect correlated with sample-collection time.
**Option C — upgrade and re-run all 400**: consistent and better, at the cost of time and budget.

**Assumptions to test.** Does the main question depend on indels? Are early and late samples balanced across compared groups?

**Reasoning.** Option B looks attractive but is usually wrong: if early samples are mostly controls and late ones cases, the
pipeline change can create a false group difference. If indels matter, choose C; if not, A is defensible with
the limitation documented. **Uncertainty:** HG002 gains may not transfer to your samples; re-run a few and compare first.
:::

### Learning objectives and mastery check

After this page you can explain the five pins and two gates, design an input gate, measure accuracy against a truth set,
decide how much assurance an analysis needs and run a sample-swap response.

::: answer A colleague says the pipeline is reproducible because every tool runs in a container. What is missing?
The other four pins: the pipeline release with its engine version, the reference bundle, parameters, and input checksums. Even
the software pin needs digests, not tags.
:::

::: answer One sample shows far more heterozygous calls than expected but matches no other sample. Swap or contamination?
Contamination: a swap makes a sample match someone else, while mixed DNA raises heterozygosity. Run a contamination estimator.
:::

::: answer Your variant comparison against GIAB finds almost no matches. What do you check first?
Genome build and contig naming (`chr1` versus `1`), then whether the right benchmark BED and reference were used.
:::

::: answer Why hold the whole batch when only two samples look swapped?
Swaps often come from shifted rows or plates, so other samples may be affected; holding the batch prevents releasing wrong results
while the cause is found.
:::

::: answer When does a research group not need the full practice?
For one-off, single-person exploratory analyses whose results will be regenerated anyway: a pinned environment, Git and a run log
are enough.
:::

### Conclusion

BioOps is a borrowed name for an old discipline: making computational biology answerable. Apart from one team name, nobody uses
the label, and that is fine. What matters is that a result can be regenerated from its manifest, attributed to the right sample, and paid for knowingly.
The tools are mature; the hard part is habit: pin everything, gate inputs, check identity, benchmark each release, delete what you do
not need. Start with the analysis you run most often and make it rerun from its manifest.

## Sources and further reading

All sources were opened on 2026-10-23; the evidence ledger is `research/bioops-ledger.md`.

[^ge-bioops]: [What does a bioinformatician do?](https://www.genomicsengland.co.uk/blog/what-does-a-bioinformatician-do) — Genomics England blog (Aminata Tengbe), 2024-06-17. Checked 2026-10-23.
[^patsnap]: [bioops — company profile](https://discovery.patsnap.com/company/bioops/) — Patsnap Discovery, undated. Checked 2026-10-23.
[^biops-aws]: [BIOps: Amazon QuickSight object migration and version control](https://aws.amazon.com/blogs/big-data/biops-amazon-quicksight-object-migration-and-version-control/) — AWS Big Data Blog, 2021-08-26. Checked 2026-10-23.
[^biops-py]: [biops documentation](https://biops.readthedocs.io/en/latest/) — biops project, undated. Checked 2026-10-23.
[^bioopsec]: [Biosensing Surveillance Technologies: A Quick Guide to Biological Operations Security](https://jsou.edu/Press/PublicationDashboard/285) — JSOU Press, 2025-07-30. Checked 2026-10-23.
[^aws-guidance]: [Guidance for Bioinformatics Workflow Development Using DevOps on AWS](https://docs.aws.amazon.com/solutions/bioinformatics-workflow-development-using-devops-on-aws/) — AWS Solutions Library, undated. Checked 2026-10-23.
[^codecommit-ga]: [AWS CodeCommit returns to General Availability](https://aws.amazon.com/blogs/devops/aws-codecommit-returns-to-general-availability/) — AWS DevOps Blog, 2025-11-24. Checked 2026-10-23.
[^biogitflow]: [biogitflow: development workflow protocols for bioinformatics pipelines with git and GitLab](https://f1000research.com/articles/9-632/v3) — Kamoun et al., F1000Research 9:632, v3, 2021. Checked 2026-10-23.
[^sciops]: [SciOps: Achieving Productivity and Reliability in Data-Intensive Research](https://arxiv.org/abs/2401.00077) — Johnson et al., arXiv v2, 2024-11-06 (some authors declare equity in DataJoint Inc.). Checked 2026-10-23.
[^nf2017]: [Nextflow enables reproducible computational workflows](https://www.nature.com/articles/nbt.3820) — Di Tommaso et al., Nature Biotechnology 35:316–319, 2017 (paywalled; summary and first paragraph read). Checked 2026-10-23.
[^nf-cache]: [Caching and resuming](https://docs.seqera.io/nextflow/cache-and-resume) — Nextflow documentation, as of October 2026. Checked 2026-10-23.
[^nf-container]: [Containers](https://docs.seqera.io/nextflow/container) — Nextflow documentation. Checked 2026-10-23.
[^nf-executor]: [Executors](https://docs.seqera.io/nextflow/executor) — Nextflow documentation. Checked 2026-10-23.
[^nf-reports]: [Reports](https://docs.seqera.io/nextflow/reports) — Nextflow documentation. Checked 2026-10-23.
[^nf-trace]: [trace configuration](https://docs.seqera.io/nextflow/reference/config/trace) — Nextflow configuration reference (26.10). Checked 2026-10-23.
[^nf-process]: [Processes: dynamic resources and retries](https://docs.seqera.io/nextflow/process) — Nextflow documentation. Checked 2026-10-23.
[^nf-lineage]: [Getting started with data lineage](https://docs.seqera.io/nextflow/tutorials/data-lineage) — Nextflow documentation (experimental since 25.04). Checked 2026-10-23.
[^nf-update]: [Updating Nextflow](https://docs.seqera.io/nextflow/updating-nextflow) — Nextflow documentation. Checked 2026-10-23.
[^nf-install]: [Installation](https://docs.seqera.io/nextflow/install) — Nextflow documentation. Checked 2026-10-23.
[^nf-releases]: [nextflow-io/nextflow releases](https://github.com/nextflow-io/nextflow/releases) — GitHub, v26.10.3 on 2026-10-20. Checked 2026-10-23.
[^nf-awsbatch]: [AWS Batch and AWS cloud](https://docs.seqera.io/nextflow/aws) — Nextflow documentation. Checked 2026-10-23.
[^bash-exit]: [Exit Status](https://www.gnu.org/software/bash/manual/html_node/Exit-Status.html) — GNU Bash Reference Manual. Checked 2026-10-23.
[^nf-aws]: [aws configuration](https://docs.seqera.io/nextflow/reference/config/aws) — Nextflow configuration reference (`aws.batch.maxSpotAttempts`). Checked 2026-10-23.
[^smk-paper]: [Sustainable data analysis with Snakemake](https://pmc.ncbi.nlm.nih.gov/articles/PMC8114187/) — Mölder et al., F1000Research 10:33, 2021 (v3 2025). Checked 2026-10-23.
[^smk-deploy]: [Distribution and reproducibility](https://snakemake.readthedocs.io/en/stable/snakefiles/deployment.html) — Snakemake documentation. Checked 2026-10-23.
[^smk-cli]: [Command line interface](https://snakemake.readthedocs.io/en/stable/executing/cli.html) — Snakemake documentation (`--rerun-triggers`). Checked 2026-10-23.
[^smk-releases]: [snakemake/snakemake releases](https://github.com/snakemake/snakemake/releases) — GitHub, v9.27.1 on 2026-10-21. Checked 2026-10-23.
[^nfcore-paper]: [The nf-core framework for community-curated bioinformatics pipelines](https://pubmed.ncbi.nlm.nih.gov/32055031/) — Ewels et al., Nature Biotechnology 38:276–278, 2020. Checked 2026-10-23.
[^nfcore-about]: [About nf-core](https://nf-co.re/about) — nf-core ("came about at the start of 2018"). Checked 2026-10-23.
[^nfcore-rnaseq]: [nf-core/rnaseq 3.27.0 usage: samplesheet input](https://nf-co.re/rnaseq/3.27.0/docs/usage/) — nf-core. Checked 2026-10-23.
[^nfcore-json]: [nf-core pipelines.json](https://nf-co.re/pipelines.json) — nf-core, read 2026-10-23 (counts are this Atlas's). Checked 2026-10-23.
[^nfcore-demo]: [nf-core/demo 1.2.0](https://github.com/nf-core/demo) — nf-core on GitHub, README and configuration at tag 1.2.0 (2026-06-24). Checked 2026-10-23.
[^nftest]: [nf-test](https://www.nf-test.com/) — nf-test project. Checked 2026-10-23.
[^cwl]: [Common Workflow Language](https://www.commonwl.org/) — CWL project. Checked 2026-10-23.
[^wdl]: [OpenWDL](https://openwdl.org/) — OpenWDL. Checked 2026-10-23.
[^ga4gh-wes]: [Workflow Execution Service (WES)](https://www.ga4gh.org/product/workflow-execution-service-wes/) — GA4GH. Checked 2026-10-23.
[^ga4gh-tes]: [Task Execution Service (TES)](https://www.ga4gh.org/product/task-execution-service-tes/) — GA4GH. Checked 2026-10-23.
[^apptainer]: [Introduction to Apptainer](https://apptainer.org/docs/user/latest/introduction.html) — Apptainer User Guide 1.5. Checked 2026-10-23.
[^biocontainers]: [BioContainers: an open-source and community-driven framework for software standardization](https://pmc.ncbi.nlm.nih.gov/articles/PMC5870671/) — da Veiga Leprevost et al., Bioinformatics, 2017. Checked 2026-10-23.
[^bioconda]: [Bioconda: sustainable and comprehensive software distribution for the life sciences](https://pmc.ncbi.nlm.nih.gov/articles/PMC11070151/) — Grüning et al., Nature Methods 15:475–476, 2018. Checked 2026-10-23.
[^quay-samtools]: [biocontainers/samtools on Quay](https://quay.io/repository/biocontainers/samtools) — Quay.io tag listing (also `biocontainers/hap.py` `0.3.15--py27hcb73b3d_0` = `sha256:d63b963a…f96d0`), read 2026-10-23. Checked 2026-10-23.
[^happy]: [hap.py user's manual](https://github.com/Illumina/hap.py/blob/master/doc/happy.md) — Illumina on GitHub. Checked 2026-10-23.
[^hts-specs]: [SAM/BAM and related specifications](https://samtools.github.io/hts-specs/) — GA4GH Large Scale Genomics work stream. Checked 2026-10-23.
[^cram]: [CRAM format specification (version 3)](https://samtools.github.io/hts-specs/CRAMv3.pdf) — hts-specs. Checked 2026-10-23.
[^vcf45]: [The Variant Call Format Specification, VCFv4.5 and BCFv2.2](https://samtools.github.io/hts-specs/VCFv4.5.pdf) — hts-specs; finalised 2024-06-28, document last revised 2026-02-25 ([commit history](https://github.com/samtools/hts-specs/commits/master/VCFv4.5.tex)). Checked 2026-10-23.
[^fastq]: [The Sanger FASTQ file format for sequences with quality scores, and the Solexa/Illumina FASTQ variants](https://pmc.ncbi.nlm.nih.gov/articles/PMC2847217/) — Cock et al., Nucleic Acids Research, 2010. Checked 2026-10-23.
[^galaxy-chr]: [Why do we change the chromosome names in the Ensembl GTF to match the UCSC genome reference?](https://training.galaxyproject.org/training-material/topics/proteomics/tutorials/proteogenomics-dbcreation/faqs/chromosome_names.html) — Galaxy Training Network FAQ. Checked 2026-10-23.
[^giab]: [Genome in a Bottle](https://www.nist.gov/programs-projects/genome-bottle) — NIST. Checked 2026-10-23.
[^giab-ftp]: [GIAB HG002 v5.0q release files and README](https://ftp-trace.ncbi.nlm.nih.gov/ReferenceSamples/giab/release/AshkenazimTrio/HG002_NA24385_son/v5.0q/) — NIST via NCBI FTP; README updated 2026-07-17. Checked 2026-10-23.
[^sandve]: [Ten Simple Rules for Reproducible Computational Research](https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1003285) — Sandve et al., PLOS Computational Biology, 2013. Checked 2026-10-23.
[^jupyter]: [Computational reproducibility of Jupyter notebooks from biomedical publications](https://pmc.ncbi.nlm.nih.gov/articles/PMC10783158/) — Samuel and Mietchen, GigaScience, 2024. Checked 2026-10-23.
[^ziemann]: [Gene name errors are widespread in the scientific literature](https://pmc.ncbi.nlm.nih.gov/articles/PMC4994289/) — Ziemann et al., Genome Biology 17:177, 2016. Checked 2026-10-23.
[^fair4rs]: [Introducing the FAIR Principles for research software](https://pmc.ncbi.nlm.nih.gov/articles/PMC9562067/) — Barker et al., Scientific Data, 2022. Checked 2026-10-23.
[^wrroc]: [Workflow Run RO-Crate](https://www.researchobject.org/workflow-run-crate/) — RO-Crate community. Checked 2026-10-23.
[^ampcap]: [Standards and Guidelines for Validating Next-Generation Sequencing Bioinformatics Pipelines](https://doi.org/10.1016/j.jmoldx.2017.11.003) — Roy et al. (AMP and CAP), Journal of Molecular Diagnostics 20(1), 2018 (full text read from a hosted PDF copy; publisher site blocked automated access). Checked 2026-10-23.
[^part11]: [21 CFR Part 11 — Electronic Records; Electronic Signatures](https://www.ecfr.gov/current/title-21/chapter-I/subchapter-A/part-11) — eCFR, current to 2026-10-21. Checked 2026-10-23.
[^annex11]: [EudraLex Volume 4, Annex 11: Computerised Systems](https://health.ec.europa.eu/system/files/2016-11/annex11_01-2011_en_0.pdf) — European Commission, revision January 2011. Checked 2026-10-23.
[^annex11-draft]: [Stakeholders' consultation on EudraLex Volume 4: Chapter 4, Annex 11 and new Annex 22](https://health.ec.europa.eu/consultations/stakeholders-consultation-eudralex-volume-4-good-manufacturing-practice-guidelines-chapter-4-annex_en) — European Commission, 7 July – 7 October 2025; status of the final text checked on the [EudraLex Volume 4 page](https://health.ec.europa.eu/medicinal-products/eudralex/eudralex-volume-4_en). Checked 2026-10-23.
[^gamp5]: [ISPE GAMP 5 Guide, 2nd Edition](https://ispe.org/publications/guidance-documents/gamp-5-guide-2nd-edition) — ISPE, July 2022 (publisher page; guide not read). Checked 2026-10-23.
[^ec-sensitive]: [What personal data is considered sensitive?](https://web.archive.org/web/20251220155325/https://commission.europa.eu/law/law-topic/data-protection/rules-business-and-organisations/legal-grounds-processing-data/sensitive-data/what-personal-data-considered-sensitive_en) — European Commission, Internet Archive copy of 2025-12-20 (live page returned 404). Checked 2026-10-23.
[^krusche]: [Best practices for benchmarking germline small-variant calls in human genomes](https://pmc.ncbi.nlm.nih.gov/articles/PMC6699627/) — Krusche et al., Nature Biotechnology 37:555–560, 2019. Checked 2026-10-23.
[^verifybamid]: [Ancestry-agnostic estimation of DNA sample contamination from sequence reads](https://pmc.ncbi.nlm.nih.gov/articles/PMC7050530/) — Zhang et al. (verifyBamID2), Genome Research 30(2):185–194, 2020. Checked 2026-10-23.
[^ngscheckmate]: [NGSCheckMate: software for validating sample identity in next-generation sequencing studies](https://pmc.ncbi.nlm.nih.gov/articles/PMC5499645/) — Lee et al., Nucleic Acids Research 45(11):e103, 2017. Checked 2026-10-23.
[^sarek]: [Scalable and efficient DNA sequencing analysis on different compute infrastructures aiding variant discovery](https://pmc.ncbi.nlm.nih.gov/articles/PMC11044436/) — Hanssen et al., NAR Genomics and Bioinformatics, 2024. Checked 2026-10-23.
[^aws-spot]: [Spot Instance interruption notices](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-instance-termination-notices.html) — Amazon EC2 User Guide. Checked 2026-10-23.
[^healthomics]: [AWS HealthOmics variant store and annotation store availability change](https://docs.aws.amazon.com/omics/latest/dev/variant-store-availability-change.html) — AWS documentation. Checked 2026-10-23.
[^globus]: [How To Log In and Transfer Files with Globus](https://docs.globus.org/guides/tutorials/manage-files/transfer-files/) — Globus documentation. Checked 2026-10-23.
[^ercc]: [Synthetic spike-in standards for RNA-seq experiments](https://pmc.ncbi.nlm.nih.gov/articles/PMC3166838/) — Jiang et al., Genome Research 21(9):1543–1551, 2011. Checked 2026-10-23.

**Maintainer note.** Values that drift first: engine and Action versions, the nf-core pipeline count, Quay digests, the Nextflow
launcher digest, the GIAB v5.0q draft status and the AWS service banners.

**Further reading**

- [WorkflowHub](https://about.workflowhub.eu/), a registry for computational workflows.

**Changelog**

- 2026-10-23 — First draft: term status confirmed as ambiguous; research ledger `research/bioops-ledger.md`; Lab 2 executed with
  Python 3.8.10; other examples syntax-checked only or not executed, as marked.
- 2026-10-23 — Revision after review r1 (`reviews/bioops-r1-response.md`): Genomics England "Bio-Ops" team disclosed; CodeCommit
  return to GA; Lab 3 moved to GIAB v5.0q (download and checksum steps run); AWS Batch profile completed; Lab 2 allows multi-lane
  samples (re-run); five pins made consistent; swap vs contamination separated; sarek, VCF and nf-core dates corrected.
- 2026-10-23, post-score polish: Workflow Run RO-Crate support credited to nf-prov (Nextflow) and the Snakemake report plugin;
  Lab 1 launcher pinned and checksum-verified (download step run); accuracy KPI scoped to variant calling with an RNA-seq
  spike-in pointer; maintainer note on fast-drifting values. After the polish check: Lab 1 checksum gates `chmod` with `&&` and
  gives the macOS `shasum -a 256 -c -` form (both re-run); labs intro lists the Lab 1 lines that were run.