BioOps

BioOps is not an established term. This Atlas uses it for the operational side of computational biology: running bioinformatics pipelines (alignment, variant calling, expression quantification) as a dependable service, so that every result can be reproduced, traced back to the samples, code, software and reference data that produced it, and delivered on time and at a known cost.

In one line: BioOps asks “if we ran this analysis again next year, on different hardware, would we get the same answer, and could we prove which sample, which code and which genome build it came from?”

The idea is older than the label: the first of Sandve and colleagues’ ten rules for reproducible computational research (2013) is “For Every Result, Keep Track of How It Was Produced”.^[1] This page turns that rule into an operating model.

For the users listed below; ContainerOps is the recommended prerequisite.

Computational biology fails quietly. A script gives a slightly different answer on the cluster; a spreadsheet renames genes; two FASTQ files are swapped in a samplesheet; chr1 meets 1. Nothing crashes; the numbers are just wrong, or nobody can say how they were made. The evidence:

• Notebooks do not rerun. Of 10,388 Jupyter notebooks from biomedical papers whose dependencies installed, 1,203 ran without errors and 879 reproduced the original results.^[7]
• Spreadsheets corrupt data. Excel with default settings “is known to convert gene names to dates and floating-point numbers”; a scan of leading genomics journals found that “approximately one-fifth of papers with supplementary Excel gene lists contain erroneous gene name conversions”.^[8]
• Platforms differ. The Nextflow paper reported that three analyses produced different results on different execution platforms.^[9]
• Formats do not carry identity. AMP/CAP notes that FASTQ, BAM and VCF do not require sample, patient or run information, so without a robust mechanism results “may be assigned to the wrong patient and/or wrong sample”.^[10]

Before: a folder of scripts and one person’s memory. After: a manifest answers “which samples, which code, which genome build, at what cost” without asking anyone.

• Not LabOps. Wet lab, instruments, LIMS and sample custody belong to LabOps. BioOps starts where data leaves the instrument and consumes LabOps’ sample metadata. Editorial distinction.
• Not ResearchDataOps. Repositories, DOIs, FAIR metadata and preservation belong to ResearchDataOps. BioOps ends where data is published and hands over its provenance. Editorial distinction.
• Not bioprocess operations. Running bioreactors and manufacturing processes is process engineering; the biops simulator shares a name, not a field.^[5]
• Not DevOps with a lab coat. Version control, CI and staged releases transfer and are documented for bioinformatics pipelines,^[11] but correctness here is statistical, inputs are terabytes, and clinical or GxP changes are regulated. See DevOps for the general practice.
• Not a product. Managed services can run pipelines; they do not decide what to pin, validate or sign off.

There is no BioOps movement; the practices grew inside bioinformatics. Shell scripts on shared clusters gave way to workflow managers: Snakemake, and Nextflow, whose 2017 paper showed “three bioinformatics data analyses produce different results when executed on different execution platforms” and argued that a workflow manager plus containers could control this.^[9] Engine-independent languages followed: CWL, an open standard,^[12] and the openly governed WDL.^[13] Packaged software made pins practical: Bioconda reported “over 3,000 Conda software packages” in 2018,^[14] and BioContainers packages tools as containers.^[15] Community pipelines followed with nf-core (started 2018,^[16] described in a 2020 paper^[17]); on 2026-10-23 its pipelines.json listed 159 pipeline repositories, 106 of them released and not archived (our count).^[18] Standard APIs and provenance came from GA4GH (WES, TES) and Workflow Run RO-Crate,^[19][20] and regulated use from the AMP/CAP validation guideline (2018).^[10]

Mental model: five pins and two gates. A result is reproducible when five things are pinned: pipeline release (code revision plus engine version), software environment (container digests), reference bundle, parameters (including any random seeds, Sandve’s Rule 6^[1]), and inputs (samplesheet plus file checksums). It is trustworthy when it passes two gates: an input gate before the run (files exist, checksums match, samplesheet is consistent) and an identity and quality gate after it (genotype or sex checks, QC thresholds, benchmark accuracy where applicable).

flowchart TB
  L["LabOps: samples, LIMS"]
  subgraph Core["BioOps"]
    B1["Input gate"]
    B2["Pipeline run"]
    B3["Identity, QC, provenance"]
    B1 --> B2 --> B3
  end
  S["WorkflowOps, ContainerOps"]
  R["ResearchDataOps: publish"]
  H["HealthOps: report"]
  L -->|"data leaves instrument"| B1
  S -.->|"support"| B2
  B3 -->|"data is published"| R
  B3 -->|"clinical result"| H
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  class B1,B2,B3 core
  class L,R,H,S neutral

Pipelines that feed business analytics belong to DataOps; model training on omics data to MLOps; organisation-wide cloud cost to FinOps. ResearchOps is UX research and unrelated.

No published BioOps charter was found. This is this Atlas’s proposal, built on the ten rules^[1] and on the Snakemake authors’ argument that reproducibility “is by no means sufficient” without “automation, scalability, portability, readability, traceability, and documentation”.^[26]

Mission. Every analysis result can be regenerated, explained and attributed to the right sample, at a cost the group can predict.

Principles (proposed):

1. Pin all five. Pipeline release (with engine version), container digests, reference bundle, parameters and input checksums are recorded for every run. A tag such as latest is not a pin.
2. Identity is a first-class check. Sample identity is verified by data (genotype, sex, expected variants), not only by file names.
3. No manual steps between raw data and result. Sandve’s Rule 2 is “Avoid Manual Data Manipulation Steps”.^[1] If a human must decide, record the decision as an input.
4. Tests before trust. Pipelines have automated tests on small data and are benchmarked on truth sets where they exist.
5. Change is versioned and, where required, validated. Every release has notes; in regulated work, significant changes trigger supplemental validation.^[10]
6. Storage is designed, not accumulated. Every retained file has a reason and an owner; intermediates have a deletion date.
7. Provenance travels with the data to whoever publishes or reports it.

Non-goals. BioOps does not choose the scientific method, interpret variants clinically, or replace the laboratory’s quality system.

Small groups combine roles, but each decision has one accountable person.

In clinical laboratories, AMP/CAP expects a qualified medical professional to oversee validation and the laboratory director to approve each pipeline component.^[10]

A pipeline’s lifecycle is a software release cycle plus validation against biological truth.

flowchart TB
  D["Develop on a branch"] --> T["Test: unit and small data"]
  T --> V["Validate: truth sets, real samples"]
  V --> R["Release: version and notes"]
  R --> P["Run in production"]
  P -->|"bug, new tool, new build"| C["Change request"]
  C -->|"back to development"| D
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  class D,T,R,P core
  class V,C decision

• Day 0 — design: intended use (research, service, clinical test, GxP), data types, volumes, genome builds, acceptance criteria.
• Day 1 — first release: tests, pinned images, frozen reference bundle, benchmark, manifest format, a first batch with extra review.
• Day 2 — operate: gates, triage, cost review, releases on a cadence. The biogitflow protocol uses development, “validation for pre-production” and production environments, with a hotfix path.^[11]

• Intake. Batches arrive with a samplesheet and checksums; gate failures go back to the provider, not into the pipeline.
• Release cadence. Service and clinical groups batch changes into planned releases because each costs validation effort. Nextflow ships stable releases in April and October and monthly edge releases,^[27] so plan engine upgrades.
• Change classes (proposed). Patch: no change to outputs (documentation, logging) — tests only. Minor: outputs may change within tolerance (tool patch, resource settings) — tests plus benchmark comparison. Major: method change (new caller, new genome build) — full revalidation, and in clinical work the supplemental validation the guideline requires “whenever a significant change is made”.^[10]
• Review rhythm. Weekly: failed runs and gate failures. Monthly: cost per sample, storage growth, open defects. Per release: benchmark deltas and sign-off.
• Escalation. Any suspected wrong-sample or wrong-result event stops release of affected results and follows the sample-swap runbook.
• Live operations. Running cases need a named owner. Genomics England’s Bio-Ops team, for example, is “the first port of call for issues affecting live cases”, gives second-line support and maintains pipeline-monitoring tools.^[2]

The architecture separates what is computed (pipeline code and pinned environments), where it runs (executors), and what is remembered (provenance and retained data). That separation is what lets the same release run on a laptop for tests, on a Slurm cluster for routine batches, and in a cloud batch service for bursts.

flowchart TB
  subgraph Inputs["Inputs, read-only"]
    I1["Raw reads"]
    I2["Samplesheet, checksums"]
    I3["Reference bundle"]
  end
  subgraph Control["Control"]
    G["Input gate"]
    E["Pinned engine"]
    Reg["Images by digest"]
  end
  subgraph Exec["Execution"]
    X1["HPC: Slurm, Apptainer"]
    X2["Cloud batch, spot"]
  end
  subgraph Keep["Keep"]
    WD["Work dir: expires"]
    O["Results"]
    P["Provenance"]
  end
  I1 --> G
  I2 --> G
  G --> E
  I3 --> E
  Reg --> E
  E --> X1
  E --> X2
  X1 --> WD
  X2 --> WD
  WD --> O
  G --> P
  E --> P
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  class G,E,P core
  class I1,I2,I3,Reg,X1,X2,O neutral
  class WD risk

Design choices and their trade-offs:

A routine batch, from delivery to release, with the two exception paths (reject, hold).

sequenceDiagram
  participant Lab as Lab or provider
  participant Own as Pipeline owner
  participant Eng as Workflow engine
  participant Cmp as Compute
  Lab->>Own: reads, sheet, checksums
  Own->>Own: input gate
  Own-->>Lab: reject if checks fail
  Own->>Eng: start pinned release
  Eng->>Cmp: submit tasks
  Cmp-->>Eng: exit codes, outputs
  Eng->>Eng: retry or resume
  Eng->>Own: QC, identity, manifest
  Own-->>Lab: hold if identity fails
  Own->>Own: release results

After release, results and manifest go to publication (ResearchDataOps) or reporting, and the work directory expires.

1. Engine. Require a minimum version (manifest.nextflowVersion = '!>=25.10.4', strict) and pin the exact one per run with NXF_VER.^[27]
2. Code. Run a tagged release (-r 1.2.0), never a branch; record the commit ID.
3. Software. Resolve each image tag to a digest at release time and write the digest into the config (on 2026-10-23 Quay listed samtools:1.24--h9dcdb79_1 as sha256:a1304475…).^[30] Tags can be rebuilt; digests cannot change content.
4. Reference. Store the bundle read-only, with a checksum file, and a bundle ID such as GRCh38-noalt-gencode49-v3.
5. Parameters. Keep them in a versioned params file; nf-core asks for parameters via the command line or -params-file, not config files.^[31]
6. Record. Write all five pins (engine version with the release) into the run manifest.

Nextflow reuses a cached task on -resume when its hash (inputs, script, container and more) matches and its outputs are still in the work directory; an input file is hashed by “full path, last modified timestamp, and file size”.^[24] Consequences:

• Keep both halves. Resume needs the cache (.nextflow/cache in the launch directory) and the work directory; on cloud VMs use the documented cloud cache in object storage.^[24]
• Shared file systems can lie about time. NFS “may report inconsistent file timestamps”; the lenient cache mode ignores the timestamp.^[24]
• Code can make resume wrong, not just slow. “If a process modifies its input files, it cannot be resumed”, and a process that receives a BAM channel and an index channel merged without matching on ID gets “incorrect and non-deterministic” inputs; the fix is to join on the sample ID.^[24] That is a correctness bug: a BAM paired with another sample’s index.

Snakemake decides re-runs with --rerun-triggers; by default code, input, modification time, parameters and software environment all trigger a re-run, which “guarantees that results are consistent with the workflow code and configuration”.^[32]

Retry only failures that a retry can fix. Nextflow’s documented pattern raises memory and time on each attempt and retries only exit codes 137–140, terminating on anything else.^[33] Codes above 128 usually mean a signal kill (128 + signal number^[34]): 137 is SIGKILL, for example an out-of-memory kill or a scheduler limit; 139 is a segmentation fault, which more memory rarely fixes:

process ALIGN {
    memory { 8.GB * task.attempt }
    time   { 4.h * task.attempt }
    errorStrategy { task.exitStatus in 137..140 ? 'retry' : 'terminate' }
    maxRetries 2
    // ...
}

Blanket retries turn a corrupt input into three failed attempts and a bigger bill. Spot reclaims on AWS Batch are separate: aws.batch.maxSpotAttempts controls them (default 0 since Nextflow 24.10).^[35] AWS gives two minutes’ notice,^[36] so short, resumable tasks matter more than clever retry rules.

Separate what is mandatory in your context from what is recommended:

Regulations for GxP computing (21 CFR Part 11, EU GMP Annex 11) and GAMP 5 are covered in quality, security, privacy and compliance.

Named tools are examples verified to exist on 2026-10-23, not endorsements.

Automate the deterministic parts fully: gates, pinning, tests, benchmark comparisons, clean-up, cost reports. Keep humans on release decisions for clinical and GxP results and on changes beyond agreed tolerances. Code drafted by language models must pass the same tests, and models should not choose parameters, genome builds or QC thresholds unreviewed. For GMP, the EU consulted on a draft Annex 22 on artificial intelligence until 7 October 2025.^[48]

Assumptions. A small bioinformatics group runs germline short-read pipelines for a few thousand research samples a year on a Slurm cluster with Apptainer, bursting to cloud batch. Nextflow is the engine (Snakemake works the same way). Versions as of October 2026.

Repository layout.

germline-pipeline/
  main.nf, modules/, subworkflows/   # pipeline code (or nf-core pipeline + config)
  conf/site.config                   # executors, pinned images, trace, lineage
  conf/refs/GRCh38-noalt-v3.json     # reference bundle manifest (paths + md5)
  params/batch-template.yaml         # parameters, versioned
  tests/                             # nf-test tests and snapshots
  benchmarks/                        # truth-set comparison scripts and past results
  docs/RELEASE_NOTES.md, docs/validation/
  .github/workflows/test.yml         # CI

Site configuration. One config file holds everything that varies by site, so the pipeline code stays portable.

Example status: written against the Nextflow 26.10 configuration reference; not executed (Nextflow is not installed on the machine that built this page). Queue names, paths and the region are placeholders.

// conf/site.config
manifest {
    nextflowVersion = '!>=25.10.4'          // refuse older engines
}
process {
    // Retries help only if memory/time scale with task.attempt (nf-core base configs do)
    errorStrategy = { task.exitStatus in 137..140 ? 'retry' : 'terminate' }
    maxRetries    = 2
    withName: 'SAMTOOLS_.*' {
        // samtools:1.24--h9dcdb79_1; digest only (some runtimes reject tag+digest)
        container = 'quay.io/biocontainers/samtools' +
            '@sha256:a130447589651ed09252aa95a5e4f4132942cdb54d835d81a04a9a930d656561'
    }
}
profiles {
    hpc {
        process.executor   = 'slurm'
        process.queue      = 'standard'          // placeholder
        apptainer.enabled  = true
        apptainer.cacheDir = '/shared/apptainer-cache'
    }
    cloud {
        // Needs a Batch compute environment + job queue; every process needs a container
        process.executor = 'awsbatch'
        process.queue    = 'genomics-spot'       // placeholder
        workDir          = 's3://<bucket>/work'  // must be S3, with a subdirectory
        aws.region       = 'eu-west-1'           // placeholder
        aws.batch.maxSpotAttempts = 2            // default is 0 since 24.10
    }
}
trace {
    enabled = true                               // default name trace-<timestamp>.txt
    fields  = 'task_id,hash,name,status,exit,attempt,' +
              'cpus,%cpu,memory,peak_rss,realtime,container'
}
report.enabled   = true
timeline.enabled = true
lineage.enabled  = true                          // experimental since 25.04

The digest is the pin. On AWS Batch, Nextflow’s setup guide needs a compute environment, a job queue bound to it, an S3 work directory and a container for every process, and its setup attaches AmazonS3FullAccess to the instance role;^[49] scope that role to the batch’s input, work and output prefixes instead (see security). The trace fields add cpus, memory, attempt and container so efficiency and software can be computed per task; %cpu is “relative to a single CPU” (four busy CPUs report about 400%).^[50] Lineage records the run (repository, commit, parameters, resolved config), tasks and output checksums under lid:// identifiers, but “is an experimental feature added in Nextflow 25.04”,^[51] so the run manifest below stays the durable record.

Parameters file.

Example status: YAML syntax-checked; parameter names other than input and outdir are illustrative.

# params/batch-2026-10-B07.yaml
input: /data/batches/2026-10-B07/samplesheet.csv
outdir: /results/2026-10-B07
genome_bundle: GRCh38-noalt-v3        # resolved via conf/refs/GRCh38-noalt-v3.json
batch_id: 2026-10-B07

Continuous integration. Every change runs the pipeline’s tests with the pinned engine.

Example status: YAML syntax-checked; not executed. Action versions were the latest releases on GitHub on 2026-10-23; pin them to commit SHAs in production.

name: pipeline-tests
on:
  pull_request:
  push:
    branches: [main]
permissions:
  contents: read
jobs:
  nf-test:
    runs-on: ubuntu-24.04
    steps:
      - uses: actions/checkout@v7
      - uses: nf-core/setup-nextflow@v3
        with:
          version: "26.10.3"
      - uses: nf-core/setup-nf-test@v2
        with:
          version: "0.9.5"
      - name: Run tests with the test profile
        run: nf-test test --profile test,docker --verbose

Run manifest. Written by the input gate (Lab 2) and completed after the run.

Example status: JSON syntax-checked; values illustrative.

{
  "batch_id": "2026-10-B07",
  "pipeline": "germline-pipeline",
  "revision": "2.3.1",
  "commit": "<40-character commit id>",
  "engine": "nextflow 26.10.3",
  "reference_bundle": "GRCh38-noalt-v3",
  "params_file_sha256": "<sha256>",
  "samplesheet_sha256": "<sha256>",
  "images": {
    "samtools": "sha256:a130447589651ed09252aa95a5e4f4132942cdb54d835d81a04a9a930d656561"
  },
  "data_access_approval": "<consent or data-access approval id>",
  "identity_check": "passed",
  "released_by": "<role mailbox>",
  "released_at_utc": "2026-10-23T14:00:00Z"
}

The same idea in Snakemake. Snakemake rules can declare a container: or conda: environment, and the run chooses how to deploy them with --software-deployment-method (--sdm), for example --sdm apptainer on a cluster.^[52]

Example status: illustrative Snakefile; not executed and not machine-checked.

rule flagstat:
    input:  "results/{sample}.cram"
    output: "results/{sample}.flagstat.txt"
    # samtools:1.24--h9dcdb79_1, pinned by digest
    container: ("docker://quay.io/biocontainers/samtools"
        "@sha256:a130447589651ed09252aa95a5e4f4132942cdb54d835d81a04a9a930d656561")
    shell:  "samtools flagstat {input} > {output}"

Three labs, from running a community pipeline to measuring accuracy. Lab 2, the launcher download and checksum lines of Lab 1 and the download steps of Lab 3 were executed for this page; the rest is written from the current documentation and says so.

Objective: see caching, resume and per-task metrics on a real nf-core pipeline.

Prerequisites: Linux or macOS; Bash 3.2+ and Java 17–26;^[53] Docker; about 2 GB free disk; internet access. nf-core/demo 1.2.0 requires Nextflow 25.10.4 or later and runs FastQC, seqtk trimming and MultiQC on small test data.^[31] Cost and risk: none beyond downloads; runs locally.

Status: not executed for this page (Java and Docker are not available on the machine that built it), except the launcher download and checksum lines, run on 2026-10-23. Commands follow the Nextflow and nf-core documentation as of October 2026.

mkdir -p ~/bioops-lab1 && cd ~/bioops-lab1
# pinned launcher (not `curl ... | bash`); SHA-256 from the GitHub release
curl -fsSLo nextflow \
  https://github.com/nextflow-io/nextflow/releases/download/v26.10.3/nextflow
SHA=5f9b4069c4eb62104442e0a64d01a7b6557146008c6b180bf78a995605540896
# macOS has no sha256sum: use `shasum -a 256 -c -` instead
echo "$SHA  nextflow" | sha256sum -c - && chmod +x nextflow  # "nextflow: OK"
export NXF_VER=26.10.3                          # pin the engine for this shell
./nextflow run nf-core/demo -r 1.2.0 -profile test,docker --outdir results
./nextflow run nf-core/demo -r 1.2.0 -profile test,docker --outdir results -resume
./nextflow log                                  # list runs, session IDs, status
./nextflow log last -f name,hash,exit,attempt,realtime,%cpu,peak_rss
ls results/pipeline_info/

Expected output: the first run completes with status OK in nextflow log; results/ contains FastQC and MultiQC output and results/pipeline_info/ holds execution_trace_*.txt, an execution report, a timeline and a DAG.^[31] The second run reuses every task from the cache and finishes much faster.^[24]

Validation: the second run reports every process as cached; every task shows exit 0. Compare %cpu with requested CPUs.

Troubleshooting: nextflow: FAILED from the checksum → && skips chmod, so the launcher cannot run; delete it and download again. sha256sum: command not found on macOS → shasum -a 256 -c -. Java errors → install Java 17+. Docker permission errors → -profile test,apptainer. Resume re-runs everything → see making resume work.

Teardown: rm -rf ~/bioops-lab1 (also removes work/, .nextflow/ and results). Images stay in Docker’s cache: docker image ls and remove them if wanted.

Objective: stop a batch with a corrupted file or a reused FASTQ before any compute runs, and write the run manifest.

Prerequisites: Python 3.8+ (standard library only), gzip, md5sum (GNU coreutils). Cost and risk: none; synthetic data.

Status: executed on 2026-10-23 with Python 3.8.10 on synthetic files (version 1.1.0 after review); output below is abridged from that run.

The gate allows a repeated sample ID, because nf-core samplesheets repeat it on purpose for each lane or re-sequencing run;^[54] what it refuses is a file used on two rows, which is how a copy-paste error usually shows.

Save as prov_check.py:

"""Pre-run gate: check a samplesheet's FASTQ files and write a run manifest.

Usage: python3 prov_check.py samplesheet.csv --pipeline NAME --revision REV
Samplesheet columns: sample,fastq_1,fastq_2,md5_1,md5_2 (fastq_2/md5_2 may be empty).
Exit code 0 = all checks passed; 2 = at least one problem (nothing should run).
"""
import argparse
import csv
import datetime
import hashlib
import json
import os
import sys

CHECKER_VERSION = "1.1.0"


def digest(path, algo):
    h = hashlib.new(algo)
    with open(path, "rb") as fh:
        for block in iter(lambda: fh.read(1 << 20), b""):
            h.update(block)
    return h.hexdigest()


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("samplesheet")
    ap.add_argument("--pipeline", required=True)
    ap.add_argument("--revision", required=True)
    ap.add_argument("--out", default="run-manifest.json")
    args = ap.parse_args()

    problems, files, seen_samples, seen_paths = [], [], set(), {}
    # A sample ID may repeat (one row per lane or re-sequencing run);
    # a file may not: each path is allowed on one row only.
    with open(args.samplesheet, newline="") as fh:
        rows = list(csv.DictReader(fh))
    for n, row in enumerate(rows, start=2):  # line 1 is the header
        sample = (row.get("sample") or "").strip()
        if not sample:
            problems.append(f"line {n}: empty sample id")
            continue
        seen_samples.add(sample)
        pairs = [("fastq_1", "md5_1"), ("fastq_2", "md5_2")]
        if row.get("fastq_1") and row.get("fastq_1") == row.get("fastq_2"):
            problems.append(f"line {n}: {sample} uses the same file for R1 and R2")
        for col, md5col in pairs:
            path = (row.get(col) or "").strip()
            if not path:
                continue
            real = os.path.realpath(path)
            prev = seen_paths.setdefault(real, (n, sample))
            if prev[0] != n:
                problems.append(
                    f"line {n}: {path} already used on line {prev[0]} ({prev[1]})")
            if not os.path.isfile(path):
                problems.append(f"line {n}: missing file {path}")
                continue
            got = digest(path, "md5")
            want = (row.get(md5col) or "").strip().lower()
            if want and got != want:
                problems.append(f"line {n}: md5 mismatch {path} ({got} != {want})")
            if not want:
                problems.append(f"line {n}: no expected md5 for {path}")
            files.append({"sample": sample, "path": path, "md5": got,
                          "bytes": os.path.getsize(path)})

    manifest = {
        "checker_version": CHECKER_VERSION,
        "checked_at_utc": datetime.datetime.now(datetime.timezone.utc)
        .strftime("%Y-%m-%dT%H:%M:%SZ"),
        "pipeline": args.pipeline,
        "revision": args.revision,
        "samplesheet_sha256": digest(args.samplesheet, "sha256"),
        "samples": len(seen_samples),
        "files": files,
        "problems": problems,
    }
    with open(args.out, "w") as fh:
        json.dump(manifest, fh, indent=2)
    for p in problems:
        print("PROBLEM", p)
    print(f"{len(seen_samples)} samples, {len(files)} files, "
          f"{len(problems)} problems -> {args.out}")
    return 2 if problems else 0


if __name__ == "__main__":
    sys.exit(main())

Steps:

mkdir -p ~/bioops-lab2/fastq && cd ~/bioops-lab2      # save prov_check.py here
for s in S1 S2; do for r in 1 2; do
  printf "@r1\nACGT%s%s\n+\nIIIIII\n" "$s" "$r" | gzip -n > "fastq/${s}_R${r}.fastq.gz"
done; done
m() { md5sum "$1" | cut -d' ' -f1; }
{ echo "sample,fastq_1,fastq_2,md5_1,md5_2"
  for s in S1 S2; do
    r1="fastq/${s}_R1.fastq.gz"; r2="fastq/${s}_R2.fastq.gz"
    echo "$s,$r1,$r2,$(m "$r1"),$(m "$r2")"
  done; } > samplesheet.csv
python3 prov_check.py samplesheet.csv --pipeline nf-core/demo --revision 1.2.0
echo "exit=$?"
# now break it: a copy-paste row that reuses S1's file, and a changed file
cp samplesheet.csv bad.csv
echo "S3,fastq/S1_R1.fastq.gz,,$(m fastq/S1_R1.fastq.gz)," >> bad.csv
printf "@r1\nACGTXX\n+\nIIIIII\n" | gzip -n > fastq/S2_R2.fastq.gz
python3 prov_check.py bad.csv --pipeline nf-core/demo --revision 1.2.0 \
  --out bad-manifest.json; echo "exit=$?"

Expected output (abridged from the executed run; hashes depend on your files):

2 samples, 4 files, 0 problems -> run-manifest.json
exit=0
PROBLEM line 3: md5 mismatch fastq/S2_R2.fastq.gz (b762e80b… != 530cc53c…)
PROBLEM line 4: fastq/S1_R1.fastq.gz already used on line 2 (S1)
3 samples, 5 files, 2 problems -> bad-manifest.json
exit=2

Validation: exit code 0 for the clean sheet and 2 for the broken one; run-manifest.json contains the samplesheet SHA-256 and one entry per file. In a real pipeline wrapper, a non-zero exit must stop the launch.

Troubleshooting: md5sum: command not found on macOS → use md5 -q in the helper. Every file reported as mismatched → the provider’s checksums were computed on uncompressed data; ask which. Teardown: rm -rf ~/bioops-lab2.

Extend it: check BAM/CRAM contig names against the bundle and sample IDs against the LIMS export (LabOps).

Objective: replace “the pipeline finished” with measured recall and precision per variant type, as the GA4GH benchmarking framework recommends (match representations, standard metrics, stratification).^[42]

Prerequisites: your pipeline’s GRCh38 VCF for GIAB sample HG002; the GIAB HG002 v5.0q GRCh38 small-variant benchmark VCF and BED;^[55] the reference FASTA you called against; Apptainer or Docker; about 20 GB disk. Cost and risk: public HG002 data is listed by GIAB.^[25]

Which truth set. As of 2026-10-23 NIST marks the long-used v4.2.1 small-variant benchmark “deprecated for HG002 by v5.0q”.^[25] v5.0q is assembly-based, covers small variants and SVs, and its README calls it a draft whose results “should be carefully evaluated”; some regions in v4.2.1 are excluded from it.^[55] Numbers from the two versions are therefore not comparable: record the version, and do not switch mid-study without re-running the earlier comparisons.

Status: download and checksum steps executed on 2026-10-23; the comparison was not (no query VCF or container runtime here). File names and --gender male follow the v5.0q README; the hap.py image (built 2023-05-23, Python 2.7) was listed on Quay on 2026-10-23.^[30] The README recommends hap.py’s vcfeval engine, which needs RTG Tools in the image (bioconda lists it only as a build dependency), so the command uses the default xcmp engine.

GIAB=https://ftp-trace.ncbi.nlm.nih.gov/ReferenceSamples/giab/release
GIAB=$GIAB/AshkenazimTrio/HG002_NA24385_son/v5.0q
T=HG002_GRCh38_v5.0q_smvar
# hap.py:0.3.15--py27hcb73b3d_0, pinned by digest
IMG=docker://quay.io/biocontainers/hap.py
IMG=$IMG@sha256:d63b963a6cb01b4830393b22369e7b91d298e4156dde353739e74e4cfa4f96d0
mkdir -p ~/bioops-lab3 && cd ~/bioops-lab3
for f in "$T.vcf.gz" "$T.vcf.gz.tbi" "$T.benchmark.bed" checksum.md5; do
  curl -sfO "$GIAB/$f"
done
grep -E "$T\.(vcf\.gz|benchmark\.bed)$" checksum.md5 | md5sum -c -   # expect 2 x OK
# query.vcf.gz = your HG002 calls; ref.fa (+ .fai) = the reference you called against
apptainer exec "$IMG" hap.py "$T.vcf.gz" query.vcf.gz -f "$T.benchmark.bed" \
  -r ref.fa -o hg002_vs_giab --pass-only --gender male
column -s, -t < hg002_vs_giab.summary.csv | cut -c1-120

Expected output: hg002_vs_giab.summary.csv with rows for SNP and INDEL, and columns including TRUTH.TP, TRUTH.FN, QUERY.FP, METRIC.Recall, METRIC.Precision and METRIC.F1_Score.^[56] The metrics section shows how to read them.

Validation: both downloads report OK against GIAB’s checksums; the benchmark uses chr names for chromosomes 1–22, X and Y, so your query must too; rerunning gives the same numbers. Record the truth-set version, BED, reference, image digest and options next to the result.

Troubleshooting: almost no matches → contig names or build differ;^[39] fix the query, never the truth set. Low indel recall → check normalisation and indel filters.

Teardown: rm -rf ~/bioops-lab3; delete the cached image from Apptainer’s cache (apptainer cache clean) if wanted.

Problem. A variant-calling pipeline had a large storage footprint, partly from “Nextflow’s requirement for a work directory with intermediate results to facilitate resuming”.^[57] Method. Version 3 switched internal files after duplicate marking from BAM to CRAM and increased parallelism within each sample. Result reported by the authors, benchmarking five tumour–normal pairs: the pre-processing work directory dropped by 65%, from 170.4 TB to 59.7 TB; average CPU hours fell from 3,761.07 to 3,252.37; memory use rose by 34%. Together with the runtime changes, the abstract reports “a 70% cost reduction in commercial clouds” (3.1.1 against 2.7.x, AWS Batch spot); other sections give 50–70% and name 2.7.1 or 2.7.2 as the baseline, so treat 70% as the upper end.^[57] Trade-off. CRAM saves storage but costs memory and ties every file to its exact reference. Success criteria: storage per sample down, benchmark unchanged, reference archived by MD5. Pitfall: a “same” reference with different contigs leaves archived CRAMs undecodable.^[21]

Context. A US clinical laboratory offers an NGS test; the pipeline is part of the test. Method. Under the AMP/CAP guideline the laboratory validates its own pipeline (recommendation 1) after design and optimisation (3), in an environment that emulates real use (4); preserves sample identity “with a minimum of four unique identifiers” (sample, patient, run, laboratory location) in each file (10); protects file integrity (13); uses in-silico validation only to supplement end-to-end validation with human samples (14); and revalidates after significant changes (17).^[10] Trade-off. Every release now costs validation work, so releases become rarer and larger. Success criteria: zero results released without passing identity checks; validation report per release; time from approved change to production. Pitfall: treating a container rebuild as “no change”; the tool version inside may have moved.

Measure correctness first, then speed and cost.

The accuracy KPI applies to variant calling. Other assays need their own reference material, such as ERCC RNA spike-ins, used “to measure sensitivity, accuracy, and biases in RNA-seq experiments”.^[60] Set targets from your own baseline. An accuracy figure means nothing without its regions: SNV concordance of two methods was “99.7% inside versus 76.5% outside high-confidence regions”.^[42]

Quality. Small-data tests catch code regressions, benchmarks scientific ones, identity checks swaps, and a contamination estimate mixed samples.^[29] NGSCheckMate compares allele read fractions at known SNPs from FASTQ, BAM or VCF above 0.5×.^[46] Keep thresholds and their rationale in the validation documents.

Security. The compute role should read the batch’s inputs and reference bundle and write its work and output prefixes, nothing else: replace setup-guide policies such as AmazonS3FullAccess with prefix-scoped ones.^[49] Update pinned images on a schedule (ContainerOps). On shared clusters, Apptainer users “cannot gain additional privilege on the host system by default”.^[28] Encrypt sensitive files at rest; crypt4gh wraps genomic files in an encryption layer.^[37]

Privacy (jurisdiction-specific, not legal advice). In the EU, genetic data is listed as sensitive personal data “subject to specific processing conditions” (GDPR Articles 4(13) and 9).^[61] Record in each manifest the consent or data-access approval the batch was processed under. The privacy programme belongs to PrivacyOps; data-platform access policies to DataSecOps.

Regulated computing (GxP and clinical). If results feed patient care or regulated manufacturing, the pipeline is part of a validated test or computerised system:

• US FDA-regulated records: 21 CFR Part 11 requires validation “to ensure accuracy, reliability, consistent intended performance, and the ability to discern invalid or altered records”, and “secure, computer-generated, time-stamped audit trails” (§11.10, eCFR current to 2026-10-21).^[62]
• EU GMP: Annex 11 (revision January 2011) applies risk management “throughout the lifecycle of the computerised system”, requires validation and qualified infrastructure, and controlled change.^[58] A revised Annex 11 and a new Annex 22 (AI) were consulted on until 7 October 2025; on 2026-10-23 the EudraLex page still listed the 2011 text.^[48]
• Industry guidance: GAMP 5 (second edition, July 2022) describes a risk-based approach and stresses “critical thinking” by subject-matter experts.^[63]
• US clinical laboratories: the AMP/CAP recommendations in case 2.^[10]

What BioOps contributes is evidence: release records, validation reports, run manifests, audit trails of who ran and approved what. The compliance programme itself belongs to ComplianceOps.

Drivers are compute, storage, transfer and people time spent on failed runs.

• Storage first. Work directories hold every intermediate and are often the biggest line. Expire them after release and consider CRAM (case 1).^[57]
• Right-size requests per process from the trace, with retry-on-memory for outliers.^[33]
• Spot capacity suits short, resumable tasks; long tasks lose work on each interruption.^[36]
• Data gravity. Move compute to large or legally constrained data; WES aims to bring “algorithms to genomic data”.^[19]
• Exit costs. Managed-service availability changes in both directions. AWS closed HealthOmics variant and annotation stores to new customers “starting November 7th, 2025”.^[64] Its bioinformatics CI/CD guidance relies on CodeCommit, which AWS closed to new customers in July 2024 and returned to general availability on 24 November 2025;^[65] on 2026-10-23 the guidance page still said CodeCommit “is no longer available to new customers”.^[66] Keep pipelines in a portable engine and data in open formats.

The sarek authors present their savings as lowering both cost and CO2 emissions;^[57] see GreenOps for carbon accounting and FinOps for organisation-wide cloud cost.

Four scheduled checks show whether the operating model works:

1. Monthly re-run of one archived batch from its manifest alone, by someone else. Differences are defects.
2. Manifest audit: can ten sampled results each be traced to sheet line, checksums, release, images and bundle in 15 minutes?
3. Gate drill: feed the gate a known-bad batch (like Lab 2) after every gate change.
4. Retention audit: stored bytes without an owner and deletion date are findings.

Trigger. The post-run identity check shows that sample P-0142’s genotype matches sample P-0147 from the same batch, and P-0142’s reported sex disagrees with the LIMS record. Nothing from the batch has been released. (Variant: a swap found after release, for example when a later sample from the same person disagrees; then step 5 applies to everyone who received affected results.)

flowchart TB
  A["Identity check fails"] --> B["Hold all results of the batch"]
  B --> C{"Swap inside the pipeline?"}
  C -->|"samplesheet or code"| D["Fix sheet, re-run affected samples"]
  C -->|"before data arrived"| E["Lab investigates tubes and plates"]
  E --> F{"Cause found?"}
  F -->|"yes"| D
  F -->|"no"| G["Withdraw samples, request re-sequencing"]
  D --> H["Identity re-check passes"]
  H --> I["Release; notify if already released"]
  G --> K["Tell requesters; notify if already released"]
  K --> J["Post-incident review"]
  I --> J
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  class A,B,G risk
  class C,F decision
  class D,E,H,I,J,K core

1. Contain (first hour). Hold the whole batch, not just two samples: swaps are often shifted rows. Record the hold.
2. Locate. Compare the samplesheet with the LIMS export (LabOps); check the gate manifest for duplicated paths; run the identity tool across the batch and against earlier data from the same individuals.
3. Decide where. Sheet or pipeline error → correct and re-run from raw data (never rename outputs). Lab error → the lab investigates; if unresolved, withdraw the samples.
4. Verify. Identity checks must pass for every sample in the batch.
5. Notify. Before release: tell the requesters about the delay or withdrawal. After release: notify everyone who received affected results; in clinical settings the laboratory director decides reporting.
6. Prevent. Review within a week; add the pattern to the input gate; track actions (IncidentOps).

• “It’s in a container, so it’s reproducible.” Containers pin software, not reference, parameters, inputs or engine.^[9]
• latest tags and branch runs in production. Results change with no code change.
• Notebooks as pipelines. Fine for exploration, but they rarely rerun;^[7] promote stable analysis to the pipeline.
• Keeping every work directory “just in case”. Storage grows without an owner; keep manifests and final outputs instead.
• Myth: “validation is only for clinical labs.” Research groups gain from a light version: a benchmark per release and a written intended use.
• Myth: “a managed platform means no operations.” Someone still owns pins, gates, costs and exit plans (NoOps).

Use the full practice when analyses are shared or repeated (a core facility, a consortium, a service), when results inform publications others build on, when data volumes make re-runs expensive, or when results enter clinical or GxP decisions.

Use a light version (pinned environment, Git, a run log) for one-off exploratory analyses by one person.

Do not over-build when a project is a short pilot whose results will be regenerated anyway, or when the team has no capacity to own a pipeline; a well-maintained community pipeline run at a pinned release is often better than an in-house one. Hidden costs of the full practice: validation effort per release, release discipline that slows experimentation, and storage of provenance.

flowchart TB
  Q1{"Results used in patient care or GMP?"}
  Q1 -->|"yes"| R1["Validated pipeline, change control, audit trails"]
  Q1 -->|"no"| Q2{"Shared, repeated, or published?"}
  Q2 -->|"no"| R3["Light: pinned env, Git, run log"]
  Q2 -->|"yes"| Q3{"Community pipeline fits?"}
  Q3 -->|"yes"| R2a["Run it pinned, local config, gates"]
  Q3 -->|"no"| R2b["Own pipeline: tests, benchmark, releases"]
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  class Q1,Q2,Q3 decision
  class R1 risk
  class R2a,R2b core
  class R3 neutral

This Atlas’s model, not an industry standard. A published, broader five-level model for scientific operations, SciOps, also derives from DevOps;^[67] the levels below are narrower and specific to pipelines.

Level 4 is required only where regulation demands it; level 3 is a sensible target for shared research services.

Quick win: digest pinning and the input gate are cheap and catch silent failures in the first month.

• Tagged release with notes; engine version pinned
• Every image pinned by digest; digests listed in the release record
• Reference bundle ID and checksum file referenced
• CI tests pass on the release commit
• Benchmark run on the release; deltas against the previous release reviewed and signed
• Change class decided (patch, minor, major); validation done where required

• Input gate passed (checksums, duplicates, contig names); manifest started
• Release and profile recorded; run launched with -r <tag> and pinned engine
• Identity and QC gates passed for every sample
• Results and manifest stored together; work directory expiry date set

Pipeline / release:        germline-pipeline 2.3.1   (commit: <id>)
Change class:              minor  (tool patch: samtools 1.23 -> 1.24)
Engine:                    nextflow 26.10.3          Reference bundle: GRCh38-noalt-v3
Images (digests):          see release-images.json
Tests:                     CI run <link>, all passed
Benchmark (HG002 v5.0q):   SNV F1 0.9965 (prev 0.9964); indel F1 0.9601 (prev 0.9598)
Intended use unchanged:    yes
Approved by / date:        <scientific lead or QA>, <YYYY-MM-DD>

Requested change:   what and why (bug, new tool version, new genome build)
Affected outputs:   files and fields that may change
Risk:               low / medium / high, with reason
Evidence required:  tests only | tests + benchmark | full revalidation
Rollback:           previous release tag; results re-run needed?
Decision:           approved / rejected, by whom, date

Criteria with evidence only; versions as of October 2026.

Choose by team language and the community pipelines you will use.

Is BioOps a job title? Not a common one. Genomics England has a “Bio-Ops” team with the role “Operations Bioinformatician”;^[2] elsewhere bioinformaticians, research software engineers and research-computing staff do this work.

Is a Conda environment file enough? Only if it pins exact builds; images pinned by digest are easier to prove.

Why pin the reference bundle when the genome build does not change? Because indexes, annotation releases, decoys and contig names do change within a build, and CRAM decoding needs the exact sequence.^[21]

Results differ slightly between HPC and cloud. Is that a bug? Not always; platform differences exist.^[9] Compare against a tolerance agreed in advance and investigate anything outside it.

Is our research pipeline “GxP”? Only if its results support regulated decisions. If they do, your quality unit decides the validation approach; this page is not regulatory advice.

• Queryable provenance (in development): Nextflow lineage and Workflow Run RO-Crate;^[51][20] expect format changes.
• Federated analysis (standards exist, adoption uneven): WES and TES run one workflow where the data is.^[41]
• Wider benchmarks (active research): GIAB’s assembly-based HG002 v5.0q covers small variants and SVs together, and its older SV benchmark is listed as deprecated.^[25]
• Citable pipelines (community principles): FAIR4RS identifiers and metadata for workflows.^[40]

• Prerequisites: Linux shell, Git, containers (ContainerOps).
• One day: run Lab 1; read the ten simple rules.^[1]
• One week: Lab 2 on a real batch; write a site config for your cluster; add one nf-test or Snakemake test.
• 30 days: Lab 3 with your pipeline; a release record; a cost-per-sample report.
• Specialise: clinical validation (AMP/CAP); GxP computerised systems (Annex 11, Part 11, GAMP 5); federated analysis (GA4GH).
• Portfolio project: take a published analysis, rebuild it as a pinned pipeline with a manifest, and show it reruns.

After this page you can explain the five pins and two gates, design an input gate, measure accuracy against a truth set, decide how much assurance an analysis needs and run a sample-swap response.

BioOps is a borrowed name for an old discipline: making computational biology answerable. Apart from one team name, nobody uses the label, and that is fine. What matters is that a result can be regenerated from its manifest, attributed to the right sample, and paid for knowingly. The tools are mature; the hard part is habit: pin everything, gate inputs, check identity, benchmark each release, delete what you do not need. Start with the analysis you run most often and make it rerun from its manifest.

All sources were opened on 2026-10-23; the evidence ledger is research/bioops-ledger.md.

Maintainer note. Values that drift first: engine and Action versions, the nf-core pipeline count, Quay digests, the Nextflow launcher digest, the GIAB v5.0q draft status and the AWS service banners.

Further reading

• WorkflowHub, a registry for computational workflows.

Changelog

• 2026-10-23 — First draft: term status confirmed as ambiguous; research ledger research/bioops-ledger.md; Lab 2 executed with Python 3.8.10; other examples syntax-checked only or not executed, as marked.
• 2026-10-23 — Revision after review r1 (reviews/bioops-r1-response.md): Genomics England “Bio-Ops” team disclosed; CodeCommit return to GA; Lab 3 moved to GIAB v5.0q (download and checksum steps run); AWS Batch profile completed; Lab 2 allows multi-lane samples (re-run); five pins made consistent; swap vs contamination separated; sarek, VCF and nf-core dates corrected.
• 2026-10-23, post-score polish: Workflow Run RO-Crate support credited to nf-prov (Nextflow) and the Snakemake report plugin; Lab 1 launcher pinned and checksum-verified (download step run); accuracy KPI scoped to variant calling with an RNA-seq spike-in pointer; maintainer note on fast-drifting values. After the polish check: Lab 1 checksum gates chmod with && and gives the macOS shasum -a 256 -c - form (both re-run); labs intro lists the Lab 1 lines that were run.