SEOps

SEOps, as this page uses the word, is technical SEO run as an engineering property: the rules that let search engines discover, crawl, render and index a site are written down, tested in the delivery pipeline and monitored in production. A release that adds noindex to every product page, points canonicals at the staging host or makes robots.txt return a server error can remove a site from search for weeks. SEOps treats that as a defect.

In one line: search visibility is a testable property of every release. Rankings are not; whether each page is reachable, crawlable, renderable, indexable and correctly canonicalised is.

For web and platform engineers, technical SEOs, and leaders who need search traffic to survive site changes.

Google Search works in three stages, crawling, indexing and serving, “and not all pages make it through each stage”.^[4] The controls that decide the first two live in code and server configuration (status codes, robots.txt, robots meta tags, canonicals, redirects, sitemaps, rendered HTML), which engineers change weekly for reasons unrelated to search.

The cost is asymmetric. Breaking indexability takes one deploy; recovery waits for recrawling, which Google says “may take months” for some pages.^[5] A robots.txt server error stops crawling for 12 hours;^[6] a misplaced noindex is quieter, “meaning there would be a slower drop in traffic”.^[7] Slow, quiet failures are what tests and monitors are for.

• Not ranking manipulation. SEOps keeps releases inside Google’s spam policies (cloaking, doorways, sneaky redirects and more).^[8]
• Not page-speed engineering. Core Web Vitals matter to search, but measuring and fixing them is WebOps work.
• Not the SEO programme. What to publish, for which searches, belongs to SEO Operations and ContentOps.
• Not Google-only. Many controls are open standards (RFC 9309, sitemaps, HTTP); Google-specific behaviour is labelled as such.

The label has no origin story; the practice grew out of technical SEO as sites became continuously delivered software. Milestones in what it relies on:

• Robots Exclusion Protocol: Martijn Koster, 1994; RFC 9309 since September 2022, whose rules “are not a form of access authorization”.^[9]
• Sitemaps: introduced in 2005; in June 2023 Google deprecated sitemap “ping” and stressed accurate lastmod.^[10]
• JavaScript rendering: Google calls dynamic rendering “a workaround and not a long-term solution” and recommends server-side rendering, static rendering or hydration.^[11]
• Byte limits: in March 2026 Google explained that bytes past an HTML URL’s first 2MB “aren’t fetched, they aren’t rendered, and they aren’t indexed”.^[12]

Mental model: four gates you control, one you do not. A URL is discovered, crawled, rendered and indexed before it can be served. Releases can break the first four; none can force the fifth.

flowchart TB
  A["Discover: links, sitemap"] --> B{"Crawl allowed and served?"}
  B -- "robots.txt disallow, 5xx, 429" --> X1["Not crawled"]
  B -- yes --> C{"Rendered HTML has content and links?"}
  C -- "JS-only links, content after 2MB" --> X2["Partly seen"]
  C -- yes --> D{"Indexable and canonical?"}
  D -- "noindex, soft 404, duplicate" --> X3["Not indexed"]
  D -- yes --> E["Eligible for serving"]
  E --> F["Ranking: not testable by a release"]
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  class A,E core
  class B,C,D decision
  class X1,X2,X3 risk
  class F neutral

flowchart TB
  SO["SEO Operations: topics"] --> CO["ContentOps: editorial"]
  CO --> SE["SEOps: indexable?"]
  WO["WebOps"] -- serving --> SE
  SE -- checks --> RO["ReleaseOps"]
  SE -- alerts --> OB["ObservabilityOps"]
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  class SE core
  class SO,CO,RO,WO,OB neutral

No official SEOps charter exists. This is this Atlas’s proposal:

Mission. Every release keeps the pages we want in search eligible for it, keeps the others out by the right mechanism, and tells us within one working day when Google’s view diverges from our intent. Non-goals. Choosing topics or keywords; promising rankings; manipulating search systems.

1. Write the intent down as a search contract per template.
2. Right control for the job. Crawling is controlled by robots.txt, indexing by noindex, duplication by redirects and canonicals; mixing them causes most failures.^[13][5][18]
3. Test what you ship, monitor what Google does. Only Search Console and crawler logs show Google’s behaviour.
4. Prefer server-side truth. JavaScript may fail to remove a noindex and should not change the canonical.^[19]
5. Fail closed for exclusions. Non-production hosts are never indexable (authentication, not robots rules).
6. Spam policies by design. Programmatic pages, redirects and bot-specific responses are reviewed before they ship.^[8]
7. Proportion. Google says well-linked sites of about 500 pages or fewer may not need a sitemap;^[20] they need a checklist, not a platform.

SEOps is rarely a team; it is a set of responsibilities.

R = responsible, A = accountable, C = consulted, I = informed; the SEO-programme team is consulted on contracts and migrations. The technical SEO owns what the contract says; engineering owns the code meeting it.

• Day 0 — design. URL scheme, rendering strategy (Google calls server-side or pre-rendering “still a great idea”^[19]), canonical policy, parameter handling, indexable templates; write the contracts.
• Day 1 — launch or migration. Generate robots.txt and sitemaps, run the full contract on authenticated staging, map every old URL, verify the Search Console property, submit sitemaps.
• Day 2 — every release. Contract checks in CI and after deploy; monitoring; monthly review of crawl waste, canonical disagreements and exclusions.

1. A change-review trigger. Changes to routing, rendering, robots.txt, redirects, canonicals, sitemaps, CDN headers or a template’s <head> need a technical-SEO reviewer (code-owners file).
2. A gate policy. Which failures block and which warn; the policy belongs to ReleaseOps, SEOps supplies the checks and severities (see the decision framework).
3. A weekly signal review (checklist).

Checks sit where each failure becomes visible: build, preview, production and Google’s reports.

flowchart TB
  subgraph PRE["Before release"]
    A["Pull request"] --> B["Unit: robots.txt and sitemap tests"]
    B --> C["Preview deploy, password-protected"]
    C --> D["Contract check: status, noindex, canonical"]
    D --> E["Rendered check: headless browser"]
  end
  subgraph POST["After release"]
    F["Production smoke test"] --> G["Synthetic monitor every hour"]
    H["Access logs: verified Googlebot"] --> I["Search signals dashboard"]
    J["Search Console API"] --> I
    G --> I
  end
  E --> F
  I --> K["Alert to technical SEO on-call"]
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  class B,D,E,F,G core
  class I,K decision
  class A,C,H,J neutral

Trust boundaries. Protect previews with authentication, not robots.txt, which is not access control and “exposes them publicly”.^[9] The Search Console credential needs only the webmasters.readonly scope.^[24]

sequenceDiagram
  autonumber
  participant Dev as Engineer
  participant CI as CI and preview
  participant Prod as Production
  participant G as Google
  Dev->>CI: Pull request
  CI-->>Dev: Checks fail: noindex
  Dev->>CI: Fix pushed
  CI->>Prod: Release and smoke test
  G->>Prod: Crawls changed pages
  Note over Prod,G: Team reads index state via API and logs

Happy path: the contract passes everywhere and, a week later, URL Inspection shows Google’s canonical equals the declared one. Exception paths: a failed check blocks the merge unless a reviewer records an override with the contract updated in the same change; a failed production smoke test is an incident (worked runbook); a later “Duplicate, Google chose different canonical than user” is a signal conflict, not an outage.^[25]

1. Generate robots.txt per environment from a reviewed source; previews also require authentication.
2. Keep expected outcomes (URL, user agent, allowed or not) and run them in CI (Lab 1).
3. A group naming Googlebot is the only one Googlebot follows; it is not merged with *.^[6] Changing a rule in one group but not the other is a classic regression.
4. Monitor the production file. A 4xx (except 429) means “no crawl restrictions”; a 5xx stops crawling for 12 hours, then Google uses the last good copy for up to 30 days.^[6]
5. Never use it to hide pages from results, to protect anything, or to canonicalise duplicates.^[13][18]

• One absolute, self-referencing canonical per indexable page, in the server HTML; internal links, sitemaps, hreflang and redirects all point to the same URL, because signals stack.^[18] HTTPS only: a bad certificate or an HTTPS-to-HTTP redirect makes Google prefer HTTP “very strongly”.^[18]
• Google queues 200 pages for rendering (unless a robots meta tag or header says noindex), and a page “may stay on this queue for a few seconds, but it can take longer than that”. Use <a href> links and History API routes, not # fragments; give unknown IDs a real 404 (or a redirect to one, or an injected noindex); never let JavaScript change the canonical; fingerprint asset names, as the renderer may ignore caching headers.^[19] The renderer is stateless, so content gated on local or session storage may not render.^[12]

Good Core Web Vitals, “along with other page experience aspects, aligns with what our core ranking systems seek to reward”, says Google (Largest Contentful Paint, LCP, within 2.5 s; Interaction to Next Paint, INP, under 200 ms; Cumulative Layout Shift, CLS, under 0.1).^[28] SEOps only watches the Search Console report by template; measuring and fixing belong to WebOps. One crawler point stays here: heavy inline scripts or base64 images can push content or JSON-LD past Googlebot’s 2MB cut-off.^[12]

Engines can differ from the RFC: it says a crawler “MUST assume complete disallow” when robots.txt is unreachable, while Google documents a 12-hour stop, then up to 30 days on its last cached copy.^[9][6]

Check what a “robots.txt tester” implements: Python 3.8’s urllib.robotparser, tested for this page, used first match and treated * literally; current CPython sources use longest match.^[33] Judging third-party “visibility” and AI-search tools is programme work: see SEO Operations.

• Automate the deterministic parts (contracts, robots expectations, redirect maps, sitemaps) in CI. Language models can cluster thousands of excluded URLs for triage; they should not write robots rules or canonicals without tests.
• Generated page sets go through the spam-policy review.^[8]
• AI features in Google Search have a property setting (as of October 2026). Besides being indexed and snippet-eligible, a site must be included via Search Console’s Search generative AI control.^[34] Its values are Include, Exclude and Inherit (from a parent property; the default where one exists, else Include). It governs links and content in AI Overviews, AI Mode and Discover’s generative AI features; Google says it is not a ranking signal elsewhere and does not affect training (Google-Extended’s job), while noindex leaves Search altogether.^[35][36] Whether to exclude is a business decision for SEO Operations; SEOps records the value per property and checks it has not drifted. llms.txt plays no part in Google Search.^[34]

Assumptions. CI that can deploy password-protected previews; Python 3.11+ (3.10 reached end of life on 2026-10-01^[37]) and, for part 3, Node.js 20+; Google as the main search engine.

1. The search contract. One entry per representative URL of each template; self means “canonical equals this path on the production host”.

Example status: syntax-checked by the validator and executed against a local test site on 2026-10-23.

{
  "base": "http://127.0.0.1:8765/",
  "canonical_host": "https://shop.example.com/",
  "agent": "googlebot",
  "sitemap": "/sitemap.xml",
  "pages": [
    {"path": "/", "status": 200, "indexable": true, "canonical": "self"},
    {"path": "/shoes", "status": 301, "location": "/shoes/"},
    {"path": "/shoes/", "status": 200, "indexable": true, "canonical": "self"},
    {"path": "/shoes/trail-runner/", "status": 200, "indexable": true,
     "canonical": "self", "structured_data": ["Product"]},
    {"path": "/account/", "status": 200, "indexable": false},
    {"path": "/no-such-page/", "status": 404}
  ]
}

Optional keys: robots_file (judge indexability by a local file, normally the generated production robots.txt) and structured_data (JSON-LD types that part 3 requires).

2. The contract checker. It requests each URL without following redirects, reading 2MB like Googlebot.^[12] It honours X-Robots-Tag only when unscoped or scoped to its crawler (bingbot: noindex does not count^[38]), takes canonicals from <head>, where Google accepts them,^[18] and the Link header, which must agree. It applies robots.txt with Lab 1’s matcher, follows a (gzipped) sitemap index, and fails on an unreadable or empty sitemap. SEO_GATE_AUTH carries a preview credential to the base host only.

Example status: executed on 2026-10-23 with Python 3.11.17 (and 3.8.10): Lab 2 plus tests against a local Basic-auth server (gzipped index, empty and 3MB sitemaps, body-only canonical, conflicting Link header, crawler-scoped X-Robots-Tag).

"""seo_gate.py: check a running build against a search contract (JSON).
Fails (exit 1) when a page's status, indexability or canonical differs from
the contract, or when a sitemap (or sitemap index) lists a URL that is not
indexable and self-canonical, or lists none. Standard library only; does not
execute JavaScript.
Env SEO_GATE_AUTH (optional): Authorization header value for a protected
preview, e.g. "Basic <base64>"; sent only to the contract's base host.
Usage: python3 seo_gate.py contract.json
"""
import gzip
import json
import os
import re
import sys
import urllib.error
import urllib.request
import xml.etree.ElementTree as ET
from html.parser import HTMLParser
from urllib.parse import urljoin, urlsplit

import robots_check

UA = "seo-gate/1.1 (+https://example.com/seo-gate)"
PAGE_LIMIT = 2 * 1024 * 1024        # Googlebot reads the first 2MB of HTML
SITEMAP_LIMIT = 50 * 1024 * 1024    # sitemaps.org: 50MB uncompressed
HEAD_TAGS = {"html", "head", "title", "meta", "link", "base", "script",
             "style", "noscript", "template"}
SM = "{http://www.sitemaps.org/schemas/sitemap/0.9}"


class NoRedirect(urllib.request.HTTPRedirectHandler):
    def redirect_request(self, *args, **kwargs):
        return None  # report the first response, do not follow


class HeadParser(HTMLParser):
    """Robots meta tags anywhere; canonical links only inside <head>."""

    def __init__(self, agent):
        super().__init__()
        self.names = ("robots", agent)
        self.in_head, self.robots, self.canonicals = True, [], []

    def handle_starttag(self, tag, attrs):
        a = {k.lower(): (v or "") for k, v in attrs}
        if tag not in HEAD_TAGS:
            self.in_head = False  # <body> or any body element ends the head
        if tag == "meta" and a.get("name", "").lower() in self.names:
            self.robots.append(a.get("content", "").lower())
        rel = a.get("rel", "").lower().split()
        if tag == "link" and "canonical" in rel and self.in_head:
            self.canonicals.append(a.get("href", ""))

    def handle_endtag(self, tag):
        if tag == "head":
            self.in_head = False


def fetch(url, limit=PAGE_LIMIT):
    opener = urllib.request.build_opener(NoRedirect)
    headers = {"User-Agent": UA}
    if os.environ.get("SEO_GATE_AUTH"):
        headers["Authorization"] = os.environ["SEO_GATE_AUTH"]
    req = urllib.request.Request(url, headers=headers)
    try:
        with opener.open(req, timeout=30) as resp:
            return resp.status, resp.headers, resp.read(limit)
    except urllib.error.HTTPError as err:
        return err.code, err.headers, b""


def header_rules(headers, agent):
    """X-Robots-Tag values for all crawlers or for this agent only."""
    words = set()
    for value in headers.get_all("X-Robots-Tag") or []:
        m = re.match(r"\s*([a-z0-9_-]+)\s*:\s*(.*)", value.lower())
        if m and m.group(1) not in ("unavailable_after", "max-snippet",
                                    "max-image-preview", "max-video-preview"):
            if m.group(1) != agent:
                continue  # e.g. "bingbot: noindex" does not apply
            value = m.group(2)
        words |= {w.split(":")[0].strip() for w in value.lower().split(",")}
    return words


def inspect(contract, groups, path):
    url = urljoin(contract["base"], path)
    status, headers, raw = fetch(url)
    agent = contract["agent"]
    page = HeadParser(agent)
    page.feed(raw.decode("utf-8", "replace"))
    words = header_rules(headers, agent)
    words |= {w.strip() for d in page.robots for w in d.split(",")}
    noindex = "noindex" in words or "none" in words
    crawlable = robots_check.allowed(groups, agent, url)
    links = ",".join(headers.get_all("Link") or [])
    canons = re.findall(r'<([^>]+)>\s*;[^,]*rel="?canonical"?', links)
    canons = set(canons + page.canonicals)  # HTTP header and HTML must agree
    return {
        "status": status,
        "location": headers.get("Location"),
        "indexable": status == 200 and crawlable and not noindex,
        "canonical": next(iter(canons)) if len(canons) == 1 else None,
        "canonical_count": len(canons),
    }


def check_page(contract, groups, spec):
    got = inspect(contract, groups, spec["path"])
    errors = []
    if got["status"] != spec["status"]:
        errors.append(f"status {got['status']} != {spec['status']}")
    if "location" in spec and got["location"] != spec["location"]:
        errors.append(f"Location {got['location']} != {spec['location']}")
    if "indexable" in spec and got["indexable"] != spec["indexable"]:
        errors.append(f"indexable {got['indexable']} != {spec['indexable']}")
    if "canonical" in spec:
        target = spec["path"] if spec["canonical"] == "self" else spec["canonical"]
        want = urljoin(contract["canonical_host"], target)
        if got["canonical_count"] != 1:
            errors.append(f"{got['canonical_count']} distinct canonicals (want 1)")
        elif got["canonical"] != want:
            errors.append(f"canonical {got['canonical']} != {want}")
    return errors


def sitemap_paths(contract, path, depth=0):
    """Paths listed by a sitemap; follows one level of sitemap index."""
    status, _, raw = fetch(urljoin(contract["base"], path), SITEMAP_LIMIT)
    if status != 200:
        raise ValueError(f"{path} returned {status}")
    if raw[:2] == b"\x1f\x8b":
        raw = gzip.decompress(raw)
    root = ET.fromstring(raw)
    if root.tag == SM + "sitemapindex" and depth == 0:
        found = []
        for loc in root.iter(SM + "loc"):
            found += sitemap_paths(contract, urlsplit(loc.text.strip()).path, 1)
        return found
    if root.tag != SM + "urlset":
        raise ValueError(f"{path}: unexpected root element {root.tag}")
    locs = [e.text.strip() for e in root.iter(SM + "loc")]
    return [urlsplit(u).path or "/" for u in locs]


def main(contract_file):
    with open(contract_file, encoding="utf-8") as fh:
        contract = json.load(fh)
    failures = 0
    if contract.get("robots_file"):  # e.g. the generated production file
        with open(contract["robots_file"], encoding="utf-8") as fh:
            robots_txt = fh.read()
    else:
        robots_status, _, raw = fetch(urljoin(contract["base"], "/robots.txt"))
        robots_txt = raw.decode("utf-8", "replace")
        if robots_status != 200:  # 5xx pauses crawling; 4xx means "no rules"
            failures += 1
            print(f"FAIL /robots.txt returned {robots_status}")
    groups = robots_check.parse(robots_txt)
    for spec in contract["pages"]:
        errors = check_page(contract, groups, spec)
        failures += bool(errors)
        print(("FAIL " if errors else "PASS ") + spec["path"], "; ".join(errors))
    try:
        paths = sitemap_paths(contract, contract["sitemap"])
        problem = "" if paths else "no URLs listed"
    except (ValueError, OSError, ET.ParseError) as err:
        paths, problem = [], str(err)
    if problem:  # an empty or unreadable sitemap fails, never passes
        failures += 1
        print(f"FAIL sitemap {contract['sitemap']}: {problem}")
    for path in paths:
        errors = check_page(contract, groups,
                            {"path": path, "status": 200, "indexable": True,
                             "canonical": "self"})
        failures += bool(errors)
        if errors:
            print("FAIL sitemap entry " + path, "; ".join(errors))
    print(f"{len(paths)} sitemap URL(s) checked; {failures} failing check(s)")
    return 1 if failures else 0


if __name__ == "__main__":
    sys.exit(main(sys.argv[1]))

Design choices: redirects are asserted, not followed; a missing robots.txt fails, although Google reads a 404 as “no crawl restrictions”,^[6] because in production it usually means a broken deploy. Known limits: robots meta tags count anywhere in the document (the stricter reading); no JavaScript runs.

3. The rendered check. A headless browser loads each indexable page with JavaScript off and on, and fails when scripts change the title, canonical or robots tags or when JSON-LD is invalid or missing; links that exist only after rendering are a warning, as Google renders later.^[19] Install with npm install puppeteer@24.43.1.

Example status: executed on 2026-10-23 (Node.js 20.19.5, Puppeteer 24.43.1) in Lab 2 step 6 and against the Basic-auth test server.

// render_check.mjs: compare server HTML with the rendered DOM for each
// indexable contract page. Fails when JavaScript adds a robots rule, changes
// the canonical or title, or when JSON-LD is invalid or a required type is
// missing; warns when links exist only after rendering.
// Env SEO_GATE_AUTH as for seo_gate.py. Usage: node render_check.mjs contract.json
import { readFileSync } from 'node:fs';
import puppeteer from 'puppeteer';

const contract = JSON.parse(readFileSync(process.argv[2], 'utf8'));
const auth = process.env.SEO_GATE_AUTH;
const origin = new URL(contract.base).origin;

const snapshot = () => ({  // runs inside the page
  title: document.title,
  canonical: [...document.head.querySelectorAll('link[rel~="canonical"]')]
    .map((l) => l.getAttribute('href')).join(' '),
  robots: [...document.querySelectorAll('meta[name="robots" i],meta[name="googlebot" i]')]
    .map((m) => m.content.toLowerCase()).join(','),
  links: [...document.querySelectorAll('a[href]')].map((a) => a.getAttribute('href')),
  jsonld: [...document.querySelectorAll('script[type="application/ld+json"]')]
    .map((s) => s.textContent),
});

async function view(browser, url, js) {
  const page = await browser.newPage();
  await page.setJavaScriptEnabled(js);
  await page.setRequestInterception(true);
  page.on('request', (req) => {
    const same = new URL(req.url()).origin === origin;  // credential stays on the preview
    const headers = { ...req.headers(), authorization: auth };
    req.continue(same && auth ? { headers } : {});
  });
  await page.goto(url, { waitUntil: 'networkidle0', timeout: 30000 });
  const result = await page.evaluate(snapshot);
  await page.close();
  return result;
}

function types(blocks, errors) {
  const found = new Set();
  for (const text of blocks) {
    try {
      for (const item of [JSON.parse(text)].flat()) {
        [item['@type']].flat().forEach((t) => found.add(t));
      }
    } catch {
      errors.push('JSON-LD does not parse');
    }
  }
  return found;
}

const browser = await puppeteer.launch();
let failures = 0;
for (const spec of contract.pages.filter((p) => p.indexable)) {
  const url = new URL(spec.path, contract.base).href;
  const server = await view(browser, url, false);   // JavaScript off: server HTML
  const rendered = await view(browser, url, true);  // JavaScript on: rendered DOM
  const errors = [];
  for (const key of ['title', 'canonical', 'robots']) {
    if (server[key] !== rendered[key]) {
      errors.push(`${key} changed by JavaScript: "${server[key]}" -> "${rendered[key]}"`);
    }
  }
  const found = types(rendered.jsonld, errors);
  for (const t of spec.structured_data || []) {
    if (!found.has(t)) errors.push(`JSON-LD type ${t} missing`);
  }
  const jsOnly = rendered.links.filter((h) => !server.links.includes(h));
  failures += errors.length > 0;
  console.log(`${errors.length ? 'FAIL' : 'PASS'} ${spec.path} ${errors.join('; ')}`);
  if (jsOnly.length) {
    console.log(`  WARN ${jsOnly.length} link(s) only after rendering: ${jsOnly[0]}`);
  }
}
await browser.close();
console.log(`${failures} failing page(s)`);
process.exit(failures ? 1 : 0);

4. The pipeline job runs on every pull request: a paths filter would skip the worked runbook’s lockfile-only upgrade, and a required check skipped by path filtering stays “Pending”, blocking the merge.^[39] It tests the generated production robots.txt and judges indexability by it (robots_file), since a preview’s own file normally disallows everything. Fork pull requests get no Actions secrets and Dependabot ones only Dependabot secrets, so store PREVIEW_AUTH in both; without it the gate fails, never passes.^[40][41]

Example status: linted with actionlint 1.7.12, not executed; the two .sh scripts are placeholders. Actions pinned by commit SHA (versions as of October 2026).

name: search-contract
on:
  pull_request:  # no paths filter: lockfile, layout and CDN changes break search too
permissions:
  contents: read
jobs:
  seo-gate:
    runs-on: ubuntu-latest
    env:
      SEO_GATE_AUTH: ${{ secrets.PREVIEW_AUTH }}  # "Basic <base64>"
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1  # v7.0.1
      - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97  # v7.0.0
        with:
          python-version: "3.13"
      - name: Test the generated production robots.txt
        run: |
          ./seo/generate-robots.sh production > robots.prod.txt
          python3 seo/robots_check.py robots.prod.txt seo/expectations.json
      - name: Deploy password-protected preview
        run: ./deploy-preview.sh > preview-url.txt
      - name: Run the search contract against the preview
        run: |
          jq --arg base "$(cat preview-url.txt)" \
            '.base = $base | .robots_file = "robots.prod.txt"' \
            seo/contract.json > contract.ci.json
          python3 seo/seo_gate.py contract.ci.json
      - name: Rendered check (templates that need JavaScript)
        run: |
          npm ci
          node seo/render_check.mjs contract.ci.json

5. Production monitoring. The same contract after each deploy and hourly; a weekly URL Inspection sample (Lab 4); a daily verified-Googlebot log summary (Lab 3).

Labs 1–3 need Python 3.11+ and a terminal (Lab 2’s last step also Node.js 20+); they were run on 2026-10-23 with Python 3.11.17 and show actual outputs. Lab 4 was not executed.

Objective. Catch a robots.txt regression before it ships, matching as RFC 9309 and Google describe: one group per crawler token (googlebot/1.2 means googlebot), percent-encoded comparison, longest rule wins, allow wins a tie.^[6][9] Prerequisites. Python 3.11+; no network. Cost and risk. None; nothing is published.

1. Save the matcher as robots_check.py.

Example status: executed with Python 3.11.17 (and 3.8.10) on 2026-10-23; it also agreed with 32 cases from Google’s documentation and RFC 9309 (precedence, path matching, user-agent groups, percent-encoding, byte-order mark). Known limit: a reserved character literal on one side and encoded on the other (=, %3D) does not match.

"""robots_check.py: decide allow/disallow for one crawler token, following RFC
9309 and Google's robots.txt documentation: the group naming the token (else
the '*' group), user-agent text after the token ignored, paths compared in
percent-encoded form, longest matching rule wins, allow wins a tie.
Standard library only. Usage:
    python3 robots_check.py robots.txt expectations.json
"""
import json
import re
import sys
from urllib.parse import quote, urlsplit

UNRESERVED = re.compile(r"%(4[1-9A-F]|5[0-9A]|6[1-9A-F]|7[0-9A]|3[0-9]|2D|2E|5F|7E)")


def normalise(path):
    """Percent-encode non-ASCII octets, upper-case escapes, decode unreserved."""
    path = quote(path, safe="".join(chr(c) for c in range(33, 127)))
    path = re.sub(r"%[0-9a-fA-F]{2}", lambda m: m.group(0).upper(), path)
    return UNRESERVED.sub(lambda m: chr(int(m.group(1), 16)), path)


def agent_token(value):
    """'Googlebot/1.2' and 'googlebot*' both mean 'googlebot'; '*' stays."""
    value = value.strip().lower()
    match = re.match(r"[a-z_-]+", value)
    return "*" if value.startswith("*") else (match.group(0) if match else value)


def parse(text):
    """Return a list of (agents, rules) groups; rules are (kind, path)."""
    groups, agents, rules, in_agents = [], [], [], False
    for raw in text.splitlines():
        line = raw.split("#", 1)[0].strip().lstrip("")
        if ":" not in line:
            continue
        field, value = (part.strip() for part in line.split(":", 1))
        field = field.lower()
        if field == "user-agent":
            if not in_agents and agents:
                groups.append((agents, rules))
                agents, rules = [], []
            agents.append(agent_token(value))
            in_agents = True
        elif field in ("allow", "disallow"):
            if agents and value:  # rules without a path are ignored
                rules.append((field, normalise(value)))
            in_agents = False
        # other fields (sitemap, crawl-delay, ...) neither end nor start a group
    if agents:
        groups.append((agents, rules))
    return groups


def rules_for(groups, token):
    """Merge every group naming the token; fall back to the '*' groups."""
    token = agent_token(token)
    named = [r for a, r in groups if token in a]
    chosen = named or [r for a, r in groups if "*" in a]
    return [rule for rules in chosen for rule in rules]


def to_regex(path):
    anchored = path.endswith("$")
    body = re.escape(path[:-1] if anchored else path).replace(r"\*", ".*")
    return re.compile("^" + body + ("$" if anchored else ""))


def allowed(groups, token, url):
    parts = urlsplit(url)
    target = (parts.path or "/") + ("?" + parts.query if parts.query else "")
    target = normalise(target)
    if target == "/robots.txt":
        return True
    best_len, verdict = -1, True
    for kind, path in rules_for(groups, token):
        if to_regex(path).match(target):
            length = len(path)
            if length > best_len or (length == best_len and kind == "allow"):
                best_len, verdict = length, kind == "allow"
    return verdict


def main(robots_file, expectations_file):
    with open(robots_file, encoding="utf-8") as fh:
        groups = parse(fh.read())
    with open(expectations_file, encoding="utf-8") as fh:
        cases = json.load(fh)
    failures = 0
    for case in cases:
        got = allowed(groups, case["agent"], case["url"])
        ok = got == case["allowed"]
        failures += not ok
        where = urlsplit(case["url"])
        target = where.path + ("?" + where.query if where.query else "")
        print(f"{'PASS' if ok else 'FAIL'} {case['agent']:<10} {target:<28} "
              f"allowed={got} expected={case['allowed']}")
    print(f"{len(cases) - failures}/{len(cases)} expectations met")
    return 1 if failures else 0


if __name__ == "__main__":
    sys.exit(main(sys.argv[1], sys.argv[2]))

2. Save this illustrative shop robots.txt. Note the separate Googlebot group: Googlebot will ignore the * group entirely.^[6]

# Production robots.txt (illustrative shop)
User-agent: *
Disallow: /cart
Disallow: /search
Disallow: /*?*sort=
Disallow: /*?*color=
Allow: /*?page=
Disallow: /*.json$

User-agent: Googlebot
Disallow: /cart
Disallow: /search
Disallow: /*?*sort=
Disallow: /*?*color=
Disallow: /*.json$
Allow: /search/help

Sitemap: https://shop.example.com/sitemap.xml

3. Save the expectations as expectations.json.

[
  {"agent": "googlebot", "url": "https://shop.example.com/", "allowed": true},
  {"agent": "googlebot", "url": "https://shop.example.com/shoes/trail-runner",
   "allowed": true},
  {"agent": "googlebot", "url": "https://shop.example.com/shoes?color=red",
   "allowed": false},
  {"agent": "googlebot", "url": "https://shop.example.com/shoes?page=2&sort=price",
   "allowed": false},
  {"agent": "googlebot", "url": "https://shop.example.com/search/help",
   "allowed": true},
  {"agent": "googlebot", "url": "https://shop.example.com/api/stock.json",
   "allowed": false},
  {"agent": "bingbot", "url": "https://shop.example.com/shoes?page=2",
   "allowed": true},
  {"agent": "googlebot", "url": "https://shop.example.com/static/app.js",
   "allowed": true}
]

4. Run python3 robots_check.py robots.txt expectations.json.

Expected output (actual output of the run):

PASS googlebot  /                            allowed=True expected=True
PASS googlebot  /shoes/trail-runner          allowed=True expected=True
PASS googlebot  /shoes?color=red             allowed=False expected=False
PASS googlebot  /shoes?page=2&sort=price     allowed=False expected=False
PASS googlebot  /search/help                 allowed=True expected=True
PASS googlebot  /api/stock.json              allowed=False expected=False
PASS bingbot    /shoes?page=2                allowed=True expected=True
PASS googlebot  /static/app.js               allowed=True expected=True
8/8 expectations met

Validation. Delete Disallow: /*?*color= from the Googlebot group only and rerun: the third expectation fails, exit status 1. Troubleshooting. Paths are case-sensitive and must start with /.^[6] Teardown. Delete the three files.

Objective. Catch an accidental noindex, a staging canonical and a noindex that only JavaScript adds. Prerequisites. robots_check.py (Lab 1), seo_gate.py, render_check.mjs and contract.json (reference implementation) in one directory; port 8765 free; for step 6, Node.js 20+ and npm install puppeteer@24.43.1. Cost and risk. None; the server listens on 127.0.0.1 only.

1. Create a tiny site. The trail-runner page carries an injected noindex, the kind of line a template change adds.

Example status: executed on 2026-10-23 (bash 5, Python 3.11.17).

mkdir -p site/shoes/trail-runner site/account
printf 'User-agent: *\nDisallow: /cart/\n' > site/robots.txt
page() {  # title, canonical path, extra head markup
  printf '<!doctype html><html><head><title>%s</title>%s' "$1" "$3"
  printf '<link rel="canonical" href="https://shop.example.com%s">' "$2"
  printf '</head><body><h1>%s</h1></body></html>\n' "$1"
}
page "Shop" "/" "" > site/index.html
page "Shoes" "/shoes/" "" > site/shoes/index.html
page "Trail runner" "/shoes/trail-runner/" \
  '<meta name="robots" content="noindex">' > site/shoes/trail-runner/index.html
page "Account" "/account/" \
  '<meta name="robots" content="noindex, nofollow">' > site/account/index.html
cat > site/sitemap.xml <<'XML'
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url><loc>https://shop.example.com/</loc></url>
  <url><loc>https://shop.example.com/shoes/</loc></url>
  <url><loc>https://shop.example.com/shoes/trail-runner/</loc></url>
</urlset>
XML
python3 -m http.server 8765 --bind 127.0.0.1 --directory site &
echo $! > server.pid

2. Run python3 seo_gate.py contract.json.

Expected output (actual):

PASS /
PASS /shoes
PASS /shoes/
FAIL /shoes/trail-runner/ indexable False != True
PASS /account/
PASS /no-such-page/
FAIL sitemap entry /shoes/trail-runner/ indexable False != True
3 sitemap URL(s) checked; 2 failing check(s)

Exit status 1. /shoes passes because Python’s server answers 301 to /shoes/, as the contract expects.

3. Remove the noindex from site/shoes/trail-runner/index.html and rerun: all checks pass, 3 sitemap URL(s) checked; 0 failing check(s), exit 0.
4. Change the canonical in site/shoes/index.html to https://staging.example.com/shoes/ and rerun: the checker reports canonical https://staging.example.com/shoes/ != https://shop.example.com/shoes/ for the page and for its sitemap entry. Change it back.
5. Turn the sitemap into an index with a gzipped child, as large sites do, and rerun: the same three URLs are checked, 0 failing check(s). (The first version read no URLs from an index, and passed.)

gzip -c site/sitemap.xml > site/sitemap-pages.xml.gz
cat > site/sitemap.xml <<'XML'
<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap><loc>https://shop.example.com/sitemap-pages.xml.gz</loc></sitemap>
</sitemapindex>
XML

6. Give the trail-runner page JSON-LD and a script that adds noindex and a link, then run node render_check.mjs contract.json.

cat > site/shoes/trail-runner/index.html <<'HTML'
<!doctype html><html><head><title>Trail runner</title>
<link rel="canonical" href="https://shop.example.com/shoes/trail-runner/">
<script type="application/ld+json">
{"@context": "https://schema.org", "@type": "Product", "name": "Trail runner"}
</script>
<script>
  const tag = document.createElement("meta");  // what a faulty SEO plug-in does
  tag.name = "robots";
  tag.content = "noindex";
  document.head.append(tag);
  addEventListener("DOMContentLoaded", () => document.body.insertAdjacentHTML(
    "beforeend", '<a href="/shoes/sizes/">Size guide</a>'));
</script>
</head><body><h1>Trail runner</h1></body></html>
HTML

Expected output (actual):

PASS /
PASS /shoes/
FAIL /shoes/trail-runner/ robots changed by JavaScript: "" -> "noindex"
  WARN 1 link(s) only after rendering: /shoes/sizes/
1 failing page(s)

seo_gate.py passes this page, as the noindex exists only after JavaScript runs. Delete the second <script> block and rerun: three PASS lines, exit 0.

Validation. Each defect fails; the fixed site passes. Troubleshooting. Connection refused: check the server and base. Teardown. kill "$(cat server.pid)", then delete site/, server.pid, node_modules/ and the scripts.

Objective. Separate verified Googlebot requests from impostors; see the status mix and crawl on parameter URLs. Google’s manual check (reverse DNS in googlebot.com, google.com or googleusercontent.com, then a forward lookup returning the same IP) covers all its crawlers and fetchers; for Googlebot the name is crawl-…googlebot.com or geo-crawl-…geo.googlebot.com.^[42] googleusercontent.com also covers Google Cloud customers’ machines,^[43] so the script accepts only googlebot.com, over IPv4 and IPv6. Prerequisites. Python 3.11+, outbound DNS, an Apache/Nginx “combined” access log.^[44] Risk. Logs contain personal data; use a copy under your retention rules.

Example status: executed on 2026-10-23 with live DNS, Python 3.11.17 (and 3.8.10).

"""googlebot_log.py: summarise verified Googlebot requests in an Apache/Nginx
"combined" access log. A request counts only if the reverse DNS name of the
client IP (IPv4 or IPv6) is in googlebot.com, Google's mask for its common
crawlers, AND a forward lookup of that name returns the same IP.
Usage: python3 googlebot_log.py access.log
"""
import collections
import ipaddress
import re
import socket
import sys

LINE = re.compile(r'^(\S+) \S+ \S+ \[[^\]]+\] "(\S+) (\S+) [^"]*" (\d{3}) \S+ '
                  r'"[^"]*" "([^"]*)"')
# crawl-*.googlebot.com or geo-crawl-*.geo.googlebot.com. Not google.com or
# googleusercontent.com: those also cover fetchers and Google Cloud customers.
GOOGLEBOT = ".googlebot.com"


def verified(ip, cache={}):
    if ip not in cache:
        try:
            addr = ipaddress.ip_address(ip)
            host = socket.gethostbyaddr(ip)[0].rstrip(".").lower()
            forward = {ipaddress.ip_address(info[4][0].split("%")[0])
                       for info in socket.getaddrinfo(host, None)}
            cache[ip] = host.endswith(GOOGLEBOT) and addr in forward
        except (OSError, ValueError):
            cache[ip] = False
    return cache[ip]


def section(path):
    if "?" in path:
        return "parameterised"
    parts = path.split("/")
    return "/" + parts[1] if len(parts) > 1 and parts[1] else "/"


def main(log_file):
    statuses = collections.Counter()
    sections = collections.Counter()
    spoofed = collections.Counter()
    with open(log_file, encoding="utf-8", errors="replace") as fh:
        for line in fh:
            m = LINE.match(line)
            if not m or "Googlebot" not in m.group(5):
                continue
            ip, _method, path, status, _ua = m.groups()
            if not verified(ip):
                spoofed[ip] += 1
                continue
            statuses[status[0] + "xx"] += 1
            sections[section(path)] += 1
    total = sum(statuses.values())
    print(f"verified Googlebot requests: {total}")
    for key, n in sorted(statuses.items()):
        print(f"  {key}: {n} ({n / total:.0%})" if total else f"  {key}: {n}")
    for key, n in sections.most_common(10):
        print(f"  section {key}: {n}")
    print(f"claimed-Googlebot requests that failed verification: "
          f"{sum(spoofed.values())} from {len(spoofed)} IP(s)")


if __name__ == "__main__":
    main(sys.argv[1])

Sample log: Google’s example IP, an IPv6 Googlebot address, two impostors and a visitor. Set GCP_IP to a Google Cloud external IP whose reverse name is under bc.googleusercontent.com.

: "${GCP_IP:?set GCP_IP to a Google Cloud external IP}"
UA='Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)'
T='[23/Oct/2026:06:00:00 +0000]'
while read -r ip req code; do
  ua=$UA; [ "$ip" = 198.51.100.23 ] && ua='Mozilla/5.0 (X11; Linux x86_64)'
  echo "$ip - - $T \"GET $req HTTP/1.1\" $code 512 \"-\" \"$ua\""
done > access.log <<LOG
66.249.66.1 /robots.txt 200
66.249.66.1 /shoes/trail-runner 200
66.249.66.1 /shoes/?color=red&sort=price 200
66.249.66.1 /account/orders 503
2001:4860:4801:10::1 /shoes/rain-jacket 200
203.0.113.7 /admin/ 200
$GCP_IP /shoes/ 200
198.51.100.23 / 200
LOG
python3 googlebot_log.py access.log

Expected output (actual):

verified Googlebot requests: 5
  2xx: 4 (80%)
  5xx: 1 (20%)
  section /shoes: 2
  section /robots.txt: 1
  section parameterised: 1
  section /account: 1
claimed-Googlebot requests that failed verification: 2 from 2 IP(s)

Validation. host 66.249.66.1 returns crawl-66-249-66-1.googlebot.com, as in Google’s example.^[42] The IPv6 and Google Cloud lines are regression tests: an earlier version rejected the IPv6 line and accepted the Google Cloud line. Troubleshooting. No verified requests: the log may record a CDN IP. For large logs, match Google’s IP-range files (common-crawlers.json) instead of DNS.^[45] Teardown. Delete the script and any log copies.

Objective. Compare Google’s selected canonical and index state for key URLs with the contract. Prerequisites. A verified Search Console property; in GSC_TOKEN, an OAuth 2.0 access token (a short-lived API credential) with the webmasters.readonly scope for a user of that property.^[24] Cost and risk. Free; quota is 2,000 inspections per site per day and 600 per minute (as of October 2026).^[46] The API reports the indexed version; it cannot test a live URL.^[24]

Example status: syntax-checked, not executed (requires a verified property and credentials).

"""inspect_urls.py: report Google's indexed view of key URLs.
Env GSC_TOKEN: OAuth 2.0 access token, scope webmasters.readonly.
Usage: python3 inspect_urls.py sc-domain:example.com urls.txt
"""
import json
import os
import sys
import time
import urllib.request

API = "https://searchconsole.googleapis.com/v1/urlInspection/index:inspect"


def inspect(site, url, token):
    body = json.dumps({"inspectionUrl": url, "siteUrl": site}).encode()
    req = urllib.request.Request(API, data=body, method="POST", headers={
        "Authorization": "Bearer " + token, "Content-Type": "application/json"})
    with urllib.request.urlopen(req, timeout=30) as resp:
        return json.load(resp)["inspectionResult"]["indexStatusResult"]


def main(site, urls_file):
    token = os.environ["GSC_TOKEN"]
    with open(urls_file, encoding="utf-8") as fh:
        urls = [line.strip() for line in fh if line.strip()][:200]
    for url in urls:  # 200 per run keeps far below the daily per-site quota
        r = inspect(site, url, token)
        same = r.get("googleCanonical") == r.get("userCanonical")
        flag = "OK   " if r.get("verdict") == "PASS" and same else "CHECK"
        print(flag, url, r.get("coverageState"), r.get("lastCrawlTime"),
              "google-canonical=" + str(r.get("googleCanonical")))
        time.sleep(0.2)


if __name__ == "__main__":
    main(sys.argv[1], sys.argv[2])

Expected output. One line per URL; OK when the verdict is PASS (“Valid”) and Google’s canonical equals the declared one.^[47] Validation. Compare a few lines with the URL Inspection tool. Troubleshooting. 403: no access, or a wrong siteUrl (URL-prefix properties need the trailing slash).^[24] Teardown. Revoke the token if created for the lab.

All four are illustrative scenarios built from documented mechanisms, not reported case studies.

Pitfalls. Blocking facets in robots.txt hides any noindex or canonical on them.^[5] Serving bots different content from users is cloaking, even as a rendering workaround.^[11][8] Redirecting every old URL to the home page throws the mapping away. A sitemap claiming every page changed today teaches Google to ignore lastmod.^[10]

SEOps measures whether the controls hold; rankings and clicks belong to SEO Operations. The metric names are this Atlas’s proposals.

Do not target 100 % indexing: “Don’t expect every URL on your site to be indexed”, and the Page indexing report shows at most 1,000 example URLs per status.^[25] Use indexed counts per sitemap as a trend.

Targets. Blocking checks on production: 100 %. Sitemap hygiene above 99 %. For crawl waste and error rate set a baseline and a direction; no “good” value is published.^[17]

Spam policies, release by release. Code changes can create unintended policy problems:^[8]

Security. robots.txt is public and advisory; listing a secret path advertises it.^[9] User agents “can be spoofed”, so verify crawlers by DNS or Google’s IP-range files before granting allowlists or higher rate limits, and accept only googlebot.com names for Googlebot (Lab 3): google.com and gae.googleusercontent.com names also belong to user-triggered fetchers, which ignore robots.txt.^[36][42] Watch the Security Issues report for hacked content alongside SecOps.^[8][7] A structured-data manual action removes rich-result eligibility, not web ranking.^[23]

Privacy. Logs hold IP addresses; retention rules come from PrivacyOps.

• Capacity is a search input. Google’s crawl capacity limit falls with slow responses, 5xx or 429 and is shared by all Google crawlers on the host.^[17] Cheap wins: 304 Not Modified, HTML under 2MB, less filter-URL crawling, which can cost “large amounts of computing resources”.^[17][12][27]
• Tooling. Search Console and its API are free within quotas;^[46] tool prices are not quoted. The main cost is engineering time.
• Sustainability. Crawl waste is wasted compute on both sides; removing it costs no visibility.

1. Canary fixtures. A deliberately noindex page in a test sitemap must keep failing the gate; check monthly.
2. Monitor the monitors. Alert when a contract job, log summary or export has not run for 24 hours.
3. Override log. Each gate override records who, why and the contract change; review monthly.
4. Coverage audit. Quarterly, compare the route list with the contract.
5. Reconcile with Google. Crawl Stats may differ slightly from your logs;^[49] large gaps mean a missing log source (CDN, second origin).
6. Property settings. Monthly, compare each property’s Search generative AI control with the recorded decision; a child property can override its parent, and the top-level domain property’s owners are notified.^[35]

flowchart TB
  A["Index or traffic drop"] --> B{"Site-wide or one template?"}
  B -- "template" --> D["Live test vs indexed, 5 URLs"]
  B -- "site-wide" --> C{"robots.txt, 5xx or DNS?"}
  D --> E{"Defect in our HTML?"}
  E -- no --> G["Align canonical signals"]
  E -- yes --> F["Roll back or hot-fix"]
  C -- yes --> F
  C -- no --> M{"Manual action or security?"}
  M -- yes --> R["Fix; Request Review"]
  M -- no --> U["Update or demand: to SEO Operations"]
  G --> H["Request indexing; resubmit sitemap"]
  F --> H
  H --> I["Monitor; post-incident review"]
  R --> I
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  class A risk
  class B,C,E,M decision
  class F,G,H,I,R core
  class U neutral

1. Classify (15 minutes). One template, so not robots.txt or an outage. Google’s traffic-drop guide separates site-wide technical issues from page-level ones such as “a misplaced noindex tag”.^[7] (A manual action would instead be fixed and then submitted with Request Review in the Manual actions report.^[51])
2. Confirm (30 minutes). URL Inspection on five product URLs: the indexed version may look fine while the live test reports noindex.^[52] curl -s https://shop.example.com/p/123 | grep -i robots confirms it in server HTML.
3. Contain (same day). Roll back or hot-fix; run the contract on production; check the CDN added no X-Robots-Tag.
4. Recover (days to weeks). Resubmit the product sitemap; request indexing for the most important URLs (daily limit per property).^[52] Recrawling the rest can take months.^[5] Ship no “helpful” extra changes meanwhile.
5. Learn (two weeks). Post-incident review: add the product template to the contract, a production smoke test after deploys and an alert on indexed count per sitemap. Check that the gate runs on dependency-only pull requests: this upgrade changed a lockfile, which a paths filter would have skipped (pipeline job). Nine days to detect is the number to beat.

A maintenance rule served 503 for every path, including /robots.txt. Google says not to return 503 for robots.txt “because this blocks all crawling”. Exclude it, keep 503 on pages for a day or two at most, and use an indexable 200 placeholder for longer closures.^[50]

Example status: syntax-checked (bash -n), not executed against a live host.

# Smoke test after any maintenance-mode or CDN rule change (illustrative host).
code=$(curl -s -o /dev/null -w '%{http_code}' https://shop.example.com/robots.txt)
if [ "$code" != "200" ]; then
  echo "robots.txt returned $code: crawling will pause or rules will be ignored" >&2
  exit 1
fi
echo "robots.txt OK"

Use it when organic search matters and the site has many templates, ships weekly or faster, renders with JavaScript, or faces a migration.

Do not build it when the site is small, rarely changes and is well linked: a checklist and a monthly look at Search Console are enough.^[20][17] Nor without template owners: unowned checks become ignored red builds. Hidden costs: current contracts, override reviews, the Search Console property and alerts.

flowchart TB
  A{"Is organic search a material channel?"} -- no --> N["Checklist before releases; monthly console review"]
  A -- yes --> B{"More than a few templates or frequent releases?"}
  B -- no --> N
  B -- yes --> S["Base: contract in CI + production smoke test"]
  S --> C{"JavaScript renders indexable content?"}
  C -- yes --> R["Add rendered checks in CI"]
  C -- no --> D{"Over 10k changing URLs or a migration ahead?"}
  R --> D
  D -- yes --> L["Add verified logs + crawl-waste review"]
  D -- no --> K["Keep the base; revisit when either changes"]
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  class A,B,C,D decision
  class R,L,S core
  class N,K neutral

Which failures block a release? The Atlas’s rule: block what is silent, broad and slow to recover.

This Atlas’s model, not an industry standard; move up only when the exit criterion is met.

Quick wins: a robots.txt monitor, clean sitemaps, authenticated staging, a recorded AI-control setting.

Pre-release (routes, templates, rendering or CDN rules changed)

• Contract passes on the preview, including new templates
• robots.txt expectations pass; file returns 200, under 500 KiB
• One absolute, self-referencing canonical per indexable page in server HTML
• No noindex (HTML or header) on indexable templates; mobile and desktop identical^[53]
• Removed pages 404/410; moved pages one-hop 301/308
• Sitemaps list only canonical, indexable URLs with honest lastmod
• Rendered check passes; HTML under 2MB
• Preview host requires authentication

Migration

• Complete old-to-new URL map, including images and PDFs with traffic
• Every mapped URL tested: one hop, permanent, correct target
• Canonicals, hreflang, internal links and sitemaps updated
• New property verified; Change of Address for domain moves; redirects kept at least a year^[48]

Weekly review

• Indexed counts per sitemap stable
• Crawl Stats host status, 5xx/429 share, response time
• Canonical agreement sample; Manual actions and Security Issues empty
• Search generative AI setting per property unchanged; owner of the decision recorded^[35]
• Monitors and exports ran in the last 24 hours

Incident: use the worked runbook.

Search contract entry (per template), for review alongside the JSON contract:

Template:            product detail
Example URLs:        /p/123-trail-runner, /p/987-rain-jacket
Status:              200; 404 for unknown IDs (server-side)
Indexable:           yes
Canonical:           self, absolute, https://www.example.com/p/<id>-<slug>
Duplicates:          ?ref=, ?utm_*, /print -> canonical to clean URL
Sitemap:             products.xml, lastmod = last price/stock/content change
hreflang:            en-GB, en-US, x-default; reciprocal
Structured data:     Product (gate checks the type; review required properties by hand)
Rendering:           server HTML contains title, description, price, links
Owner:               web team "catalogue"; reviewer: technical SEO

SEO change review (pull-request template section).

Search impact (complete if routes, <head>, rendering, redirects, robots,
sitemaps or CDN headers change):
- Templates affected:
- Indexability change intended? (yes/no; which URLs, why):
- Canonical or URL changes (attach redirect map if any):
- robots.txt change and expectation tests updated? (link):
- Contract updated in this PR? (yes/no):
- Rollback: how to revert within one hour:
- Reviewer (technical SEO):

Redirect map (CSV header and rows).

old_url,new_url,status,owner,notes
https://old.example.com/shoes/trail,https://www.example.com/p/123-trail-runner,301,catalogue,
https://old.example.com/blog/2019/fit,https://www.example.com/guides/fit,301,content,merged

Search incident review.

Incident:        <title>                      Date of release: <YYYY-MM-DD>
Detected by:     <alert / person / report>    Time to detect: <hours or days>
Scope:           <templates, URL count, engines>
Mechanism:       <noindex / robots / canonical / status / rendering / other>
Timeline:        deploy -> first crawl of defect -> detection -> fix -> recovery
Why no check:    <no contract entry / advisory only / path filter / monitor down>
Actions:         <new check>, <owner>, <due date>

Which control for which goal?

For neighbouring pages, see how SEOps relates to other Ops.

Is SEOps a job title? Not in any source found for this page. The advert cited here is for a “Lead Technical SEO Manager” and lists CI quality gates and monitoring among the duties.^[3]

Does passing every check mean we will rank? No; checks establish eligibility. Meeting every requirement “doesn’t mean that Google will crawl, index, or serve its content”.^[30]

Should we exclude the site from AI Overviews? That is a business decision with a traffic cost, made in SEO Operations; SEOps makes sure the Search generative AI control matches it.^[35]

Do we need IndexNow? Not for Google, which is not a listed participant (October 2026); use sitemaps with honest lastmod.^[31][10] For engines such as Bing it announces changes; a 200 only means “received”.^[31]

How do we get staging out of Google? Authentication. Google drops URLs that return 4xx, including 401 and 403, over time;^[15] for urgent cases use Search Console removals, as Google’s noindex guide suggests.^[5]

Recent changes to track: Google’s crawler documentation moved to a crawling-infrastructure section, with IP-range files under developers.google.com/crawling/ipranges/ (2026);^[45] the 2MB HTML fetch limit was explained in March 2026;^[12] generative-AI performance reports and the Search generative AI control reached all websites by 31 August 2026.^[54][35]

Active and uncertain: Google mentions browser agents that read the DOM and accessibility tree;^[34] other AI products publish their own crawler rules. Edge platforms that rewrite HTML, headers and redirects are invisible to application tests, so run production checks against the CDN hostname.

• What it is: an informal label; here, technical SEO as engineering, protecting discover, crawl, render and index.
• Right control: robots.txt = crawling; noindex = indexing (page must be crawlable); canonical and redirects = duplicates; 404/410 = gone; 503 = briefly down, never for robots.txt.
• Checks: status, noindex, canonical, robots expectations, sitemap hygiene, rendered content, HTML under 2MB.
• Metrics: contract pass rate and coverage, sitemap hygiene, canonical agreement, crawl waste, crawler errors, time to detect.
• Watch-outs: a Googlebot group ignores * rules; JavaScript can add but not reliably undo noindex; staging needs authentication; verify Googlebot by googlebot.com names only.

After this page you can explain the label and its scope here; choose the right control for a goal; design a contract and gate; diagnose an indexing regression; judge whether a site needs SEOps.

“SEOps” is not a discipline with a definition; it is a useful name for a habit: treating the few controls that decide whether search engines can find, crawl, render and index a site as code, written down, tested before release and watched afterwards. The failures they prevent are cheap to catch and slow to repair. SEOps cannot make a page rank. Get the four gates right on every release, measure the contract rather than the rankings, keep a runbook for the day it fails, and leave what to publish to the people who run the SEO programme.

All sources were opened on 2026-10-23; dates are the “last updated” dates shown then. Claim-by-claim notes: research/seops-ledger.md.

Further reading

• Fix canonicalization issues — Google Search Central, updated 2026-08-21.

Changelog

• 2026-10-23 — First version: term status confirmed as ambiguous; engineering scope per coverage matrix §3.2; three labs executed (robots matcher, contract gate, verified-crawler log summary), one lab syntax-checked only; 48 sources checked.
• 2026-10-23 — Revision r1 (independent review): crawler verification, contract gate, robots matcher and pipeline fixed and re-executed; rendered check added; Search generative AI control added; Python 3.11+; three diagrams redrawn; cuts to stay in the length band.
• 2026-10-23, post-score polish: render-queue, AI-control and crawler-DNS wording aligned with sources; Lab 3 sample log printed; contract template matched to the gate.