---
title: "SEOps"
slug: seops
category: business
term_status: ambiguous
evidence_status: verified
summary: "SEOps is an informal label. This page uses it for technical SEO run as engineering: crawlability, indexability and canonical signals written down as a contract, tested before every release and monitored after it."
audience: [Web and platform engineers, Technical SEOs, Engineering and marketing leaders, Beginners]
last_verified: 2026-10-23
key_takeaways:
  - "\"SEOps\" has no standard definition, and no source found for this page uses that spelling. A vendor guide uses \"SEO operations\" for running an SEO programme; a job advert describes CI quality gates for technical SEO. This page takes the engineering reading; SEO Operations covers the programme."
  - "Search visibility can be tested at four controllable gates: discovery, crawl, render and index. Ranking is not something a release can test; the gates are."
  - "Most search-breaking releases are a handful of lines: a noindex left in a template, a robots.txt rule, a staging canonical, a 5xx robots.txt, a JavaScript-only link. Each can be caught with a cheap automated check."
  - "robots.txt controls crawling, not indexing; noindex only works on a page that crawlers are allowed to fetch; canonical tags and sitemaps are hints that Google may overrule."
  - "Pre-release checks catch what you broke; Search Console and verified-Googlebot logs catch what Google actually did. You need both, because Google's view lags by days to weeks."
  - "Measure the contract (pass rate, canonical agreement, sitemap hygiene, crawl waste), not rankings, and keep a runbook for the day a release removes pages from the index."
related: [seo-operations, webops, releaseops, contentops, observabilityops, marketingops, devops, incidentops]
---

## What SEOps is

### In 30 seconds

**SEOps, as this page uses the word, is technical SEO run as an engineering property: the rules that let search engines
discover, crawl, render and index a site are written down, tested in the delivery pipeline and monitored in
production.** A release that adds `noindex` to every product page, points canonicals at the staging host or makes
`robots.txt` return a server error can remove a site from search for weeks. SEOps treats that as a defect.

**In one line: search visibility is a testable property of every release.** Rankings are not; whether each page is
reachable, crawlable, renderable, indexable and correctly canonicalised is.

::: note Not SecOps, not "SE Ops", and not the whole of SEO
- **SecOps** is security operations; see [SecOps](../secops/).
- **"SE Ops"** appears in job adverts for Sales Engineering Operations.[^seopsjob] Unrelated.
- **SEO as a programme** (keyword research, content production, intake, reporting) belongs to
  [SEO Operations](../seo-operations/). The split is an **editorial distinction** of this Atlas, not an industry
  standard; see [how SEOps relates to other Ops](#how-seops-relates-to-other-ops).
:::

**For** web and platform engineers, technical SEOs, and leaders who need search traffic to survive site changes.

### At a glance

| Item | Summary |
|---|---|
| Term status | **Ambiguous.** No standard, body or book defines "SEOps", and no source found for this page uses that spelling. A vendor guide uses "SEO operations" for running an SEO programme;[^gradial] a 2026 technical-SEO job advert asks for "automated testing, monitoring, dashboards, CI/CD quality gates, proactive alerting, and validation scripts that detect technical SEO issues and prevent regressions from reaching production".[^zoomjob] This page takes the second reading. |
| Goal | Wanted pages stay discoverable, crawlable, renderable, indexable and self-canonical; others are excluded deliberately; regressions are caught early. |
| In scope | robots.txt, robots directives, status codes, redirects, canonicals, sitemaps, hreflang, rendering, structured-data validity, crawler logs, Search Console, CI checks, spam-policy compliance, migrations. |
| Out of scope | Keyword research, content production and SEO reporting → [SEO Operations](../seo-operations/); measuring and fixing page speed → [WebOps](../webops/); release strategy and gate policy → [ReleaseOps](../releaseops/); editorial workflow → [ContentOps](../contentops/). |
| Prerequisites | HTTP basics and how your site is built and deployed ([WebOps](../webops/), [DevOps](../devops/)). |
| Key outputs | Search contracts per template; tested robots.txt and sitemaps; pre- and post-release checks; dashboards from Search Console and verified logs; runbooks. |

### The problem it solves

Google Search works in three stages, crawling, indexing and serving, "and not all pages make it through each
stage".[^howsearch] The controls that decide the first two live in code and server configuration (status codes,
`robots.txt`, robots meta tags, canonicals, redirects, sitemaps, rendered HTML), which engineers change weekly for
reasons unrelated to search.

| Without SEOps | With SEOps |
|---|---|
| An audit finds a template has carried `noindex` for months | A CI check fails the pull request that added it |
| Canonicals point to the staging host after an upgrade | A post-deploy smoke test compares canonicals with production |
| `robots.txt` is a hand-edited file nobody owns | It is generated, reviewed and tested against expected outcomes |
| A migration ships an untested redirect spreadsheet | Every old URL is requested and must return one 301 to its target |
| "Did Google see it?" is a guess | Search Console API and verified-Googlebot logs answer it |

The cost is asymmetric. Breaking indexability takes one deploy; recovery waits for recrawling, which Google says "may
take months" for some pages.[^noindex] A `robots.txt` server error stops crawling for 12 hours;[^robotsspec] a
misplaced `noindex` is quieter, "meaning there would be a slower drop in traffic".[^trafficdrops] Slow, quiet failures
are what tests and monitors are for.

### What SEOps is not

- **Not ranking manipulation.** SEOps keeps releases inside Google's spam policies (cloaking, doorways, sneaky
  redirects and more).[^spam]
- **Not page-speed engineering.** Core Web Vitals matter to search, but measuring and fixing them is [WebOps](../webops/) work.
- **Not the SEO programme.** What to publish, for which searches, belongs to [SEO Operations](../seo-operations/) and
  [ContentOps](../contentops/).
- **Not Google-only.** Many controls are open standards (RFC 9309, sitemaps, HTTP); Google-specific behaviour is
  labelled as such.

## Where SEOps comes from and where it sits

### Origins and evolution

The *label* has no origin story; the *practice* grew out of technical SEO as sites became continuously delivered
software. Milestones in what it relies on:

- **Robots Exclusion Protocol:** Martijn Koster, 1994; RFC 9309 since September 2022, whose rules "are not a form of
  access authorization".[^rfc9309]
- **Sitemaps:** introduced in 2005; in June 2023 Google deprecated sitemap "ping" and stressed accurate `lastmod`.[^ping]
- **JavaScript rendering:** Google calls dynamic rendering "a workaround and not a long-term solution" and recommends
  server-side rendering, static rendering or hydration.[^dynrender]
- **Byte limits:** in March 2026 Google explained that bytes past an HTML URL's first 2MB "aren't fetched, they aren't
  rendered, and they aren't indexed".[^bytes]

### Glossary and mental model

| Term | Meaning | Why it matters to a release |
|---|---|---|
| Crawling | A crawler (Googlebot) fetches a URL | Blocked by `robots.txt`, server errors, rate limits |
| Rendering | Executing JavaScript to get the final HTML | JavaScript-only content and links may be delayed or missed |
| Indexing | Analysing and storing a page; choosing a canonical | Blocked by `noindex`, soft 404s, duplication |
| `robots.txt` | Host-root file with crawl rules per user agent | Controls crawling, not indexing[^robotsintro] |
| `noindex` | Meta tag or `X-Robots-Tag` header | Only seen if the page can be crawled[^noindex] |
| Canonical | The representative URL among duplicates | Your `rel="canonical"` is "a hint, not a rule"[^canonwhat] |
| Soft 404 | A 200 response that looks like an error page | Reported as an error, wastes crawl[^httpstatus] |
| Sitemap | File listing URLs you want indexed | A hint; Google ignores `priority` and `changefreq`[^sitemapsbuild] |
| Crawl budget | URLs Google can and wants to crawl on a host | Matters mainly for large or fast-changing sites[^crawlbudget] |
| Search contract | *This Atlas's term:* the expected search state per page template | What SEOps automates |

**Mental model: four gates you control, one you do not.** A URL is *discovered*, *crawled*, *rendered* and *indexed*
before it can be *served*. Releases can break the first four; none can force the fifth.

::: figure The four controllable gates between a URL and a search result
```mermaid
flowchart TB
  A["Discover: links, sitemap"] --> B{"Crawl allowed and served?"}
  B -- "robots.txt disallow, 5xx, 429" --> X1["Not crawled"]
  B -- yes --> C{"Rendered HTML has content and links?"}
  C -- "JS-only links, content after 2MB" --> X2["Partly seen"]
  C -- yes --> D{"Indexable and canonical?"}
  D -- "noindex, soft 404, duplicate" --> X3["Not indexed"]
  D -- yes --> E["Eligible for serving"]
  E --> F["Ranking: not testable by a release"]
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  class A,E core
  class B,C,D decision
  class X1,X2,X3 risk
  class F neutral
```
Text equivalent: A URL is discovered through links or a sitemap. Gate one asks whether crawling is allowed and served; a
robots.txt disallow, server error or 429 stops it. Gate two asks whether rendered HTML contains the content and links;
JavaScript-only links or content past the 2MB limit leave it partly seen. Gate three asks whether it is indexable and
canonical; noindex, a soft 404 or duplication stop it. A page that passes is eligible for serving, where ranking happens
outside a release's control.
:::

### How SEOps relates to other Ops

| Neighbour | Shared ground | SEOps owns | The neighbour owns |
|---|---|---|---|
| [SEO Operations](../seo-operations/) | Search Console, spam policies | Crawl and index engineering, CI checks, log analysis | Keyword research, content production, intake, prioritisation, reporting |
| [WebOps](../webops/) | Caching, CDNs, Core Web Vitals, robots.txt delivery | What these mean for crawlers and indexing | Measuring and fixing speed; serving the site safely |
| [ReleaseOps](../releaseops/) | Release gates, rollback | The search checks that feed a gate | Gate policy, progressive delivery, release decisions |
| [ContentOps](../contentops/) | Templates, metadata fields | Template-level search contract | Editorial workflow and content lifecycle |
| [ObservabilityOps](../observabilityops/) | Logs, alerting | Search-specific signals and alerts | Telemetry pipelines and tooling |
| [MarketingOps](../marketingops/), [AdOps](../adops/) | Landing pages | Organic indexability of landing pages | Campaigns and paid search |

::: figure Where SEOps sits among its neighbours
```mermaid
flowchart TB
  SO["SEO Operations: topics"] --> CO["ContentOps: editorial"]
  CO --> SE["SEOps: indexable?"]
  WO["WebOps"] -- serving --> SE
  SE -- checks --> RO["ReleaseOps"]
  SE -- alerts --> OB["ObservabilityOps"]
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  class SE core
  class SO,CO,RO,WO,OB neutral
```
Text equivalent: SEO Operations decides what to publish and ContentOps produces it through its editorial workflow; the
pages reach SEOps, which asks whether search engines can reach and index them, while WebOps supplies serving behaviour. SEOps feeds checks to ReleaseOps' gate
policy and signals to ObservabilityOps' alert routing.
:::

## Principles and operating model

### Charter and principles

No official SEOps charter exists. This is **this Atlas's proposal**:

> **Mission.** Every release keeps the pages we want in search eligible for it, keeps the others out by the right
> mechanism, and tells us within one working day when Google's view diverges from our intent.
> **Non-goals.** Choosing topics or keywords; promising rankings; manipulating search systems.

1. **Write the intent down** as a *search contract* per template.
2. **Right control for the job.** Crawling is controlled by `robots.txt`, indexing by `noindex`, duplication by
   redirects and canonicals; mixing them causes most failures.[^robotsintro][^noindex][^canonhow]
3. **Test what you ship, monitor what Google does.** Only Search Console and crawler logs show Google's behaviour.
4. **Prefer server-side truth.** JavaScript may fail to remove a `noindex` and should not change the canonical.[^jsbasics]
5. **Fail closed for exclusions.** Non-production hosts are never indexable (authentication, not robots rules).
6. **Spam policies by design.** Programmatic pages, redirects and bot-specific responses are reviewed before they ship.[^spam]
7. **Proportion.** Google says well-linked sites of about 500 pages or fewer may not need a sitemap;[^sitemapsabout]
   they need a checklist, not a platform.

### Building blocks

| Control | Lives in | Does | Does not |
|---|---|---|---|
| HTTP status code | Server, framework, CDN | 200 may be indexed; 301/308 strong canonical signal; 4xx removed; 5xx and 429 slow crawling[^httpstatus] | Guarantee indexing (2xx is not enough) |
| `robots.txt` | Host root, per protocol, host and port[^robotsspec] | Allows or disallows crawling per user agent | Remove pages from results or protect anything[^rfc9309] |
| `noindex` (meta or `X-Robots-Tag`) | HTML head or response header | Drops the page from results once crawled[^noindex] | Work on a page blocked by `robots.txt` |
| `rel="canonical"` (HTML or `Link` header) | Head or header | Strong hint for the representative URL[^canonhow] | Override Google's choice in every case |
| Redirect (301/308) | Server or CDN | Moves signals to the target; target shown in results[^redirects] | Work when chained endlessly; Googlebot follows up to 10 hops[^httpstatus] |
| XML sitemap | Generated file, listed in `robots.txt` or Search Console | Lists canonical URLs, optional `lastmod`[^sitemapsbuild] | Guarantee crawling; `priority` and `changefreq` are ignored |
| hreflang | Head, headers or sitemap | Links language and region versions[^hreflang] | Work one-way |
| Structured data (JSON-LD) | Page | Describes the page; may enable rich results[^sdpolicies] | Guarantee a rich result; describe content users cannot see |
| Rendered HTML | Browser-like renderer | Exposes content and `<a href>` links[^jsbasics] | Include content after a 2MB cut-off[^bytes] |

### Roles, ownership and RACI

SEOps is rarely a team; it is a set of responsibilities.

| Activity | Web engineers | Technical SEO | Platform / SRE | Engineering lead |
|---|---|---|---|---|
| Write and update search contracts | C | **A/R** | I | I |
| Implement controls in templates and server | **R** | C | C | **A** |
| Maintain CI checks and fixtures | **R** | C | C | **A** |
| Generate and test `robots.txt` and sitemaps | **R** | **A** | C | I |
| Review URL, rendering and redirect changes | C | **A/R** | C | I |
| Monitor Search Console and crawler logs | I | **A/R** | C | I |
| Respond to a search incident or run a migration | **R** | **A** | R | C |

R = responsible, A = accountable, C = consulted, I = informed; the SEO-programme team is consulted on contracts and
migrations. The technical SEO owns *what* the contract says; engineering owns the code *meeting* it.

### Lifecycle: Day 0, Day 1, Day 2

- **Day 0 — design.** URL scheme, rendering strategy (Google calls server-side or pre-rendering "still a great
  idea"[^jsbasics]), canonical policy, parameter handling, indexable templates; write the contracts.
- **Day 1 — launch or migration.** Generate `robots.txt` and sitemaps, run the full contract on authenticated staging,
  map every old URL, verify the Search Console property, submit sitemaps.
- **Day 2 — every release.** Contract checks in CI and after deploy; monitoring; monthly review of crawl waste,
  canonical disagreements and exclusions.

### Operating model and governance

1. **A change-review trigger.** Changes to routing, rendering, `robots.txt`, redirects, canonicals, sitemaps, CDN
   headers or a template's `<head>` need a technical-SEO reviewer (code-owners file).
2. **A gate policy.** Which failures block and which warn; the policy belongs to [ReleaseOps](../releaseops/), SEOps
   supplies the checks and severities (see the [decision framework](#decision-framework)).
3. **A weekly signal review** ([checklist](#checklists)).

## Architecture and workflow

### Reference architecture

Checks sit where each failure becomes visible: build, preview, production and Google's reports.

::: figure Reference architecture: where search checks run
```mermaid
flowchart TB
  subgraph PRE["Before release"]
    A["Pull request"] --> B["Unit: robots.txt and sitemap tests"]
    B --> C["Preview deploy, password-protected"]
    C --> D["Contract check: status, noindex, canonical"]
    D --> E["Rendered check: headless browser"]
  end
  subgraph POST["After release"]
    F["Production smoke test"] --> G["Synthetic monitor every hour"]
    H["Access logs: verified Googlebot"] --> I["Search signals dashboard"]
    J["Search Console API"] --> I
    G --> I
  end
  E --> F
  I --> K["Alert to technical SEO on-call"]
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  class B,D,E,F,G core
  class I,K decision
  class A,C,H,J neutral
```
Text equivalent: Before release, a pull request runs robots.txt and sitemap unit tests, deploys a password-protected
preview, runs the contract check (status, noindex, canonical), then a headless-browser rendered check. After release, a
production smoke test and an hourly monitor run the same contract; they, verified-Googlebot logs and the Search Console
API feed a dashboard that alerts the technical-SEO on-call person.
:::

| Layer | Catches | Cannot catch | Latency |
|---|---|---|---|
| Unit tests (robots, sitemap generator) | Wrong rules, staging URLs in sitemaps | Template output | Seconds |
| Contract check on preview | `noindex`, wrong canonical or status in server HTML | JavaScript changes | Minutes |
| Rendered check | Content and links missing after rendering | Google's real render queue | Minutes |
| Production smoke and monitor | CDN header differences, config drift, robots.txt 5xx | Google's decisions | Up to an hour |
| Verified crawler logs | What Googlebot requested and received | Indexing decisions | Hours |
| Search Console reports and API | Google-selected canonical, index status, manual actions | Anything in real time | Days |

**Trust boundaries.** Protect previews with authentication, not `robots.txt`, which is not access control and
"exposes them publicly".[^rfc9309] The Search Console credential needs only the `webmasters.readonly` scope.[^inspectapi]

### End-to-end workflow

::: figure A template change from pull request to Google's view
```mermaid
sequenceDiagram
  autonumber
  participant Dev as Engineer
  participant CI as CI and preview
  participant Prod as Production
  participant G as Google
  Dev->>CI: Pull request
  CI-->>Dev: Checks fail: noindex
  Dev->>CI: Fix pushed
  CI->>Prod: Release and smoke test
  G->>Prod: Crawls changed pages
  Note over Prod,G: Team reads index state via API and logs
```
Text equivalent: An engineer's pull request changes a template. CI runs the contract and rendering checks on a preview;
they fail because the product template now carries noindex, blocking the merge. After a fix, the release reaches
production and a smoke test runs the contract on live URLs. Over the following days Google crawls the changed pages,
and the Search Console API and logs report index state back to the team.
:::

**Happy path:** the contract passes everywhere and, a week later, URL Inspection shows Google's canonical equals the
declared one. **Exception paths:** a failed check blocks the merge unless a reviewer records an override with the
contract updated in the same change; a failed production smoke test is an incident
([worked runbook](#worked-runbook-a-release-adds-noindex-to-product-pages)); a later "Duplicate, Google chose different
canonical than user" is a signal conflict, not an outage.[^pagereport]

## Practices, standards and tools

### Core practices and playbooks

| Practice | Trigger | Output | Verification |
|---|---|---|---|
| Search contract per template | New template or route | Contract entry | Passes on preview and production |
| robots.txt as code | Any rule change | Generated file plus expectation tests | Suite passes; file under 500 KiB[^robotsspec] |
| Sitemaps from the source of truth | Publish, unpublish, URL change | Files of at most 50,000 URLs or 50MB[^sitemapsorg] | Every entry passes the contract |
| Status-code discipline | Removed or empty pages | 404/410 for gone pages and empty filter combinations[^faceted] | No soft 404s reported |
| Rendering parity | Framework or hydration change | [Rendered check](#reference-implementation) | Same title, canonical and robots tags after rendering; links present |
| Redirect hygiene | URL changes, migrations | Redirect map | Each old URL: one 301/308 to its target |
| Crawler verification | Log analysis, bot rules | Verified crawler report | `googlebot.com` name, forward DNS match ([Lab 3](#lab-3-find-out-what-googlebot-really-requested)) |

#### Playbook: robots.txt as code

1. Generate `robots.txt` per environment from a reviewed source; previews also require authentication.
2. Keep **expected outcomes** (URL, user agent, allowed or not) and run them in CI ([Lab 1](#lab-1-test-robotstxt-against-expected-outcomes)).
3. A group naming `Googlebot` is the only one Googlebot follows; it is not merged with `*`.[^robotsspec] Changing a
   rule in one group but not the other is a classic regression.
4. Monitor the production file. A 4xx (except 429) means "no crawl restrictions"; a 5xx stops crawling for 12 hours,
   then Google uses the last good copy for up to 30 days.[^robotsspec]
5. Never use it to hide pages from results, to protect anything, or to canonicalise duplicates.[^robotsintro][^canonhow]

#### Playbook: canonicals and JavaScript

- One absolute, self-referencing canonical per indexable page, in the server HTML; internal links, sitemaps, hreflang and
  redirects all point to the same URL, because signals stack.[^canonhow] HTTPS only: a bad certificate or an
  HTTPS-to-HTTP redirect makes Google prefer HTTP "very strongly".[^canonhow]
- Google queues 200 pages for rendering (unless a robots meta tag or header says `noindex`), and a page "may stay on this queue
  for a few seconds, but it can take longer than that". Use `<a href>` links and History API routes, not `#` fragments;
  give unknown IDs a real 404 (or a redirect to one, or an injected `noindex`); never let JavaScript change the
  canonical; fingerprint asset names, as the renderer may ignore caching headers.[^jsbasics] The renderer is stateless,
  so content gated on local or session storage may not render.[^bytes]

#### Core Web Vitals and page experience

Good Core Web Vitals, "along with other page experience aspects, aligns with what our core ranking systems seek to
reward", says Google (Largest Contentful Paint, LCP, within 2.5 s; Interaction to Next Paint, INP, under 200 ms; Cumulative
Layout Shift, CLS, under 0.1).[^cwv] SEOps only watches the Search Console report by template; measuring and fixing
belong to [WebOps](../webops/#playbook-fixing-core-web-vitals). One crawler point stays here: heavy inline scripts or
base64 images can push content or JSON-LD past Googlebot's 2MB cut-off.[^bytes]

### Methods, standards and frameworks

| Item | Kind | Governs |
|---|---|---|
| RFC 9309 Robots Exclusion Protocol | IETF standard, 2022[^rfc9309] | robots.txt syntax, matching, caching, errors |
| Sitemaps protocol 0.9 | Industry protocol[^sitemapsorg] | Sitemap format, limits, `lastmod` |
| RFC 6596 canonical link relation | IETF standard; Google cites it[^canonhow] | `rel="canonical"` semantics |
| Schema.org | Community vocabulary founded by Google, Microsoft, Yahoo and Yandex[^schemaorg] | Structured data types |
| Google Search Essentials, spam policies | Vendor policy[^essentials][^spam] | Eligibility and penalties in Google |
| Google crawling documentation | Vendor behaviour[^robotsspec][^httpstatus] | How Googlebot applies the standards |
| IndexNow | Multi-engine protocol[^indexnow] | Push notification of changed URLs |
| Search contract, four gates, maturity model | This Atlas's proposals | Organising the work |

Engines can differ from the RFC: it says a crawler "MUST assume complete disallow" when `robots.txt` is unreachable,
while Google documents a 12-hour stop, then up to 30 days on its last cached copy.[^rfc9309][^robotsspec]

### Tooling landscape

| Tool category | Job | Examples (October 2026) | Selection criteria |
|---|---|---|---|
| Search engine consoles | Index status, canonical choice, crawl stats, manual actions | Google Search Console and its API; Bing Webmaster Tools | Property verification, quotas, data lag |
| Site crawlers | Crawl like a bot; find status, canonical and directive issues at scale | Commercial and open-source crawlers | JavaScript rendering, scheduling, crawl diffs |
| CI test runners | Contract and robots tests per change | Your own scripts ([reference implementation](#reference-implementation)); Lighthouse CI for page audits[^lhci] | Runs on previews; clear failures |
| Headless browsers | Rendered-HTML checks ([rendered check](#reference-implementation)) | Puppeteer, Playwright | Recent Chromium, as Google uses[^howsearch] |
| Log analysis | Verified crawler behaviour, crawl waste | Log pipelines, warehouse queries, scripts | Joins logs to templates; DNS verification |
| Structured data validators | Markup syntax and eligibility | Rich Results Test, Schema.org validator | Feature-specific rules |

Check what a "robots.txt tester" implements: Python 3.8's `urllib.robotparser`, tested for this page, used first match
and treated `*` literally; current CPython sources use longest match.[^pyrobot]
Judging third-party "visibility" and AI-search tools is programme work: see [SEO Operations](../seo-operations/#tooling-landscape).

### Automation and AI

- **Automate the deterministic parts** (contracts, robots expectations, redirect maps, sitemaps) in CI. Language models
  can cluster thousands of excluded URLs for triage; they should not write robots rules or canonicals without tests.
- **Generated page sets** go through the [spam-policy review](#quality-security-privacy-and-compliance).[^spam]
- **AI features in Google Search have a property setting (as of October 2026).** Besides being indexed and
  snippet-eligible, a site must be included via Search Console's **Search generative AI control**.[^aiguide] Its values
  are Include, Exclude and Inherit (from a parent property; the default where one exists, else Include). It governs
  links and content in AI Overviews, AI Mode and Discover's generative AI features; Google says it is not a ranking
  signal elsewhere and does not affect training (Google-Extended's job), while `noindex` leaves Search
  altogether.[^genaictl][^crawlers] Whether to exclude is a business decision for
  [SEO Operations](../seo-operations/#ai-features-and-zero-click-searches); SEOps records the value per property and
  checks it has not drifted. `llms.txt` plays no part in Google Search.[^aiguide]

## SEOps in practice

### Reference implementation

**Assumptions.** CI that can deploy password-protected previews; Python 3.11+ (3.10 reached end of life on
2026-10-01[^pyversions]) and, for part 3, Node.js 20+; Google as the main search engine.

**1. The search contract.** One entry per representative URL of each template; `self` means "canonical equals this path
on the production host".

_Example status: syntax-checked by the validator and executed against a local test site on 2026-10-23._

```json
{
  "base": "http://127.0.0.1:8765/",
  "canonical_host": "https://shop.example.com/",
  "agent": "googlebot",
  "sitemap": "/sitemap.xml",
  "pages": [
    {"path": "/", "status": 200, "indexable": true, "canonical": "self"},
    {"path": "/shoes", "status": 301, "location": "/shoes/"},
    {"path": "/shoes/", "status": 200, "indexable": true, "canonical": "self"},
    {"path": "/shoes/trail-runner/", "status": 200, "indexable": true,
     "canonical": "self", "structured_data": ["Product"]},
    {"path": "/account/", "status": 200, "indexable": false},
    {"path": "/no-such-page/", "status": 404}
  ]
}
```

Optional keys: `robots_file` (judge indexability by a local file, normally the generated production robots.txt) and
`structured_data` (JSON-LD types that part 3 requires).

**2. The contract checker.** It requests each URL without following redirects, reading 2MB like Googlebot.[^bytes] It
honours `X-Robots-Tag` only when unscoped or scoped to its crawler (`bingbot: noindex` does not count[^robotsmeta]),
takes canonicals from `<head>`, where Google accepts them,[^canonhow] and the `Link` header, which must agree. It applies
`robots.txt` with [Lab 1](#lab-1-test-robotstxt-against-expected-outcomes)'s matcher, follows a (gzipped) sitemap
index, and fails on an unreadable or empty sitemap. `SEO_GATE_AUTH` carries a preview credential to the base host only.

_Example status: executed on 2026-10-23 with Python 3.11.17 (and 3.8.10): [Lab 2](#lab-2-run-the-contract-gate-against-a-local-site)
plus tests against a local Basic-auth server (gzipped index, empty and 3MB sitemaps, body-only canonical, conflicting
`Link` header, crawler-scoped `X-Robots-Tag`)._

```python
"""seo_gate.py: check a running build against a search contract (JSON).
Fails (exit 1) when a page's status, indexability or canonical differs from
the contract, or when a sitemap (or sitemap index) lists a URL that is not
indexable and self-canonical, or lists none. Standard library only; does not
execute JavaScript.
Env SEO_GATE_AUTH (optional): Authorization header value for a protected
preview, e.g. "Basic <base64>"; sent only to the contract's base host.
Usage: python3 seo_gate.py contract.json
"""
import gzip
import json
import os
import re
import sys
import urllib.error
import urllib.request
import xml.etree.ElementTree as ET
from html.parser import HTMLParser
from urllib.parse import urljoin, urlsplit

import robots_check

UA = "seo-gate/1.1 (+https://example.com/seo-gate)"
PAGE_LIMIT = 2 * 1024 * 1024        # Googlebot reads the first 2MB of HTML
SITEMAP_LIMIT = 50 * 1024 * 1024    # sitemaps.org: 50MB uncompressed
HEAD_TAGS = {"html", "head", "title", "meta", "link", "base", "script",
             "style", "noscript", "template"}
SM = "{http://www.sitemaps.org/schemas/sitemap/0.9}"


class NoRedirect(urllib.request.HTTPRedirectHandler):
    def redirect_request(self, *args, **kwargs):
        return None  # report the first response, do not follow


class HeadParser(HTMLParser):
    """Robots meta tags anywhere; canonical links only inside <head>."""

    def __init__(self, agent):
        super().__init__()
        self.names = ("robots", agent)
        self.in_head, self.robots, self.canonicals = True, [], []

    def handle_starttag(self, tag, attrs):
        a = {k.lower(): (v or "") for k, v in attrs}
        if tag not in HEAD_TAGS:
            self.in_head = False  # <body> or any body element ends the head
        if tag == "meta" and a.get("name", "").lower() in self.names:
            self.robots.append(a.get("content", "").lower())
        rel = a.get("rel", "").lower().split()
        if tag == "link" and "canonical" in rel and self.in_head:
            self.canonicals.append(a.get("href", ""))

    def handle_endtag(self, tag):
        if tag == "head":
            self.in_head = False


def fetch(url, limit=PAGE_LIMIT):
    opener = urllib.request.build_opener(NoRedirect)
    headers = {"User-Agent": UA}
    if os.environ.get("SEO_GATE_AUTH"):
        headers["Authorization"] = os.environ["SEO_GATE_AUTH"]
    req = urllib.request.Request(url, headers=headers)
    try:
        with opener.open(req, timeout=30) as resp:
            return resp.status, resp.headers, resp.read(limit)
    except urllib.error.HTTPError as err:
        return err.code, err.headers, b""


def header_rules(headers, agent):
    """X-Robots-Tag values for all crawlers or for this agent only."""
    words = set()
    for value in headers.get_all("X-Robots-Tag") or []:
        m = re.match(r"\s*([a-z0-9_-]+)\s*:\s*(.*)", value.lower())
        if m and m.group(1) not in ("unavailable_after", "max-snippet",
                                    "max-image-preview", "max-video-preview"):
            if m.group(1) != agent:
                continue  # e.g. "bingbot: noindex" does not apply
            value = m.group(2)
        words |= {w.split(":")[0].strip() for w in value.lower().split(",")}
    return words


def inspect(contract, groups, path):
    url = urljoin(contract["base"], path)
    status, headers, raw = fetch(url)
    agent = contract["agent"]
    page = HeadParser(agent)
    page.feed(raw.decode("utf-8", "replace"))
    words = header_rules(headers, agent)
    words |= {w.strip() for d in page.robots for w in d.split(",")}
    noindex = "noindex" in words or "none" in words
    crawlable = robots_check.allowed(groups, agent, url)
    links = ",".join(headers.get_all("Link") or [])
    canons = re.findall(r'<([^>]+)>\s*;[^,]*rel="?canonical"?', links)
    canons = set(canons + page.canonicals)  # HTTP header and HTML must agree
    return {
        "status": status,
        "location": headers.get("Location"),
        "indexable": status == 200 and crawlable and not noindex,
        "canonical": next(iter(canons)) if len(canons) == 1 else None,
        "canonical_count": len(canons),
    }


def check_page(contract, groups, spec):
    got = inspect(contract, groups, spec["path"])
    errors = []
    if got["status"] != spec["status"]:
        errors.append(f"status {got['status']} != {spec['status']}")
    if "location" in spec and got["location"] != spec["location"]:
        errors.append(f"Location {got['location']} != {spec['location']}")
    if "indexable" in spec and got["indexable"] != spec["indexable"]:
        errors.append(f"indexable {got['indexable']} != {spec['indexable']}")
    if "canonical" in spec:
        target = spec["path"] if spec["canonical"] == "self" else spec["canonical"]
        want = urljoin(contract["canonical_host"], target)
        if got["canonical_count"] != 1:
            errors.append(f"{got['canonical_count']} distinct canonicals (want 1)")
        elif got["canonical"] != want:
            errors.append(f"canonical {got['canonical']} != {want}")
    return errors


def sitemap_paths(contract, path, depth=0):
    """Paths listed by a sitemap; follows one level of sitemap index."""
    status, _, raw = fetch(urljoin(contract["base"], path), SITEMAP_LIMIT)
    if status != 200:
        raise ValueError(f"{path} returned {status}")
    if raw[:2] == b"\x1f\x8b":
        raw = gzip.decompress(raw)
    root = ET.fromstring(raw)
    if root.tag == SM + "sitemapindex" and depth == 0:
        found = []
        for loc in root.iter(SM + "loc"):
            found += sitemap_paths(contract, urlsplit(loc.text.strip()).path, 1)
        return found
    if root.tag != SM + "urlset":
        raise ValueError(f"{path}: unexpected root element {root.tag}")
    locs = [e.text.strip() for e in root.iter(SM + "loc")]
    return [urlsplit(u).path or "/" for u in locs]


def main(contract_file):
    with open(contract_file, encoding="utf-8") as fh:
        contract = json.load(fh)
    failures = 0
    if contract.get("robots_file"):  # e.g. the generated production file
        with open(contract["robots_file"], encoding="utf-8") as fh:
            robots_txt = fh.read()
    else:
        robots_status, _, raw = fetch(urljoin(contract["base"], "/robots.txt"))
        robots_txt = raw.decode("utf-8", "replace")
        if robots_status != 200:  # 5xx pauses crawling; 4xx means "no rules"
            failures += 1
            print(f"FAIL /robots.txt returned {robots_status}")
    groups = robots_check.parse(robots_txt)
    for spec in contract["pages"]:
        errors = check_page(contract, groups, spec)
        failures += bool(errors)
        print(("FAIL " if errors else "PASS ") + spec["path"], "; ".join(errors))
    try:
        paths = sitemap_paths(contract, contract["sitemap"])
        problem = "" if paths else "no URLs listed"
    except (ValueError, OSError, ET.ParseError) as err:
        paths, problem = [], str(err)
    if problem:  # an empty or unreadable sitemap fails, never passes
        failures += 1
        print(f"FAIL sitemap {contract['sitemap']}: {problem}")
    for path in paths:
        errors = check_page(contract, groups,
                            {"path": path, "status": 200, "indexable": True,
                             "canonical": "self"})
        failures += bool(errors)
        if errors:
            print("FAIL sitemap entry " + path, "; ".join(errors))
    print(f"{len(paths)} sitemap URL(s) checked; {failures} failing check(s)")
    return 1 if failures else 0


if __name__ == "__main__":
    sys.exit(main(sys.argv[1]))
```

Design choices: redirects are asserted, not followed; a missing `robots.txt` fails, although Google reads a 404 as "no
crawl restrictions",[^robotsspec] because in production it usually means a broken deploy. Known limits: robots meta
tags count anywhere in the document (the stricter reading); no JavaScript runs.

**3. The rendered check.** A headless browser loads each indexable page with JavaScript off and on, and fails when
scripts change the title, canonical or robots tags or when JSON-LD is invalid or missing; links that exist only after
rendering are a warning, as Google renders later.[^jsbasics] Install with `npm install puppeteer@24.43.1`.

_Example status: executed on 2026-10-23 (Node.js 20.19.5, Puppeteer 24.43.1) in Lab 2 step 6 and against the Basic-auth
test server._

```javascript
// render_check.mjs: compare server HTML with the rendered DOM for each
// indexable contract page. Fails when JavaScript adds a robots rule, changes
// the canonical or title, or when JSON-LD is invalid or a required type is
// missing; warns when links exist only after rendering.
// Env SEO_GATE_AUTH as for seo_gate.py. Usage: node render_check.mjs contract.json
import { readFileSync } from 'node:fs';
import puppeteer from 'puppeteer';

const contract = JSON.parse(readFileSync(process.argv[2], 'utf8'));
const auth = process.env.SEO_GATE_AUTH;
const origin = new URL(contract.base).origin;

const snapshot = () => ({  // runs inside the page
  title: document.title,
  canonical: [...document.head.querySelectorAll('link[rel~="canonical"]')]
    .map((l) => l.getAttribute('href')).join(' '),
  robots: [...document.querySelectorAll('meta[name="robots" i],meta[name="googlebot" i]')]
    .map((m) => m.content.toLowerCase()).join(','),
  links: [...document.querySelectorAll('a[href]')].map((a) => a.getAttribute('href')),
  jsonld: [...document.querySelectorAll('script[type="application/ld+json"]')]
    .map((s) => s.textContent),
});

async function view(browser, url, js) {
  const page = await browser.newPage();
  await page.setJavaScriptEnabled(js);
  await page.setRequestInterception(true);
  page.on('request', (req) => {
    const same = new URL(req.url()).origin === origin;  // credential stays on the preview
    const headers = { ...req.headers(), authorization: auth };
    req.continue(same && auth ? { headers } : {});
  });
  await page.goto(url, { waitUntil: 'networkidle0', timeout: 30000 });
  const result = await page.evaluate(snapshot);
  await page.close();
  return result;
}

function types(blocks, errors) {
  const found = new Set();
  for (const text of blocks) {
    try {
      for (const item of [JSON.parse(text)].flat()) {
        [item['@type']].flat().forEach((t) => found.add(t));
      }
    } catch {
      errors.push('JSON-LD does not parse');
    }
  }
  return found;
}

const browser = await puppeteer.launch();
let failures = 0;
for (const spec of contract.pages.filter((p) => p.indexable)) {
  const url = new URL(spec.path, contract.base).href;
  const server = await view(browser, url, false);   // JavaScript off: server HTML
  const rendered = await view(browser, url, true);  // JavaScript on: rendered DOM
  const errors = [];
  for (const key of ['title', 'canonical', 'robots']) {
    if (server[key] !== rendered[key]) {
      errors.push(`${key} changed by JavaScript: "${server[key]}" -> "${rendered[key]}"`);
    }
  }
  const found = types(rendered.jsonld, errors);
  for (const t of spec.structured_data || []) {
    if (!found.has(t)) errors.push(`JSON-LD type ${t} missing`);
  }
  const jsOnly = rendered.links.filter((h) => !server.links.includes(h));
  failures += errors.length > 0;
  console.log(`${errors.length ? 'FAIL' : 'PASS'} ${spec.path} ${errors.join('; ')}`);
  if (jsOnly.length) {
    console.log(`  WARN ${jsOnly.length} link(s) only after rendering: ${jsOnly[0]}`);
  }
}
await browser.close();
console.log(`${failures} failing page(s)`);
process.exit(failures ? 1 : 0);
```

**4. The pipeline job** runs on **every** pull request: a `paths` filter would skip the
[worked runbook's](#worked-runbook-a-release-adds-noindex-to-product-pages) lockfile-only upgrade, and a required check
skipped by path filtering stays "Pending", blocking the merge.[^ghwf] It tests the generated production `robots.txt`
and judges indexability by it (`robots_file`), since a preview's own file normally disallows everything. Fork pull requests get
no Actions secrets and Dependabot ones only Dependabot secrets, so store `PREVIEW_AUTH` in both; without it the gate
fails, never passes.[^ghsecrets][^dependabot]

_Example status: linted with actionlint 1.7.12, not executed; the two `.sh` scripts are placeholders. Actions pinned by
commit SHA (versions as of October 2026)._

```yaml
name: search-contract
on:
  pull_request:  # no paths filter: lockfile, layout and CDN changes break search too
permissions:
  contents: read
jobs:
  seo-gate:
    runs-on: ubuntu-latest
    env:
      SEO_GATE_AUTH: ${{ secrets.PREVIEW_AUTH }}  # "Basic <base64>"
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1  # v7.0.1
      - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97  # v7.0.0
        with:
          python-version: "3.13"
      - name: Test the generated production robots.txt
        run: |
          ./seo/generate-robots.sh production > robots.prod.txt
          python3 seo/robots_check.py robots.prod.txt seo/expectations.json
      - name: Deploy password-protected preview
        run: ./deploy-preview.sh > preview-url.txt
      - name: Run the search contract against the preview
        run: |
          jq --arg base "$(cat preview-url.txt)" \
            '.base = $base | .robots_file = "robots.prod.txt"' \
            seo/contract.json > contract.ci.json
          python3 seo/seo_gate.py contract.ci.json
      - name: Rendered check (templates that need JavaScript)
        run: |
          npm ci
          node seo/render_check.mjs contract.ci.json
```

**5. Production monitoring.** The same contract after each deploy and hourly; a weekly URL Inspection sample
([Lab 4](#lab-4-compare-googles-indexed-view-with-the-contract)); a daily verified-Googlebot log summary
([Lab 3](#lab-3-find-out-what-googlebot-really-requested)).

### Hands-on labs

Labs 1–3 need Python 3.11+ and a terminal (Lab 2's last step also Node.js 20+); they were run on 2026-10-23 with Python
3.11.17 and show actual outputs. Lab 4 was not executed.

#### Lab 1: test robots.txt against expected outcomes

**Objective.** Catch a robots.txt regression before it ships, matching as RFC 9309 and Google describe: one group per
crawler token (`googlebot/1.2` means `googlebot`), percent-encoded comparison, longest rule wins, allow wins a tie.[^robotsspec][^rfc9309]
**Prerequisites.** Python 3.11+; no network. **Cost and risk.** None; nothing is published.

1. Save the matcher as `robots_check.py`.

_Example status: executed with Python 3.11.17 (and 3.8.10) on 2026-10-23; it also agreed with 32 cases from Google's
documentation and RFC 9309 (precedence, path matching, user-agent groups, percent-encoding, byte-order mark). Known
limit: a reserved character literal on one side and encoded on the other (`=`, `%3D`) does not match._

```python
"""robots_check.py: decide allow/disallow for one crawler token, following RFC
9309 and Google's robots.txt documentation: the group naming the token (else
the '*' group), user-agent text after the token ignored, paths compared in
percent-encoded form, longest matching rule wins, allow wins a tie.
Standard library only. Usage:
    python3 robots_check.py robots.txt expectations.json
"""
import json
import re
import sys
from urllib.parse import quote, urlsplit

UNRESERVED = re.compile(r"%(4[1-9A-F]|5[0-9A]|6[1-9A-F]|7[0-9A]|3[0-9]|2D|2E|5F|7E)")


def normalise(path):
    """Percent-encode non-ASCII octets, upper-case escapes, decode unreserved."""
    path = quote(path, safe="".join(chr(c) for c in range(33, 127)))
    path = re.sub(r"%[0-9a-fA-F]{2}", lambda m: m.group(0).upper(), path)
    return UNRESERVED.sub(lambda m: chr(int(m.group(1), 16)), path)


def agent_token(value):
    """'Googlebot/1.2' and 'googlebot*' both mean 'googlebot'; '*' stays."""
    value = value.strip().lower()
    match = re.match(r"[a-z_-]+", value)
    return "*" if value.startswith("*") else (match.group(0) if match else value)


def parse(text):
    """Return a list of (agents, rules) groups; rules are (kind, path)."""
    groups, agents, rules, in_agents = [], [], [], False
    for raw in text.splitlines():
        line = raw.split("#", 1)[0].strip().lstrip("")
        if ":" not in line:
            continue
        field, value = (part.strip() for part in line.split(":", 1))
        field = field.lower()
        if field == "user-agent":
            if not in_agents and agents:
                groups.append((agents, rules))
                agents, rules = [], []
            agents.append(agent_token(value))
            in_agents = True
        elif field in ("allow", "disallow"):
            if agents and value:  # rules without a path are ignored
                rules.append((field, normalise(value)))
            in_agents = False
        # other fields (sitemap, crawl-delay, ...) neither end nor start a group
    if agents:
        groups.append((agents, rules))
    return groups


def rules_for(groups, token):
    """Merge every group naming the token; fall back to the '*' groups."""
    token = agent_token(token)
    named = [r for a, r in groups if token in a]
    chosen = named or [r for a, r in groups if "*" in a]
    return [rule for rules in chosen for rule in rules]


def to_regex(path):
    anchored = path.endswith("$")
    body = re.escape(path[:-1] if anchored else path).replace(r"\*", ".*")
    return re.compile("^" + body + ("$" if anchored else ""))


def allowed(groups, token, url):
    parts = urlsplit(url)
    target = (parts.path or "/") + ("?" + parts.query if parts.query else "")
    target = normalise(target)
    if target == "/robots.txt":
        return True
    best_len, verdict = -1, True
    for kind, path in rules_for(groups, token):
        if to_regex(path).match(target):
            length = len(path)
            if length > best_len or (length == best_len and kind == "allow"):
                best_len, verdict = length, kind == "allow"
    return verdict


def main(robots_file, expectations_file):
    with open(robots_file, encoding="utf-8") as fh:
        groups = parse(fh.read())
    with open(expectations_file, encoding="utf-8") as fh:
        cases = json.load(fh)
    failures = 0
    for case in cases:
        got = allowed(groups, case["agent"], case["url"])
        ok = got == case["allowed"]
        failures += not ok
        where = urlsplit(case["url"])
        target = where.path + ("?" + where.query if where.query else "")
        print(f"{'PASS' if ok else 'FAIL'} {case['agent']:<10} {target:<28} "
              f"allowed={got} expected={case['allowed']}")
    print(f"{len(cases) - failures}/{len(cases)} expectations met")
    return 1 if failures else 0


if __name__ == "__main__":
    sys.exit(main(sys.argv[1], sys.argv[2]))
```

2. Save this illustrative shop `robots.txt`. Note the separate `Googlebot` group: Googlebot will ignore the `*` group
   entirely.[^robotsspec]

```text
# Production robots.txt (illustrative shop)
User-agent: *
Disallow: /cart
Disallow: /search
Disallow: /*?*sort=
Disallow: /*?*color=
Allow: /*?page=
Disallow: /*.json$

User-agent: Googlebot
Disallow: /cart
Disallow: /search
Disallow: /*?*sort=
Disallow: /*?*color=
Disallow: /*.json$
Allow: /search/help

Sitemap: https://shop.example.com/sitemap.xml
```

3. Save the expectations as `expectations.json`.

```json
[
  {"agent": "googlebot", "url": "https://shop.example.com/", "allowed": true},
  {"agent": "googlebot", "url": "https://shop.example.com/shoes/trail-runner",
   "allowed": true},
  {"agent": "googlebot", "url": "https://shop.example.com/shoes?color=red",
   "allowed": false},
  {"agent": "googlebot", "url": "https://shop.example.com/shoes?page=2&sort=price",
   "allowed": false},
  {"agent": "googlebot", "url": "https://shop.example.com/search/help",
   "allowed": true},
  {"agent": "googlebot", "url": "https://shop.example.com/api/stock.json",
   "allowed": false},
  {"agent": "bingbot", "url": "https://shop.example.com/shoes?page=2",
   "allowed": true},
  {"agent": "googlebot", "url": "https://shop.example.com/static/app.js",
   "allowed": true}
]
```

4. Run `python3 robots_check.py robots.txt expectations.json`.

**Expected output** (actual output of the run):

```text
PASS googlebot  /                            allowed=True expected=True
PASS googlebot  /shoes/trail-runner          allowed=True expected=True
PASS googlebot  /shoes?color=red             allowed=False expected=False
PASS googlebot  /shoes?page=2&sort=price     allowed=False expected=False
PASS googlebot  /search/help                 allowed=True expected=True
PASS googlebot  /api/stock.json              allowed=False expected=False
PASS bingbot    /shoes?page=2                allowed=True expected=True
PASS googlebot  /static/app.js               allowed=True expected=True
8/8 expectations met
```

**Validation.** Delete `Disallow: /*?*color=` from the Googlebot group only and rerun: the third expectation fails,
exit status 1. **Troubleshooting.** Paths are case-sensitive and must start
with `/`.[^robotsspec]
**Teardown.** Delete the three files.

#### Lab 2: run the contract gate against a local site

**Objective.** Catch an accidental `noindex`, a staging canonical and a `noindex` that only JavaScript adds.
**Prerequisites.** `robots_check.py` (Lab 1), `seo_gate.py`, `render_check.mjs` and `contract.json` (reference
implementation) in one directory; port 8765 free; for step 6, Node.js 20+ and `npm install puppeteer@24.43.1`. **Cost and risk.** None; the server listens on 127.0.0.1 only.

1. Create a tiny site. The trail-runner page carries an injected `noindex`, the kind of line a template change adds.

_Example status: executed on 2026-10-23 (bash 5, Python 3.11.17)._

```bash
mkdir -p site/shoes/trail-runner site/account
printf 'User-agent: *\nDisallow: /cart/\n' > site/robots.txt
page() {  # title, canonical path, extra head markup
  printf '<!doctype html><html><head><title>%s</title>%s' "$1" "$3"
  printf '<link rel="canonical" href="https://shop.example.com%s">' "$2"
  printf '</head><body><h1>%s</h1></body></html>\n' "$1"
}
page "Shop" "/" "" > site/index.html
page "Shoes" "/shoes/" "" > site/shoes/index.html
page "Trail runner" "/shoes/trail-runner/" \
  '<meta name="robots" content="noindex">' > site/shoes/trail-runner/index.html
page "Account" "/account/" \
  '<meta name="robots" content="noindex, nofollow">' > site/account/index.html
cat > site/sitemap.xml <<'XML'
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url><loc>https://shop.example.com/</loc></url>
  <url><loc>https://shop.example.com/shoes/</loc></url>
  <url><loc>https://shop.example.com/shoes/trail-runner/</loc></url>
</urlset>
XML
python3 -m http.server 8765 --bind 127.0.0.1 --directory site &
echo $! > server.pid
```

2. Run `python3 seo_gate.py contract.json`.

**Expected output** (actual):

```text
PASS /
PASS /shoes
PASS /shoes/
FAIL /shoes/trail-runner/ indexable False != True
PASS /account/
PASS /no-such-page/
FAIL sitemap entry /shoes/trail-runner/ indexable False != True
3 sitemap URL(s) checked; 2 failing check(s)
```

Exit status 1. `/shoes` passes because Python's server answers 301 to `/shoes/`, as the contract expects.

3. Remove the `noindex` from `site/shoes/trail-runner/index.html` and rerun: all checks pass,
   `3 sitemap URL(s) checked; 0 failing check(s)`, exit 0.
4. Change the canonical in `site/shoes/index.html` to `https://staging.example.com/shoes/` and rerun: the checker reports
   `canonical https://staging.example.com/shoes/ != https://shop.example.com/shoes/` for the page and for its sitemap
   entry. Change it back.
5. Turn the sitemap into an index with a gzipped child, as large sites do, and rerun: the same three URLs are
   checked, `0 failing check(s)`. (The first version read no URLs from an index, and passed.)

```bash
gzip -c site/sitemap.xml > site/sitemap-pages.xml.gz
cat > site/sitemap.xml <<'XML'
<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap><loc>https://shop.example.com/sitemap-pages.xml.gz</loc></sitemap>
</sitemapindex>
XML
```

6. Give the trail-runner page JSON-LD and a script that adds `noindex` and a link, then run
   `node render_check.mjs contract.json`.

```bash
cat > site/shoes/trail-runner/index.html <<'HTML'
<!doctype html><html><head><title>Trail runner</title>
<link rel="canonical" href="https://shop.example.com/shoes/trail-runner/">
<script type="application/ld+json">
{"@context": "https://schema.org", "@type": "Product", "name": "Trail runner"}
</script>
<script>
  const tag = document.createElement("meta");  // what a faulty SEO plug-in does
  tag.name = "robots";
  tag.content = "noindex";
  document.head.append(tag);
  addEventListener("DOMContentLoaded", () => document.body.insertAdjacentHTML(
    "beforeend", '<a href="/shoes/sizes/">Size guide</a>'));
</script>
</head><body><h1>Trail runner</h1></body></html>
HTML
```

**Expected output** (actual):

```text
PASS /
PASS /shoes/
FAIL /shoes/trail-runner/ robots changed by JavaScript: "" -> "noindex"
  WARN 1 link(s) only after rendering: /shoes/sizes/
1 failing page(s)
```

`seo_gate.py` passes this page, as the `noindex` exists only after JavaScript runs. Delete the second `<script>`
block and rerun: three `PASS` lines, exit 0.

**Validation.** Each defect fails; the fixed site passes. **Troubleshooting.** `Connection refused`: check the server
and `base`. **Teardown.** `kill "$(cat server.pid)"`, then delete `site/`, `server.pid`, `node_modules/` and the
scripts.

#### Lab 3: find out what Googlebot really requested

**Objective.** Separate verified Googlebot requests from impostors; see the status mix and crawl on parameter URLs.
Google's manual check (reverse DNS in `googlebot.com`, `google.com` or `googleusercontent.com`, then a forward lookup
returning the same IP) covers *all* its crawlers and fetchers; for Googlebot the name is `crawl-…googlebot.com` or
`geo-crawl-…geo.googlebot.com`.[^verify] `googleusercontent.com` also covers Google Cloud customers'
machines,[^gcpranges] so the script accepts only `googlebot.com`, over IPv4 and IPv6.
**Prerequisites.** Python 3.11+, outbound DNS, an Apache/Nginx "combined" access log.[^apachelog]
**Risk.** Logs contain personal data; use a copy under your retention rules.

_Example status: executed on 2026-10-23 with live DNS, Python 3.11.17 (and 3.8.10)._

```python
"""googlebot_log.py: summarise verified Googlebot requests in an Apache/Nginx
"combined" access log. A request counts only if the reverse DNS name of the
client IP (IPv4 or IPv6) is in googlebot.com, Google's mask for its common
crawlers, AND a forward lookup of that name returns the same IP.
Usage: python3 googlebot_log.py access.log
"""
import collections
import ipaddress
import re
import socket
import sys

LINE = re.compile(r'^(\S+) \S+ \S+ \[[^\]]+\] "(\S+) (\S+) [^"]*" (\d{3}) \S+ '
                  r'"[^"]*" "([^"]*)"')
# crawl-*.googlebot.com or geo-crawl-*.geo.googlebot.com. Not google.com or
# googleusercontent.com: those also cover fetchers and Google Cloud customers.
GOOGLEBOT = ".googlebot.com"


def verified(ip, cache={}):
    if ip not in cache:
        try:
            addr = ipaddress.ip_address(ip)
            host = socket.gethostbyaddr(ip)[0].rstrip(".").lower()
            forward = {ipaddress.ip_address(info[4][0].split("%")[0])
                       for info in socket.getaddrinfo(host, None)}
            cache[ip] = host.endswith(GOOGLEBOT) and addr in forward
        except (OSError, ValueError):
            cache[ip] = False
    return cache[ip]


def section(path):
    if "?" in path:
        return "parameterised"
    parts = path.split("/")
    return "/" + parts[1] if len(parts) > 1 and parts[1] else "/"


def main(log_file):
    statuses = collections.Counter()
    sections = collections.Counter()
    spoofed = collections.Counter()
    with open(log_file, encoding="utf-8", errors="replace") as fh:
        for line in fh:
            m = LINE.match(line)
            if not m or "Googlebot" not in m.group(5):
                continue
            ip, _method, path, status, _ua = m.groups()
            if not verified(ip):
                spoofed[ip] += 1
                continue
            statuses[status[0] + "xx"] += 1
            sections[section(path)] += 1
    total = sum(statuses.values())
    print(f"verified Googlebot requests: {total}")
    for key, n in sorted(statuses.items()):
        print(f"  {key}: {n} ({n / total:.0%})" if total else f"  {key}: {n}")
    for key, n in sections.most_common(10):
        print(f"  section {key}: {n}")
    print(f"claimed-Googlebot requests that failed verification: "
          f"{sum(spoofed.values())} from {len(spoofed)} IP(s)")


if __name__ == "__main__":
    main(sys.argv[1])
```

Sample log: Google's example IP, an IPv6 Googlebot address, two impostors and a visitor. Set `GCP_IP` to a Google
Cloud external IP whose reverse name is under `bc.googleusercontent.com`.

```bash
: "${GCP_IP:?set GCP_IP to a Google Cloud external IP}"
UA='Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)'
T='[23/Oct/2026:06:00:00 +0000]'
while read -r ip req code; do
  ua=$UA; [ "$ip" = 198.51.100.23 ] && ua='Mozilla/5.0 (X11; Linux x86_64)'
  echo "$ip - - $T \"GET $req HTTP/1.1\" $code 512 \"-\" \"$ua\""
done > access.log <<LOG
66.249.66.1 /robots.txt 200
66.249.66.1 /shoes/trail-runner 200
66.249.66.1 /shoes/?color=red&sort=price 200
66.249.66.1 /account/orders 503
2001:4860:4801:10::1 /shoes/rain-jacket 200
203.0.113.7 /admin/ 200
$GCP_IP /shoes/ 200
198.51.100.23 / 200
LOG
python3 googlebot_log.py access.log
```

**Expected output** (actual):

```text
verified Googlebot requests: 5
  2xx: 4 (80%)
  5xx: 1 (20%)
  section /shoes: 2
  section /robots.txt: 1
  section parameterised: 1
  section /account: 1
claimed-Googlebot requests that failed verification: 2 from 2 IP(s)
```

**Validation.** `host 66.249.66.1` returns `crawl-66-249-66-1.googlebot.com`, as in Google's example.[^verify] The
IPv6 and Google Cloud lines are regression tests: an earlier version rejected the IPv6 line and accepted the Google Cloud line.
**Troubleshooting.** No verified requests: the log may record a CDN IP. For large logs, match Google's IP-range files
(`common-crawlers.json`) instead of DNS.[^ipranges]
**Teardown.** Delete the script and any log copies.

#### Lab 4: compare Google's indexed view with the contract

**Objective.** Compare Google's selected canonical and index state for key URLs with the contract.
**Prerequisites.** A verified Search Console property; in `GSC_TOKEN`, an OAuth 2.0 access token (a short-lived API
credential) with the `webmasters.readonly` scope for a user of that property.[^inspectapi]
**Cost and risk.** Free; quota is 2,000 inspections per site per day and 600 per minute (as of October 2026).[^gsclimits]
The API reports the **indexed** version; it cannot test a live URL.[^inspectapi]

_Example status: syntax-checked, **not executed** (requires a verified property and credentials)._

```python
"""inspect_urls.py: report Google's indexed view of key URLs.
Env GSC_TOKEN: OAuth 2.0 access token, scope webmasters.readonly.
Usage: python3 inspect_urls.py sc-domain:example.com urls.txt
"""
import json
import os
import sys
import time
import urllib.request

API = "https://searchconsole.googleapis.com/v1/urlInspection/index:inspect"


def inspect(site, url, token):
    body = json.dumps({"inspectionUrl": url, "siteUrl": site}).encode()
    req = urllib.request.Request(API, data=body, method="POST", headers={
        "Authorization": "Bearer " + token, "Content-Type": "application/json"})
    with urllib.request.urlopen(req, timeout=30) as resp:
        return json.load(resp)["inspectionResult"]["indexStatusResult"]


def main(site, urls_file):
    token = os.environ["GSC_TOKEN"]
    with open(urls_file, encoding="utf-8") as fh:
        urls = [line.strip() for line in fh if line.strip()][:200]
    for url in urls:  # 200 per run keeps far below the daily per-site quota
        r = inspect(site, url, token)
        same = r.get("googleCanonical") == r.get("userCanonical")
        flag = "OK   " if r.get("verdict") == "PASS" and same else "CHECK"
        print(flag, url, r.get("coverageState"), r.get("lastCrawlTime"),
              "google-canonical=" + str(r.get("googleCanonical")))
        time.sleep(0.2)


if __name__ == "__main__":
    main(sys.argv[1], sys.argv[2])
```

**Expected output.** One line per URL; `OK` when the verdict is `PASS` ("Valid") and Google's canonical equals the
declared one.[^inspectresult] **Validation.** Compare a few lines with the URL Inspection tool. **Troubleshooting.** 403:
no access, or a wrong `siteUrl` (URL-prefix properties need the trailing slash).[^inspectapi] **Teardown.** Revoke the
token if created for the lab.

### Use cases and scenarios

All four are **illustrative scenarios** built from documented mechanisms, not reported case studies.

| Scenario | SEOps method | Trade-off | Success criterion |
|---|---|---|---|
| Faceted e-commerce catalogue: filters create endless parameter URLs | Disallow filter parameters, keep one unfiltered listing crawlable, 404 for empty combinations ([judgement call](#a-judgement-call-block-the-filter-urls-or-let-them-be-indexed))[^faceted] | Filter pages cannot rank for long-tail filter searches | Crawl share of parameter URLs falls; new products indexed sooner |
| Client-side app moving to server rendering | Server-render indexable templates, History API routes, real 404s, rendered checks in CI[^jsbasics] | Engineering cost | Server and rendered views agree; soft 404s disappear |
| Domain or URL migration (40,000 pages) | Tested one-hop 301 map, redirects kept at least a year, Change of Address for domain moves[^sitemove] | Temporary ranking fluctuation even when done well[^sitemove] | Every mapped URL redirects correctly |
| Publisher with thousands of articles a day | CMS-generated sitemaps with honest `lastmod`; crawl review if "Discovered – currently not indexed" grows[^ping][^crawlbudget] | The CMS must track meaningful edits[^sitemapsbuild] | New articles crawled the same day |

**Pitfalls.** Blocking facets in robots.txt hides any `noindex` or canonical on them.[^noindex] Serving bots different
content from users is cloaking, even as a rendering workaround.[^dynrender][^spam] Redirecting every old URL to the home
page throws the mapping away. A sitemap claiming every page changed today teaches Google to ignore `lastmod`.[^ping]

## Measure, secure and pay

### Metrics and KPIs

SEOps measures whether the controls hold; rankings and clicks belong to [SEO Operations](../seo-operations/). The
metric names are **this Atlas's proposals**.

| KPI | Formula | Data source | Pitfall |
|---|---|---|---|
| Contract pass rate | passing checks ÷ all contract checks in a run | Checker on production, hourly | Easy if coverage is low |
| Contract coverage | templates with a contract ÷ indexable templates | Route list | Counting URLs instead of templates |
| Sitemap hygiene | sitemap URLs that are 200, indexable and self-canonical ÷ all sitemap URLs | Checker, weekly | Skipping files of an index |
| Canonical agreement | sampled URLs where Google's canonical = declared ÷ sample | URL Inspection API, weekly[^inspectresult] | Small or biased sample; quota[^gsclimits] |
| Crawl waste share | verified Googlebot requests to parameter, redirecting, soft-404 or `noindex` URLs ÷ all verified Googlebot requests | Verified logs, weekly | Unverified bots inflate it |
| Crawler error rate | verified Googlebot 5xx + 429 ÷ all verified Googlebot requests | Logs; Crawl Stats host status[^crawlstats] | Averages hide a one-hour outage |
| Time to detect | first alert − deploy time, for search regressions | Incident records | Only counts noticed incidents |

Do not target 100 % indexing: "Don't expect every URL on your site to be indexed", and the Page indexing report shows
at most 1,000 example URLs per status.[^pagereport] Use indexed counts per sitemap as a trend.

::: example Worked calculation (illustrative numbers)
In 28 days a shop logs 1,200,000 verified Googlebot requests: 540,000 to filter-parameter URLs, 60,000 to redirecting
URLs, 12,000 to soft-404 pages, 18,000 answered 5xx and 2,000 answered 429. Of 48,000 sitemap entries, 1,900 fail the
contract (1,150 redirect). A URL Inspection sample of 200 product URLs finds 184 where Google's canonical matches.

- Crawl waste share = (540,000 + 60,000 + 12,000) ÷ 1,200,000 = **51 %**.
- Crawler error rate = (18,000 + 2,000) ÷ 1,200,000 = **1.7 %**.
- Sitemap hygiene = (48,000 − 1,900) ÷ 48,000 = **96.0 %**.
- Canonical agreement = 184 ÷ 200 = **92 %**.

Reading: half of Googlebot's requests go to URLs the shop does not want indexed, so faceted navigation is the first fix.
Redirecting sitemap entries are a generator bug.[^sitemapsbuild]
:::

**Targets.** Blocking checks on production: 100 %. Sitemap hygiene above 99 %. For crawl waste and error rate set a
baseline and a direction; no "good" value is published.[^crawlbudget]

### Quality, security, privacy and compliance

**Spam policies, release by release.** Code changes can create unintended policy problems:[^spam]

| Change | Policy risk | Guard |
|---|---|---|
| Different HTML for bots (pre-rendering, CDN bot detection) | Cloaking | Equivalent content for bots and users; rendered-parity check |
| Redirects by device, country or user agent | Sneaky redirects | Everyone reaches equivalent content |
| Programmatic pages (locations, combinations, AI-written variants) | Scaled content abuse; doorway abuse | Each page useful on its own; reviewer sign-off per template |
| Third-party content on your domain | Site reputation policy (different treatment in the European Economic Area, 2026) | Clear ownership and oversight |
| Collapsed content | Hidden text abuse | Accordions, tabs and screen-reader text are allowed; text hidden only from users is not |

**Security.** `robots.txt` is public and advisory; listing a secret path advertises it.[^rfc9309] User agents "can be
spoofed", so verify crawlers by DNS or Google's IP-range files before granting allowlists or higher rate limits, and
accept only `googlebot.com` names for Googlebot ([Lab 3](#lab-3-find-out-what-googlebot-really-requested)): `google.com` and
`gae.googleusercontent.com` names also belong to user-triggered fetchers, which ignore robots.txt.[^crawlers][^verify]
Watch the Security Issues report for hacked content alongside [SecOps](../secops/).[^spam][^trafficdrops] A
structured-data manual action removes rich-result eligibility, not web ranking.[^sdpolicies]

**Privacy.** Logs hold IP addresses; retention rules come from [PrivacyOps](../privacyops/).

### Cost, capacity and sustainability

- **Capacity is a search input.** Google's crawl capacity limit falls with slow responses, 5xx or 429 and is shared by
  all Google crawlers on the host.[^crawlbudget] Cheap wins: 304 Not Modified, HTML under 2MB, less filter-URL
  crawling, which can cost "large amounts of computing resources".[^crawlbudget][^bytes][^faceted]
- **Tooling.** Search Console and its API are free within quotas;[^gsclimits] tool prices are not quoted. The main cost
  is engineering time.
- **Sustainability.** Crawl waste is wasted compute on both sides; removing it costs no visibility.

### Assurance of the operating model

1. **Canary fixtures.** A deliberately `noindex` page in a test sitemap must keep failing the gate; check monthly.
2. **Monitor the monitors.** Alert when a contract job, log summary or export has not run for 24 hours.
3. **Override log.** Each gate override records who, why and the contract change; review monthly.
4. **Coverage audit.** Quarterly, compare the route list with the contract.
5. **Reconcile with Google.** Crawl Stats may differ slightly from your logs;[^crawlstats] large gaps mean a missing log
   source (CDN, second origin).
6. **Property settings.** Monthly, compare each property's Search generative AI control with the recorded decision; a
   child property can override its parent, and the top-level domain property's owners are notified.[^genaictl]

## When things go wrong

### Common failures and troubleshooting

| Symptom | Likely cause | Diagnosis | Fix and prevention |
|---|---|---|---|
| Indexed pages of one template fall over weeks | `noindex` added by a template or header change | URL Inspection live test; HTML and `X-Robots-Tag` | Remove; request indexing; contract per template |
| Crawling stops abruptly | `robots.txt` returns 5xx (WAF, CDN rule, outage) | Fetch it from outside; Crawl Stats | Restore a 200 and monitor it[^robotsspec][^pause] |
| "Google chose different canonical than user" grows | Links, sitemaps or redirects point elsewhere | Compare canonical, sitemap entry and internal links | Align all signals on one URL[^canonhow] |
| Soft 404s on a JavaScript app | Client-side "not found" views return 200 | Live test; Page indexing report | Server-side 404, redirect to one, or inject `noindex`[^jsbasics] |
| "Discovered – currently not indexed" keeps growing | Slow server or infinite URL spaces | Crawl Stats response time; log crawl waste | Faster origin, fewer parameter URLs, fix 5xx[^crawlbudget] |
| Heavy load from "Googlebot" | Impostors using its user agent | DNS (`googlebot.com` names only) or IP-range verification[^verify] | Rate-limit unverified agents |
| Content missing on very large pages | Content or JSON-LD after the 2MB cut-off | HTML size | Externalise inline assets; size budget in CI[^bytes] |

### Worked runbook: a release adds noindex to product pages

::: example Illustrative incident
A shop deploys a framework upgrade. A shared layout now renders `<meta name="robots" content="noindex">` on every route
without explicit SEO config, including 30,000 product pages. Nobody notices for nine days.
:::

::: figure Incident flow for a search-visibility regression
```mermaid
flowchart TB
  A["Index or traffic drop"] --> B{"Site-wide or one template?"}
  B -- "template" --> D["Live test vs indexed, 5 URLs"]
  B -- "site-wide" --> C{"robots.txt, 5xx or DNS?"}
  D --> E{"Defect in our HTML?"}
  E -- no --> G["Align canonical signals"]
  E -- yes --> F["Roll back or hot-fix"]
  C -- yes --> F
  C -- no --> M{"Manual action or security?"}
  M -- yes --> R["Fix; Request Review"]
  M -- no --> U["Update or demand: to SEO Operations"]
  G --> H["Request indexing; resubmit sitemap"]
  F --> H
  H --> I["Monitor; post-incident review"]
  R --> I
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  class A risk
  class B,C,E,M decision
  class F,G,H,I,R core
  class U neutral
```
Text equivalent: An index or traffic drop is site-wide or limited to one template. Site-wide: a robots.txt, server or
DNS fault is rolled back or hot-fixed; otherwise a manual action or security issue is fixed and sent with Request
Review; otherwise an algorithm update or demand change, which no rollback fixes, goes to SEO Operations. One template:
compare the live test with the indexed version for five URLs; roll back or hot-fix a defect in your HTML or headers,
otherwise align canonical, links, sitemap and redirects. Fixes end with requesting indexing and resubmitting the
sitemap; all paths except the hand-over end with weeks of monitoring and a post-incident review.
:::

1. **Classify (15 minutes).** One template, so not robots.txt or an outage. Google's traffic-drop guide separates
   site-wide technical issues from page-level ones such as "a misplaced noindex tag".[^trafficdrops] (A manual action
   would instead be fixed and then submitted with Request Review in the Manual actions report.[^manualactions])
2. **Confirm (30 minutes).** URL Inspection on five product URLs: the indexed version may look fine while the live test
   reports `noindex`.[^inspecttool] `curl -s https://shop.example.com/p/123 | grep -i robots` confirms it in server HTML.
3. **Contain (same day).** Roll back or hot-fix; run the contract on production; check the CDN added no `X-Robots-Tag`.
4. **Recover (days to weeks).** Resubmit the product sitemap; request indexing for the most important URLs (daily limit
   per property).[^inspecttool] Recrawling the rest can take months.[^noindex] Ship no "helpful" extra changes meanwhile.
5. **Learn (two weeks).** Post-incident review: add the product template to the contract, a production smoke test after
   deploys and an alert on indexed count per sitemap. Check that the gate runs on dependency-only pull requests: this
   upgrade changed a lockfile, which a `paths` filter would have skipped ([pipeline job](#reference-implementation)).
   Nine days to detect is the number to beat.

#### Second runbook: robots.txt answers 503 during maintenance

A maintenance rule served 503 for every path, including `/robots.txt`. Google says not to return 503 for `robots.txt`
"because this blocks all crawling". Exclude it, keep 503 on pages for a day or two at most, and use an indexable 200
placeholder for longer closures.[^pause]

_Example status: syntax-checked (bash -n), not executed against a live host._

```bash
# Smoke test after any maintenance-mode or CDN rule change (illustrative host).
code=$(curl -s -o /dev/null -w '%{http_code}' https://shop.example.com/robots.txt)
if [ "$code" != "200" ]; then
  echo "robots.txt returned $code: crawling will pause or rules will be ignored" >&2
  exit 1
fi
echo "robots.txt OK"
```

### Anti-patterns and myths

| Anti-pattern or myth | Why it fails | Do instead |
|---|---|---|
| `Disallow: /` as staging's only protection | Advisory, public, and copied to production by accident | Authentication |
| Canonicals as the cure for every duplicate | A hint Google may ignore[^canonwhat] | Redirect true duplicates; align signals |
| Disallowing sections to "shift" crawl budget | Not shifted unless the site is at its capacity limit[^crawlbudget] | Fix capacity, remove URL spaces |
| `lastmod` = build time | Google stops trusting it[^ping] | Real dates or omit |
| Sitemap pings in the deploy script | Deprecated by Google in 2023[^ping] | robots.txt, Search Console or the API |
| `crawl-delay` for Googlebot | Not supported[^robotsspec] | Capacity; brief 429/503 if overloaded[^httpstatus] |
| "We need llms.txt for AI Overviews" | Google Search ignores it[^aiguide] | Crawlable, indexable, useful pages; the Search generative AI control set to Include[^genaictl] |

## Deciding and adopting

### When to use and when not to use

**Use it when** organic search matters and the site has many templates, ships weekly or faster, renders with
JavaScript, or faces a migration.

**Do not build it when** the site is small, rarely changes and is well linked: a checklist and a monthly look at
Search Console are enough.[^sitemapsabout][^crawlbudget] Nor without template owners: unowned checks become ignored red
builds. **Hidden costs:** current contracts, override reviews, the Search Console property and alerts.

### Decision framework

::: figure How much SEOps does a site need?
```mermaid
flowchart TB
  A{"Is organic search a material channel?"} -- no --> N["Checklist before releases; monthly console review"]
  A -- yes --> B{"More than a few templates or frequent releases?"}
  B -- no --> N
  B -- yes --> S["Base: contract in CI + production smoke test"]
  S --> C{"JavaScript renders indexable content?"}
  C -- yes --> R["Add rendered checks in CI"]
  C -- no --> D{"Over 10k changing URLs or a migration ahead?"}
  R --> D
  D -- yes --> L["Add verified logs + crawl-waste review"]
  D -- no --> K["Keep the base; revisit when either changes"]
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  class A,B,C,D decision
  class R,L,S core
  class N,K neutral
```
Text equivalent: If organic search is not material, or the site has few templates and rare releases, use a pre-release
checklist and a monthly Search Console review. Otherwise start from a base: contract in CI plus a production smoke test.
The next two questions add to it: JavaScript-rendered content adds rendered checks; then, either way, over ten thousand
changing URLs or a migration adds verified crawler logs and a crawl-waste review. If neither applies, keep the base.
:::

**Which failures block a release?** The Atlas's rule: block what is *silent*, *broad* and *slow to recover*.

| Check | Silent | Broad | Slow to recover | Gate |
|---|---|---|---|---|
| Indexable template returns `noindex` or non-200 | Yes | Yes | Yes | Block |
| robots.txt disallows an indexable path or is not 200 | Yes | Yes | Hours to weeks | Block |
| Canonical points to another host | Yes | Yes | Weeks | Block |
| Sitemap lists non-canonical or redirecting URLs | Yes | Partly | Days | Block new; warn drift |
| Required structured-data property missing | Partly | Template | Days | Warn |
| Title or description length | No | Page | Fast | Report |

### Maturity model

**This Atlas's model**, not an industry standard; move up only when the exit criterion is met.

| Level | Observable behaviour | Exit criterion to the next level |
|---|---|---|
| 1. Reactive | Problems found from traffic drops or audits | Named owners for the Search Console property and `robots.txt` |
| 2. Checked | Manual pre-release checklist; robots.txt in version control | Written contracts for the top templates |
| 3. Gated | Contract and robots expectations in CI on previews; blocking rules agreed | Production smoke test after every deploy; detection within a day |
| 4. Observed | Verified logs and Search Console API feed a dashboard; weekly review | Canonical agreement and sitemap hygiene baselined |
| 5. Engineered | Contracts generated from the route table; rehearsed migrations; canaries | Two quarters with no search incident detected after more than a day |

### Adoption roadmap: 30/60/90 days

| Window | Actions | Milestone |
|---|---|---|
| Days 1–30 | Review Search Console properties, users and settings; `robots.txt` and sitemap generation in version control; contracts for the top five templates; Lab 1 in CI | First blocking check on previews |
| Days 31–60 | Contract gate on previews; production smoke test; severities agreed with [ReleaseOps](../releaseops/); code-owner review for routing, `<head>` and CDN headers | Every deploy runs the contract |
| Days 61–90 | Verified-Googlebot log summary; weekly URL Inspection sample; alerts; first canary; runbook rehearsal | KPI baselines |

Quick wins: a `robots.txt` monitor, clean sitemaps, authenticated staging, a recorded AI-control setting.

## Toolkit

### Checklists

**Pre-release (routes, templates, rendering or CDN rules changed)**
- [ ] Contract passes on the preview, including new templates
- [ ] `robots.txt` expectations pass; file returns 200, under 500 KiB
- [ ] One absolute, self-referencing canonical per indexable page in server HTML
- [ ] No `noindex` (HTML or header) on indexable templates; mobile and desktop identical[^mobilefirst]
- [ ] Removed pages 404/410; moved pages one-hop 301/308
- [ ] Sitemaps list only canonical, indexable URLs with honest `lastmod`
- [ ] Rendered check passes; HTML under 2MB
- [ ] Preview host requires authentication

**Migration**
- [ ] Complete old-to-new URL map, including images and PDFs with traffic
- [ ] Every mapped URL tested: one hop, permanent, correct target
- [ ] Canonicals, hreflang, internal links and sitemaps updated
- [ ] New property verified; Change of Address for domain moves; redirects kept at least a year[^sitemove]

**Weekly review**
- [ ] Indexed counts per sitemap stable
- [ ] Crawl Stats host status, 5xx/429 share, response time
- [ ] Canonical agreement sample; Manual actions and Security Issues empty
- [ ] Search generative AI setting per property unchanged; owner of the decision recorded[^genaictl]
- [ ] Monitors and exports ran in the last 24 hours

**Incident:** use the [worked runbook](#worked-runbook-a-release-adds-noindex-to-product-pages).

### Templates

**Search contract entry (per template)**, for review alongside the [JSON contract](#reference-implementation):

```text
Template:            product detail
Example URLs:        /p/123-trail-runner, /p/987-rain-jacket
Status:              200; 404 for unknown IDs (server-side)
Indexable:           yes
Canonical:           self, absolute, https://www.example.com/p/<id>-<slug>
Duplicates:          ?ref=, ?utm_*, /print -> canonical to clean URL
Sitemap:             products.xml, lastmod = last price/stock/content change
hreflang:            en-GB, en-US, x-default; reciprocal
Structured data:     Product (gate checks the type; review required properties by hand)
Rendering:           server HTML contains title, description, price, links
Owner:               web team "catalogue"; reviewer: technical SEO
```

**SEO change review (pull-request template section).**

```text
Search impact (complete if routes, <head>, rendering, redirects, robots,
sitemaps or CDN headers change):
- Templates affected:
- Indexability change intended? (yes/no; which URLs, why):
- Canonical or URL changes (attach redirect map if any):
- robots.txt change and expectation tests updated? (link):
- Contract updated in this PR? (yes/no):
- Rollback: how to revert within one hour:
- Reviewer (technical SEO):
```

**Redirect map (CSV header and rows).**

```csv
old_url,new_url,status,owner,notes
https://old.example.com/shoes/trail,https://www.example.com/p/123-trail-runner,301,catalogue,
https://old.example.com/blog/2019/fit,https://www.example.com/guides/fit,301,content,merged
```

**Search incident review.**

```text
Incident:        <title>                      Date of release: <YYYY-MM-DD>
Detected by:     <alert / person / report>    Time to detect: <hours or days>
Scope:           <templates, URL count, engines>
Mechanism:       <noindex / robots / canonical / status / rendering / other>
Timeline:        deploy -> first crawl of defect -> detection -> fix -> recovery
Why no check:    <no contract entry / advisory only / path filter / monitor down>
Actions:         <new check>, <owner>, <due date>
```

### Deep comparisons

**Which control for which goal?**

| Goal | Right control | Wrong control | Why |
|---|---|---|---|
| Keep a page out of results | `noindex` on a crawlable page, or authentication | `Disallow` | Disallowed URLs can still be indexed[^robotsintro] |
| Stop fetching endless URLs | `Disallow` | `noindex` | `noindex` still costs a fetch[^crawlbudget] |
| Merge duplicates you keep serving | `rel="canonical"` plus consistent links | robots.txt | Blocked pages pass no signals[^canonhow] |
| Retire a duplicate URL | 301/308 | Removal tool | It hides all versions[^canonhow] |
| Remove a page for good | 404 or 410 | 200 with "not found" text | That is a soft 404[^httpstatus] |
| Close the site for a day | 503 on pages, 200 on robots.txt | 503 on robots.txt | Blocks all crawling[^pause] |

For neighbouring pages, see [how SEOps relates to other Ops](#how-seops-relates-to-other-ops).

### FAQ

**Is SEOps a job title?** Not in any source found for this page. The advert cited here is for a "Lead Technical SEO Manager" and lists CI quality gates and monitoring among the duties.[^zoomjob]

**Does passing every check mean we will rank?** No; checks establish eligibility. Meeting every requirement "doesn't
mean that Google will crawl, index, or serve its content".[^essentials]

**Should we exclude the site from AI Overviews?** That is a business decision with a traffic cost, made in
[SEO Operations](../seo-operations/#ai-features-and-zero-click-searches); SEOps makes sure the Search generative AI
control matches it.[^genaictl]

**Do we need IndexNow?** Not for Google, which is not a listed participant (October 2026); use sitemaps with honest
`lastmod`.[^indexnow][^ping] For engines such as Bing it announces changes; a 200 only means "received".[^indexnow]

**How do we get staging out of Google?** Authentication. Google drops URLs that return 4xx, including 401 and 403, over
time;[^httpstatus] for urgent cases use Search Console removals, as Google's `noindex` guide suggests.[^noindex]

## Going further

### Advanced topics and emerging direction

**Recent changes to track:** Google's crawler documentation moved to a crawling-infrastructure section, with IP-range
files under `developers.google.com/crawling/ipranges/` (2026);[^ipranges] the 2MB HTML fetch limit was explained in
March 2026;[^bytes] generative-AI performance reports and the Search generative AI control reached all websites by
31 August 2026.[^aireports][^genaictl]

**Active and uncertain:** Google mentions browser agents that read the DOM and accessibility tree;[^aiguide] other AI
products publish their own crawler rules. Edge platforms that rewrite HTML, headers and redirects are invisible to
application tests, so run production checks against the CDN hostname.

### Learning path

| Stage | What to do | Evidence you can show |
|---|---|---|
| Prerequisites | HTTP status codes and headers; how your site is rendered and deployed ([WebOps](../webops/)) | You can explain what your CDN does to a response |
| One day | Google's guides to Search, robots.txt and `noindex`; [Lab 1](#lab-1-test-robotstxt-against-expected-outcomes) | A robots expectation file for your site |
| One week | Labs 2 and 3; Page indexing, Crawl Stats and URL Inspection reports | Your top three exclusions and causes |
| 30 days | Contracts for top templates; gate in CI; production smoke test | A blocked regression |
| Advanced | Rendering parity at scale, log pipelines, hreflang, migrations | A rehearsed migration |

## Wrap-up

### Executive recap and cheat sheet

- **What it is:** an informal label; here, technical SEO as engineering, protecting discover, crawl, render and index.
- **Right control:** robots.txt = crawling; `noindex` = indexing (page must be crawlable); canonical and redirects =
  duplicates; 404/410 = gone; 503 = briefly down, never for robots.txt.
- **Checks:** status, `noindex`, canonical, robots expectations, sitemap hygiene, rendered content, HTML under 2MB.
- **Metrics:** contract pass rate and coverage, sitemap hygiene, canonical agreement, crawl waste, crawler errors, time
  to detect.
- **Watch-outs:** a `Googlebot` group ignores `*` rules; JavaScript can add but not reliably undo `noindex`; staging
  needs authentication; verify Googlebot by `googlebot.com` names only.

### A judgement call: block the filter URLs or let them be indexed?

::: decision Dilemma: faceted navigation on a 20,000-product shop
**Assumptions (illustrative).** Filters create millions of parameter URLs taking 51 % of verified Googlebot requests
(the [worked calculation](#metrics-and-kpis)). Merchandising wants some filter pages ("red trail shoes") indexed. New
products take a week to appear.

**Option A: keep facets crawlable, canonicalised to the unfiltered category.** Nothing visible changes. But Google
calls canonical and nofollow "generally less effective in the long term"; canonicals "may, over time, decrease the crawl
volume", so the waste continues for months, and the canonicalised pages are not indexed anyway.[^faceted]

**Option B: disallow the filter parameters.** Google documents this for filter URLs you do not need indexed, keeping
one unfiltered listing crawlable.[^faceted] But no filter page can then be indexed with content.

**Option C: move filters into URL fragments** (`#color=red`), which "will have no impact on crawling (positive or
negative)".[^faceted] That ends the waste but means rebuilding the filter front end, and since Google generally ignores
fragments, no filter state could be indexed either.

**Decision.** Separate the goals. The crawl problem affects every product; the opportunity, a handful of combinations.
Disallow the parameter space, and turn the few combinations with proven demand into curated, path-based category pages
with their own content and sitemap entries; mass-produced near-identical pages are what the doorway and scaled-content
policies describe.[^spam]

**Uncertainty.** Nobody can promise the curated pages rank; crawl waste and how fast new products are crawled can be
measured. Review after eight weeks. Option A's appeal, changing nothing visible, is exactly why it keeps spending crawl
on URLs the team's own canonicals say should not be indexed.
:::

### Learning objectives and mastery check

After this page you can **explain** the label and its scope here; **choose** the right control for a goal; **design**
a contract and gate; **diagnose** an indexing regression; **judge** whether a site needs SEOps.

::: answer 1. A URL disallowed in robots.txt shows in Google without a description. Why, and what is the fix?
robots.txt stops crawling, not indexing, so Google can index the URL from links alone. Allow crawling and add `noindex`,
or require authentication; a `noindex` behind the disallow is never seen.
:::

::: answer 2. robots.txt has a `User-agent: *` group with ten rules and a `User-agent: Googlebot` group with one. Which apply to Googlebot?
Only the Googlebot group's rule: Google picks the most specific group and does not merge it with `*`. Copy every rule
Googlebot should obey into its group and test with expectations.
:::

::: answer 3. Maintenance returns 503 for everything, including robots.txt, for six hours. What is the risk?
Google stops crawling the whole site while robots.txt cannot be fetched (the first 12 hours of errors). Keep robots.txt
at 200 and return 503 only for pages, for a day or two at most.
:::

::: answer 4. Should 2 million filter URLs get `noindex` "to save crawl budget"?
No. Google must fetch each URL to see `noindex`, so crawling continues. Disallow URLs that should never be fetched, and
return 404 for empty filter combinations.
:::

::: answer 5. The gate passes on preview, but production pages carry noindex after release. Where do you look?
At what differs: CDN rules adding `X-Robots-Tag`, environment flags read wrongly, or JavaScript adding robots tags (the
rendered check's job). That is why the contract also runs against the production hostname after every deploy.
:::

::: answer 6. A 300-page site, one release a month, no JavaScript rendering. Build a CI gate?
Probably not. A checklist, `robots.txt` in version control and a monthly look at the Page indexing report cover the
realistic risks. Revisit if it grows, releases faster or plans a migration.
:::

### Conclusion

"SEOps" is not a discipline with a definition; it is a useful name for a habit: treating the few controls that decide
whether search engines can find, crawl, render and index a site as code, written down, tested before release and
watched afterwards. The failures they prevent are cheap to catch and slow to repair. SEOps cannot make a page rank. Get
the four gates right on every release, measure the contract rather than the rankings, keep a runbook for the day it
fails, and leave *what* to publish to the people who run the SEO programme.

## Sources and further reading

All sources were opened on 2026-10-23; dates are the "last updated" dates shown then. Claim-by-claim notes:
`research/seops-ledger.md`.

[^gradial]: [AI SEO Operations: The Enterprise Guide to Governed Search and GEO Execution](https://www.gradial.com/guides/seo-operational-guide) — Gradial (vendor), 2026-08-05. Checked 2026-10-23.
[^zoomjob]: [Lead Technical SEO Manager (Zoom)](https://www.builtinchicago.org/job/lead-technical-seo-manager/10968618) — job advert on Built In Chicago, shown as reposted on 2026-10-22. Job adverts are removed without notice. Checked 2026-10-23.
[^seopsjob]: [SE Operations Manager (Veza Technologies)](https://builtin.com/job/se-enablement-manager/3719567) — job advert on Built In, removed 2025-04-21 but still readable. Checked 2026-10-23.
[^howsearch]: [In-depth guide to how Google Search works](https://developers.google.com/search/docs/fundamentals/how-search-works) — Google Search Central, updated 2025-12-18. Checked 2026-10-23.
[^essentials]: [Google Search Essentials](https://developers.google.com/search/docs/essentials) — Google Search Central, updated 2025-12-10. Checked 2026-10-23.
[^robotsintro]: [Introduction to robots.txt](https://developers.google.com/search/docs/crawling-indexing/robots/intro) — Google Search Central, updated 2025-12-10. Checked 2026-10-23.
[^robotsspec]: [How Google interprets the robots.txt specification](https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec) — Google Crawling Infrastructure, updated 2026-08-31. Checked 2026-10-23.
[^rfc9309]: [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html) — IETF (Koster, Illyes, Zeller, Sassman), Standards Track, 2022-09. Checked 2026-10-23.
[^robotsmeta]: [Robots meta tag, data-nosnippet, and X-Robots-Tag specifications](https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag) — Google Search Central, updated 2026-03-24. Checked 2026-10-23.
[^noindex]: [Block Search indexing with noindex](https://developers.google.com/search/docs/crawling-indexing/block-indexing) — Google Search Central, updated 2025-12-10. Checked 2026-10-23.
[^canonwhat]: [What is URL canonicalization](https://developers.google.com/search/docs/crawling-indexing/canonicalization) — Google Search Central, updated 2026-08-20. Checked 2026-10-23.
[^canonhow]: [How to specify a canonical URL with rel="canonical" and other methods](https://developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls) — Google Search Central, updated 2026-07-10. Checked 2026-10-23.
[^redirects]: [Redirects and Google Search](https://developers.google.com/search/docs/crawling-indexing/301-redirects) — Google Search Central, updated 2026-04-14. Checked 2026-10-23.
[^sitemove]: [Site moves with URL changes](https://developers.google.com/search/docs/crawling-indexing/site-move-with-url-changes) — Google Search Central, updated 2026-08-20. Checked 2026-10-23.
[^sitemapsabout]: [Learn about sitemaps](https://developers.google.com/search/docs/crawling-indexing/sitemaps/overview) — Google Search Central, updated 2025-12-10. Checked 2026-10-23.
[^sitemapsbuild]: [Build and submit a sitemap](https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap) — Google Search Central, updated 2026-07-08. Checked 2026-10-23.
[^sitemapsorg]: [Sitemaps XML format](https://www.sitemaps.org/protocol.html) — sitemaps.org, protocol 0.9 (undated). Checked 2026-10-23.
[^ping]: [Sitemaps ping endpoint is going away](https://developers.google.com/search/blog/2023/06/sitemaps-lastmod-ping) — Google Search Central Blog, 2023-06-26. Checked 2026-10-23.
[^hreflang]: [Tell Google about localized versions of your page](https://developers.google.com/search/docs/specialty/international/localized-versions) — Google Search Central, updated 2026-09-21. Checked 2026-10-23.
[^mobilefirst]: [Mobile-first indexing best practices](https://developers.google.com/search/docs/crawling-indexing/mobile/mobile-sites-mobile-first-indexing) — Google Search Central, updated 2025-12-10. Checked 2026-10-23.
[^jsbasics]: [Understand the JavaScript SEO basics](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics) — Google Search Central, updated 2026-03-04. Checked 2026-10-23.
[^dynrender]: [Dynamic rendering as a workaround](https://developers.google.com/search/docs/crawling-indexing/javascript/dynamic-rendering) — Google Search Central, updated 2025-12-10. Checked 2026-10-23.
[^bytes]: [Inside Googlebot: demystifying crawling, fetching, and the bytes we process](https://developers.google.com/search/blog/2026/03/crawler-blog-post) — Google Search Central Blog, 2026-03-31. Checked 2026-10-23.
[^httpstatus]: [How HTTP status codes affect Google's crawlers](https://developers.google.com/crawling/docs/troubleshooting/http-status-codes) — Google Crawling Infrastructure, updated 2026-02-04. Checked 2026-10-23.
[^crawlbudget]: [Optimize your crawl budget](https://developers.google.com/crawling/docs/crawl-budget) — Google Crawling Infrastructure, updated 2026-07-22. Checked 2026-10-23.
[^faceted]: [Managing crawling of faceted navigation URLs](https://developers.google.com/crawling/docs/faceted-navigation) — Google Crawling Infrastructure, updated 2025-12-18. Checked 2026-10-23.
[^verify]: [Verify requests from Google crawlers and fetchers](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests) — Google Crawling Infrastructure, updated 2026-03-20. Checked 2026-10-23.
[^ipranges]: [New location for the Google crawlers' IP range files](https://developers.google.com/search/blog/2026/03/crawler-ip-ranges) — Google Search Central Blog, 2026-03-31. Checked 2026-10-23.
[^gcpranges]: [Obtain Google IP address ranges](https://support.google.com/a/answer/10026322) — Google Workspace Admin Help (undated). Checked 2026-10-23. Lists the customer ranges; the `googleusercontent.com` reverse name is our live DNS lookup of a listed address (2026-10-23).
[^crawlers]: [Google's common crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers) — Google Crawling Infrastructure, updated 2026-10-15. Checked 2026-10-23.
[^pause]: [Pause or disable a website](https://developers.google.com/search/docs/crawling-indexing/pause-online-business) — Google Search Central, updated 2025-12-10. Checked 2026-10-23.
[^sdpolicies]: [General structured data guidelines](https://developers.google.com/search/docs/appearance/structured-data/sd-policies) — Google Search Central, updated 2026-07-10. Checked 2026-10-23.
[^schemaorg]: [About Schema.org](https://schema.org/docs/about.html) — Schema.org (undated). Checked 2026-10-23.
[^spam]: [Spam policies for Google web search](https://developers.google.com/search/docs/essentials/spam-policies) — Google Search Central, updated 2026-08-28. Checked 2026-10-23.
[^cwv]: [Understanding Core Web Vitals and Google search results](https://developers.google.com/search/docs/appearance/core-web-vitals) — Google Search Central, updated 2025-12-10. Checked 2026-10-23.
[^genaictl]: [Search generative AI control](https://support.google.com/webmasters/answer/16908024) — Search Console Help (rollout note dated 2026-08-31). Checked 2026-10-23.
[^aiguide]: [Optimizing your website for generative AI features on Google Search](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide) — Google Search Central, updated 2026-07-10. Checked 2026-10-23.
[^aireports]: [Introducing Search Generative AI performance reports in Search Console](https://developers.google.com/search/blog/2026/06/gen-ai-performance-reports) — Google Search Central Blog, 2026-06-03 (note added for 2026-08-31 rollout). Checked 2026-10-23.
[^manualactions]: [Manual actions report](https://support.google.com/webmasters/answer/9044175) — Search Console Help (undated). Checked 2026-10-23.
[^trafficdrops]: [Debugging drops in Google Search traffic](https://developers.google.com/search/docs/monitor-debug/debugging-search-traffic-drops) — Google Search Central, updated 2025-12-10. Checked 2026-10-23.
[^pagereport]: [Page indexing report](https://support.google.com/webmasters/answer/7440203) — Search Console Help (undated). Checked 2026-10-23.
[^crawlstats]: [Crawl Stats report](https://support.google.com/webmasters/answer/9679690) — Search Console Help (undated). Checked 2026-10-23.
[^inspecttool]: [URL Inspection tool](https://support.google.com/webmasters/answer/9012289) — Search Console Help (undated). Checked 2026-10-23.
[^inspectapi]: [Method: urlInspection.index.inspect](https://developers.google.com/webmaster-tools/v1/urlInspection.index/inspect) — Google Search Console API, updated 2024-07-23. Checked 2026-10-23.
[^inspectresult]: [UrlInspectionResult](https://developers.google.com/webmaster-tools/v1/urlInspection.index/UrlInspectionResult) — Google Search Console API, updated 2025-01-21. Checked 2026-10-23.
[^gsclimits]: [Usage limits](https://developers.google.com/webmaster-tools/limits) — Google Search Console API, updated 2025-08-28. Checked 2026-10-23.
[^indexnow]: [IndexNow documentation](https://www.indexnow.org/documentation) and [FAQ](https://www.indexnow.org/faq) — IndexNow.org (undated pages; participating engines as listed on 2026-10-23). Checked 2026-10-23.
[^lhci]: [Lighthouse CI: getting started](https://github.com/GoogleChrome/lighthouse-ci/blob/main/docs/getting-started.md) — GoogleChrome on GitHub (undated; examples install `@lhci/cli@0.15.x`). Checked 2026-10-23.
[^pyrobot]: [urllib.robotparser — Parser for robots.txt](https://docs.python.org/3/library/urllib.robotparser.html) — Python Software Foundation, 3.15 documentation. The behaviour described for Python 3.8 is the author's own test, run 2026-10-23. Checked 2026-10-23.
[^pyversions]: [Status of Python versions](https://devguide.python.org/versions/) — Python Developer's Guide (undated). Checked 2026-10-23.
[^ghwf]: [Workflow syntax for GitHub Actions](https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions) — GitHub Docs (undated). Checked 2026-10-23.
[^ghsecrets]: [Using secrets in GitHub Actions](https://docs.github.com/en/actions/security-for-github-actions/security-guides/using-secrets-in-github-actions) — GitHub Docs (undated). Checked 2026-10-23.
[^dependabot]: [Troubleshooting Dependabot on GitHub Actions](https://docs.github.com/en/code-security/dependabot/troubleshooting-dependabot/troubleshooting-dependabot-on-github-actions) — GitHub Docs (undated). Checked 2026-10-23.
[^apachelog]: [mod_log_config](https://httpd.apache.org/docs/2.4/mod/mod_log_config.html) — Apache HTTP Server 2.4 documentation. Checked 2026-10-23.

**Further reading**

- [Fix canonicalization issues](https://developers.google.com/search/docs/crawling-indexing/canonicalization-troubleshooting) — Google Search Central, updated 2026-08-21.

**Changelog**

- 2026-10-23 — First version: term status confirmed as ambiguous; engineering scope per coverage matrix §3.2; three labs
  executed (robots matcher, contract gate, verified-crawler log summary), one lab syntax-checked only; 48 sources checked.
- 2026-10-23 — Revision r1 (independent review): crawler verification, contract gate, robots matcher and pipeline
  fixed and re-executed; rendered check added; Search generative AI control added; Python 3.11+; three diagrams redrawn;
  cuts to stay in the length band.
- 2026-10-23, post-score polish: render-queue, AI-control and crawler-DNS wording aligned with sources; Lab 3
  sample log printed; contract template matched to the gate.