DetectionOps

DetectionOps is detection engineering run like software delivery. A detection (a query in a SIEM, the platform that collects and searches security logs; a rule in an endpoint detection and response (EDR) tool; a Sigma rule) starts from a written requirement: which threat, on which assets, seen in which data. It lives in version control with tests proving it fires on malicious examples and stays quiet on benign ones, passes review, is deployed through staging by a pipeline, is judged by analysts’ verdicts, and is tuned or retired on purpose. The page serves detection and SOC engineers, purple teams, and leaders who must say what their detections cover.

Detections created by hand in a console cause three problems. Nobody can say what the library covers, because rules carry no requirement or evidence. Changes are invisible: a 2 a.m. exclusion creates an unknown blind spot. And rules decay silently: a parser update renames a field, and the query runs but never matches again.

Palantir’s incident response team described the result in its Alerting and Detection Strategy (ADS) framework: a “lack of rigor” meant “sub-optimal alerts going to production without documentation or peer-review”, which “led to fatigue, alerting apathy, or additional engineering time and resources”^[6]. SANS’s summary of its 2025 survey (264 respondents) says “Only 45% of organizations have a way to reliably measure the effectiveness of their detection engineering efforts”^[3]. The sponsor’s summary of the 2026 survey (307 practitioners) reports 62% using version control for rules, 58% peer review and 42% CI/CD integration^[7].

• Not the SOC, hunting or incident response. Triage belongs to SOC Ops, intelligence and hunting to ThreatOps, confirmed incidents to ResponseOps.
• Not log engineering. Onboarding, parsing and retention belong to SecDataOps; this page only names the data dependency.
• Not machine-learning detection. Anomaly detection, user and entity behaviour analytics (UEBA) and LLM triage are evaluated in AIOps for Security.

• 2013: behaviour over artefacts. David Bianco’s Pyramid of Pain (revised in 2014, when hash values became its bottom level) ranks indicators up to TTPs (tactics, techniques and procedures), where defenders work “directly on adversary behaviors, not against their tools”^[8].
• 2017: documentation and portable rules. Palantir published the ADS framework^[6]; Sigma, “for log files what Snort is for network traffic and YARA is for files”^[9], was presented at Hack.lu^[10].
• 2021–2024: maturity models. Kyle Bailey’s Detection Engineering Maturity Matrix (2021) treated detection-as-code as a technology dimension^[11]; Elastic’s DEBMM (2024) defined five tiers^[4].
• 2022–2026: the label. The European Commission’s security operations centre put OpenTIDE into operation in September 2022 and released it as open source in 2024^[12]. Its maintainers’ 2024 Hack.lu workshop opened with an “Introduction to DetectionOps with OpenTIDE”^[13]; a 2026 workshop builds an “Operational DetectionOps Platform” (due 23 October 2026)^[14].
• 2023–2026: measurement matures. CTID’s Summiting the Pyramid (2023; version 5.0 in 2026) defined robustness levels and coverage measures beyond technique tags^[15].

Mental model: a detection is a small product with a contract. For this behaviour, on these assets, using this data, it alerts within this time at roughly this precision, with these gaps accepted. Git stores the contract, tests check it, production measures it, and review dates force renewal or cancellation.

These boundaries follow the flow of security work; they are editorial distinctions, not a standard.

flowchart TB
  TO["ThreatOps"]
  RO["ResponseOps"]
  SD["SecDataOps"]
  DO["DetectionOps"]
  SOC["SOC Ops"]
  TO -->|"requests"| DO
  RO -->|"misses"| DO
  SD -->|"events"| DO
  DO -->|"data gaps"| SD
  DO -->|"alerts, runbooks"| SOC
  SOC -->|"dispositions"| DO
  SOC -->|"incidents"| RO
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  class DO core
  class TO,RO,SD,SOC neutral

No body publishes a DetectionOps manifesto. This charter is the Atlas’s proposal, distilled from the ADS framework, the SANS life cycle, Summiting the Pyramid and the maturity models cited here.

Mission. Detections for the threats that matter here, with known coverage, noise and gaps, kept true as things change.

1. Threat-informed, asset-aware. Every detection traces to a threat that can reach something you run.
2. Behaviour over artefacts. Prefer signals an adversary cannot easily avoid; hashes and tool names are short-lived additions^[8][21].
3. Everything as code, everything reviewed. Rules, tests, filters and mappings live in version control; no console-only edits.
4. Tests are part of the rule. A rule without a positive and a negative test is a draft; accepted blind spots are tests too.
5. Measure in production. Judge rules by analysts’ dispositions and by emulation results, not by whether they compile.
6. Every exclusion is a decision with an owner, an expiry and a record of what it hides.
7. Retire on purpose, with a reason, rather than silently disabling.

Non-goals. DetectionOps does not promise to detect everything, replace prevention, or own the SOC’s queue.

R = responsible, A = accountable, C = consulted, I = informed. Small teams merge roles; each decision keeps one accountable person.

The SANS poster’s life cycle runs Requirements Discovery, Triage, Investigation, Development, Testing, Deployment^[23]; CTID stresses that its workflow steps “are not strictly linear”^[5]; Sigma’s status marks maturity (experimental, test, stable), replacement (deprecated) or a rule that cannot be used as it stands (unsupported)^[24]. The Atlas adds the states needed after deployment.

stateDiagram-v2
  direction TB
  [*] --> Requested
  Requested --> Parked: no data yet
  Parked --> Requested: data arrives
  Requested --> Rejected: low priority
  Requested --> Building: accepted
  Building --> Testing: rule and tests
  Testing --> Building: test fails
  Testing --> Staged: CI and review pass
  Staged --> Production: volume as expected
  Staged --> Building: noisy or silent
  Production --> Tuning: verdicts or miss
  Tuning --> Production: fix merged
  Production --> Deprecated: replaced, obsolete or costly
  Deprecated --> [*]
  Rejected --> [*]

Staging deserves its own discipline. Run new and changed rules where they cannot page anyone (a test tenant, a separate index, alerting off) for about seven days and compare volume and samples with the request. OpenTide stages from the pull request and deploys production on merge^[25].

Change classes (Atlas proposal). Standard follows the full path. Emergency (active campaign) may skip staging, never tests or review, and is re-reviewed within five working days. Filter changes need SOC evidence and an expiry. Align with ITOps change routes where they apply.

The parts are ordinary (repository, CI runner, converter, platform APIs); what makes it DetectionOps is that every arrow is automated or recorded.

flowchart TB
  IN["Inputs: intel, hunts, incident lessons, SOC requests"]
  REPO["Repository: rules, tests, filters, strategies, mappings"]
  CI["Pipeline: lint, unit tests, convert per platform"]
  STG["Staging: shadow mode, no paging"]
  PRD["Production SIEM and EDR"]
  HM["Health monitoring: rule runs, data presence, canaries"]
  FB["Feedback: dispositions, emulation results"]
  COV["Coverage store: implementations, evidence, gaps"]
  IN --> REPO
  REPO --> CI
  CI --> STG
  STG --> PRD
  PRD --> HM
  HM --> FB
  FB --> COV
  FB -.->|"changes as pull requests"| REPO
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  class REPO,CI core
  class IN,STG,PRD,HM,FB,COV neutral

flowchart TB
  REQ["Detection request: hunt finding, intel or incident"]
  LIB["Search SigmaHQ and vendor content first"]
  PR["Pull request: rule, tests, strategy"]
  REV["Second engineer reviews"]
  STG["Staging: shadow mode, no paging"]
  TAG["Maintainer tags release-* on main"]
  PRD["Production: alerts reach the SOC"]
  REQ -->|"accepted"| LIB
  LIB --> PR
  PR -->|"CI green; no secrets used"| REV
  REV -->|"approve; merge to protected main"| STG
  STG -->|"7 days; promotion checklist passed"| TAG
  TAG -->|"production reviewer approves, not the author"| PRD
  PRD -.->|"dispositions, tuning requests"| REQ
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  class PR,STG,PRD core
  class REV,TAG decision
  class REQ,LIB neutral

Exception paths.

• No data. Park the request as a data gap for SecDataOps, naming the fields required; never write a rule against data you do not have.
• Tests or conversion fail. The pull request cannot merge. Conversion failures usually mean an unsupported modifier or unmapped field: fix the rule or pipeline, not the gate.
• Staging is noisy. Sample matches with an analyst before adding any filter.
• Staging is silent. Replay a positive sample through the platform; a rule that cannot be shown to fire stays out of production.

Search the community library first. For this page’s behaviour SigmaHQ already ships “Schedule Task Creation From Env Variable Or Potentially Suspicious Path Via Schtasks.EXE” and an event-4698 rule, “Suspicious Scheduled Task Creation”^[26]. Run your fixtures against them, keep their filters where they fit and record how your version differs; the labs start from scratch only to teach each step.

Write the strategy before the query. Palantir’s ADS template is a compact checklist: Goal, Categorization, Strategy Abstract, Technical Context, Blind Spots and Assumptions, False Positives, Validation, Priority, Response^[6]. Blind spots and validation keep coverage claims honest.

Write the failing test first. Panther recommends at least two tests per detection, “one false positive and one true positive”^[27]. Add a known-gap test for malicious activity the rule deliberately misses, so any change to the gap forces the documentation to change (Lab 2).

Treat tuning as a change, not a click. A filter needs evidence, a scope (one rule, or a Sigma filter for many^[28]), an owner and an expiry; CTID warns that filters “should be evaluated for the blind spots they may create”^[21].

Sigma in one paragraph. A rule needs a title, a logsource and a detection with a condition; the optional id should be a random version 4 UUID. “All values are treated as case-insensitive strings”, while regular expressions are case-sensitive by default; informational means “No case or alerting should be triggered”^[24].

Every coverage report should carry CTID’s sentence “An ATT&CK tag alone is not evidence of detection coverage”^[30]. Use three measures instead of one heatmap:

1. Implementation coverage: how many of a technique’s implementations your rules detect, say “2/8 implementations”^[22].
2. Detection quality: how robust and precise is each covering rule^[21]?
3. Validation evidence: was the behaviour emulated recently, and did the rule fire?

Mappings decay. ATT&CK v18 (October 2025) replaced per-technique detection text with Detection Strategies and Analytics and deprecated Data Sources^[31]; v19 (April 2026) split Defense Evasion into Stealth (TA0005) and Defense Impairment (TA0112)^[32][33]. On 16 October 2026 the Sigma tags appendix still mapped defense-evasion to TA0005^[34], while a SigmaHQ rule already carried attack.stealth^[35]. Tactic counts can drop or double after a release with no change in what you detect; “maintaining those mappings is part of maintaining the detection”^[30].

Named tools are examples verified on 2026-10-16, not recommendations.

Facts that change decisions (October 2026; for Lab 3 and the judgement call). Sentinel repository connections need the Owner role on the workspace’s resource group; only Defender XDR custom detection rules (Preview) add a Microsoft 365 E5 or equivalent licence and Defender-portal onboarding. Sentinel in the Azure portal is unsupported after 31 March 2027^[40], and Microsoft now recommends custom detections for new rules^[44]. Splunk’s contentctl is “no longer accepting new pull requests or feature requests” as investment moves to Detection Studio^[41].

Automate the deterministic steps (lint, tests, conversion, deployment, health checks, coverage reports); keep judgement (accepting a request, a filter or a gap) with people, on the record. SANS’s 2025 summary says 45% of organisations used AI for detection^[3]; the sponsor’s 2026 summary says 83% of practitioners use AI tools but only 42% “trust them for core work like tuning detections”^[7]. OpenTide argues that agents made writing a query cheap, but not knowing whether it “is valid, maps to a real threat and is deployed where it should be”^[25]; generated rules pass the same tests, review and staging.

The reference implementation covers one behaviour: persistence through a Windows scheduled task whose action runs from a user-writable folder (ATT&CK T1053.005). It is safe to reproduce; ATT&CK lists a logon task running C:\Users\Public\test.exe^[45], and DET0441 lists the telemetry: Security events 4698 (task created) and 4702 (task updated) and Sysmon process, file and registry events^[18]. It also shows robustness in miniature: watching the schtasks.exe command line “might be easier for an adversary to evade than tracking task scheduler event IDs”^[20]. So the repository holds a quick command-line rule (A) and a sturdier rule on task-creation and task-update events (B), each with tests and a strategy document.

Example status: structure and files executed in Labs 1–2 on 2026-10-17; deployment adapter not shown.

detections/
  rules/windows/
    proc_creation_win_schtasks_user_writable_action.yml   # rule A (command line)
    win_security_task_user_writable_action.yml            # rule B (events 4698, 4702)
  tests/
    d0fcbc79-de80-42d4-ab35-6f19f14047df.yml              # cases for rule A
    e71410f2-d333-4f3d-bb72-343d74241685.yml              # cases for rule B
  docs/strategies/                                         # one ADS-style file per rule
  tools/lint_rules.py  tools/run_tests.py  tools/deploy.py
  .github/workflows/detections.yml
  CODEOWNERS                                               # rules/ -> detection team

Rule A’s strategy document follows the ADS sections^[6].

Example status: illustrative template, filled in for rule A.

Goal:          Detect persistence through scheduled tasks whose action runs
               from a user-writable folder.
Categorization: ATT&CK T1053.005 (Execution, Persistence, Privilege Escalation).
Strategy:      Sysmon Event ID 1 / process creation: schtasks.exe (by image or
               PE OriginalFileName) with /create or -create and an action
               under \Users\Public\, \AppData\, \ProgramData\, \Temp\ or
               %Public%-style environment variables (a teaching subset of
               the folders in the SigmaHQ rules).
Context:       Legitimate software rarely registers tasks that run from these
               folders; per-user updaters are the known exception.
Blind spots:   Tasks created through PowerShell, COM, the API or XML import,
               and existing tasks repointed with /change or the API (both
               covered by rule B, events 4698 and 4702); direct registry
               writes, which need SYSTEM and raise neither rule (see Lab 4);
               forward-slash paths such as C:/Users/Public/ (accepted; see
               Lab 2, step 4); malware written to the ChatApp updater path
               that registers its own task (accepted, see filter_chatapp).
False pos.:    Per-user updaters (benign positives); administrator testing.
Validation:    In an authorised lab host, create a run-once task whose action
               is a copy of a harmless built-in program in C:\Users\Public;
               confirm one alert; delete the task and the copy.
Priority:      Medium; High when the user is privileged or the host is a server.
Response:      Runbook SOC-RB-031: check the task, the binary's signer and
               the user's recent logons; escalate to ResponseOps if unknown.

Labs 1–2 need only Python 3.8+ and PyYAML; Lab 3 needs GitHub and a SIEM; Lab 4 is a planning exercise.

Objective. Catch malformed rules before review and enforce house rules Sigma leaves optional. Prerequisites. Python 3.8+, PyYAML (python3 -m pip install pyyaml). Cost and risk. None.

1. Create rules/windows/, tests/ and tools/.
2. Save the linter as tools/lint_rules.py:

Example status: executed on 2026-10-17 (Python 3.8.10, PyYAML 5.3.1).

"""House-rule linter for Sigma rules (teaching subset; also run sigma check)."""
import pathlib
import re
import sys
import uuid

import yaml

STATUS = {"stable", "test", "experimental", "deprecated", "unsupported"}
LEVEL = {"informational", "low", "medium", "high", "critical"}
TAG = re.compile(r"^[a-z0-9_-]+(\.[a-z0-9_.-]+)+$")
TECHNIQUE = re.compile(r"^attack\.t\d{4}(\.\d{3})?$")
DATE = re.compile(r"^\d{4}-\d{2}-\d{2}$")


def problems(rule, path, tests_dir):
    out = []
    if not rule.get("title") or len(rule["title"]) > 256:
        out.append("title missing or longer than 256 characters")
    try:
        if uuid.UUID(str(rule.get("id"))).version != 4:
            out.append("id is not a version 4 UUID")
    except ValueError:
        out.append("id missing or not a UUID")
    if rule.get("status") not in STATUS:
        out.append("status must be in %s" % sorted(STATUS))
    if rule.get("level") not in LEVEL:
        out.append("level must be in %s" % sorted(LEVEL))
    if not DATE.match(str(rule.get("date", ""))):
        out.append("date must be YYYY-MM-DD")
    src = rule.get("logsource") or {}
    if not any(k in src for k in ("category", "product", "service")):
        out.append("logsource needs category, product or service")
    if "condition" not in (rule.get("detection") or {}):
        out.append("detection.condition is missing")
    tags = rule.get("tags") or []
    bad = [t for t in tags if not TAG.match(t)]
    if bad:
        out.append("malformed tags: %s" % bad)
    if not any(TECHNIQUE.match(t) for t in tags):
        out.append("house rule: at least one attack.tNNNN technique tag")
    if not rule.get("falsepositives"):
        out.append("house rule: list known false positives (or 'Unknown')")
    if rule.get("status") == "deprecated" and not rule.get("related"):
        out.append("deprecated rule must name its replacement in related")
    test_file = tests_dir / ("%s.yml" % rule.get("id"))
    needs_tests = rule.get("status") in {"test", "stable", "experimental"}
    if needs_tests and not test_file.exists():
        out.append("house rule: no test file %s" % test_file.name)
    return out


def main(rules_dir="rules", tests_dir="tests"):
    failed = 0
    for path in sorted(pathlib.Path(rules_dir).rglob("*.yml")):
        rule = yaml.safe_load(path.read_text())
        found = problems(rule, path, pathlib.Path(tests_dir))
        print("%s %s" % ("FAIL" if found else "PASS", path))
        for item in found:
            print("     - " + item)
        failed += bool(found)
    print("%d rule(s) failed" % failed)
    return 1 if failed else 0


if __name__ == "__main__":
    sys.exit(main(*sys.argv[1:]))

3. Save rule A, version 1, in rules/windows/ under the file name given in its first line:

Example status: executed on 2026-10-17 as input to Labs 1 and 2.

# file: proc_creation_win_schtasks_user_writable_action.yml
title: Scheduled Task Created With An Action In A User-Writable Folder
id: d0fcbc79-de80-42d4-ab35-6f19f14047df
status: experimental
description: >-
  Detects schtasks.exe creating a task whose action runs from a folder that
  any user can write to, a common persistence pattern.
references:
  - https://attack.mitre.org/techniques/T1053/005/
author: Detection team
date: 2026-10-16
tags:
  - attack.t1053.005
logsource:
  category: process_creation
  product: windows
detection:
  selection_img:
    Image|endswith: '\schtasks.exe'
  selection_create:
    CommandLine|contains: '/create'
  selection_path:
    CommandLine|contains:
      - '\Users\Public\'
      - '\Temp\'
  condition: all of selection_*
falsepositives:
  - Per-user software updaters that install under the user profile
level: medium

4. Save a deliberately broken rule as rules/windows/broken_example.yml (id: 1234, status: production, date: 16/10/2026, level: severe, the tag ATT&CK T1053, no falsepositives) and run python3 tools/lint_rules.py.

Expected output (executed):

FAIL rules/windows/broken_example.yml
     - id missing or not a UUID
     - status must be in ['deprecated', 'experimental', 'stable', 'test', 'unsupported']
     - level must be in ['critical', 'high', 'informational', 'low', 'medium']
     - date must be YYYY-MM-DD
     - malformed tags: ['ATT&CK T1053']
     - house rule: at least one attack.tNNNN technique tag
     - house rule: list known false positives (or 'Unknown')
FAIL rules/windows/proc_creation_win_schtasks_user_writable_action.yml
     - house rule: no test file d0fcbc79-de80-42d4-ab35-6f19f14047df.yml
2 rule(s) failed

5. Delete the broken rule. Rule A still fails until it has tests.

Validation. Exit code 1 while any rule fails, 0 once all pass (after Lab 2 step 1). Troubleshooting. ScannerError: use single quotes for Windows paths. Teardown. Keep the folder for Lab 2.

Objective. Write positive, negative, out-of-scope and known-gap tests, watch version 1 fail, improve it, and measure a filter’s precision–recall trade-off. Prerequisites. Lab 1. Cost and risk. None; synthetic events.

1. Save the cases as tests/d0fcbc79-de80-42d4-ab35-6f19f14047df.yml (the first follows an ATT&CK procedure example^[45]):

Example status: executed on 2026-10-17.

rule_id: d0fcbc79-de80-42d4-ab35-6f19f14047df
cases:
  - name: logon task runs from Users\Public
    expect: match
    event:
      Image: 'C:\Windows\System32\schtasks.exe'
      OriginalFileName: schtasks.exe
      CommandLine: >-
        schtasks /create /tn "mysc" /tr C:\Users\Public\test.exe
        /sc ONLOGON /ru "System"
  - name: action in a roaming profile
    expect: match
    event:
      Image: 'C:\Windows\System32\schtasks.exe'
      OriginalFileName: schtasks.exe
      CommandLine: >-
        schtasks /CREATE /tn Sync /sc daily
        /tr C:\Users\alice\AppData\Roaming\sync\s.exe
  - name: renamed copy of schtasks
    expect: match
    event:
      Image: 'C:\Users\Public\tsk.exe'
      OriginalFileName: schtasks.exe
      CommandLine: 'tsk.exe /create /tn upd /tr C:\Users\Public\u.exe /sc onlogon'
  - name: dash switches
    expect: match
    event:
      Image: 'C:\Windows\System32\schtasks.exe'
      OriginalFileName: schtasks.exe
      CommandLine: 'schtasks -create -tn upd -tr C:\Users\Public\u.exe -sc onlogon'
  - name: unexpanded environment variable
    expect: match
    event:
      Image: 'C:\Windows\System32\schtasks.exe'
      OriginalFileName: schtasks.exe
      CommandLine: 'schtasks /create /tn upd /tr %PUBLIC%\u.exe /sc onlogon'
  - name: action under ProgramData
    expect: match
    event:
      Image: 'C:\Windows\System32\schtasks.exe'
      OriginalFileName: schtasks.exe
      CommandLine: 'schtasks /create /tn upd /tr C:\ProgramData\u.exe /sc onlogon'
  - name: hides behind the updater path
    expect: match
    event:
      ParentImage: 'C:\Windows\System32\cmd.exe'
      Image: 'C:\Windows\System32\schtasks.exe'
      OriginalFileName: schtasks.exe
      CommandLine: >-
        schtasks /create /sc onlogon /tr C:\Users\Public\u.exe
        /tn C:\Users\eve\AppData\Local\ChatApp\Update.exe
  - name: vendor updater in Program Files
    expect: no_match
    event:
      Image: 'C:\Windows\System32\schtasks.exe'
      OriginalFileName: schtasks.exe
      CommandLine: >-
        schtasks /create /tn "Vendor\Update" /sc daily
        /tr "C:\Program Files\Vendor\update.exe"
  - name: administrator lists tasks
    expect: no_match
    event:
      Image: 'C:\Windows\System32\schtasks.exe'
      OriginalFileName: schtasks.exe
      CommandLine: 'schtasks /query /fo LIST /v'
  - name: chat app per-user updater
    expect: no_match
    event:
      ParentImage: 'C:\Users\bob\AppData\Local\ChatApp\Update.exe'
      Image: 'C:\Windows\System32\schtasks.exe'
      OriginalFileName: schtasks.exe
      CommandLine: >-
        schtasks /create /tn "ChatApp Update" /sc daily
        /tr "C:\Users\bob\AppData\Local\ChatApp\Update.exe --check" /f
  - name: malware at the updater path
    expect: known_gap
    owner: filter_chatapp
    event:
      ParentImage: 'C:\Users\eve\AppData\Local\ChatApp\Update.exe'
      Image: 'C:\Windows\System32\schtasks.exe'
      OriginalFileName: schtasks.exe
      CommandLine: >-
        schtasks /create /tn "ChatApp Update" /sc onlogon
        /tr C:\Users\eve\AppData\Local\ChatApp\Update.exe
  - name: task registered from PowerShell
    expect: out_of_scope
    owner: rule e71410f2
    event:
      Image: 'C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe'
      OriginalFileName: PowerShell.EXE
      CommandLine: 'powershell.exe -File C:\Users\Public\register.ps1'
  - name: task repointed with /change
    expect: out_of_scope
    owner: rule e71410f2
    event:
      Image: 'C:\Windows\System32\schtasks.exe'
      OriginalFileName: schtasks.exe
      CommandLine: 'schtasks /change /tn Existing /tr C:\Users\Public\u.exe'

2. Save the runner as tools/run_tests.py: a teaching subset of Sigma with case-insensitive matching as the specification requires^[24]. It ignores logsource, which real backends map through pipelines.

Example status: executed on 2026-10-17 (Python 3.8.10, PyYAML 5.3.1). Teaching code, not a replacement for pySigma.

"""Run labelled events against Sigma rules (teaching subset of Sigma).

Supports: maps and lists of maps; modifiers contains, startswith, endswith, all,
windash; * and ? wildcards; conditions with and/or/not, brackets, '1 of x*',
'all of x*'. Matching is case-insensitive, as the Sigma specification requires.
"""
import fnmatch
import itertools
import pathlib
import re
import sys

import yaml

DASHES = ("-", "/", "\u2013", "\u2014", "\u2015")  # windash characters


def windash(value):
    """Every variant of value with each flag's leading - or / swapped."""
    parts = re.split(r"\B[-/]\b", value)
    return ["".join(p + d for p, d in zip(parts, combo + ("",)))
            for combo in itertools.product(DASHES, repeat=len(parts) - 1)]


def value_matches(actual, expected, mods):
    if "windash" in mods:
        rest = [m for m in mods if m != "windash"]
        return any(value_matches(actual, v, rest) for v in windash(str(expected)))
    a, e = str(actual).lower(), str(expected).lower()
    if "contains" in mods:
        return e in a
    if "startswith" in mods:
        return a.startswith(e)
    if "endswith" in mods:
        return a.endswith(e)
    return fnmatch.fnmatchcase(a, e)


def map_matches(event, mapping):
    for key, expected in mapping.items():
        field, *mods = key.split("|")
        if field not in event:
            return False
        values = expected if isinstance(expected, list) else [expected]
        hits = [value_matches(event[field], v, mods) for v in values]
        if not (all(hits) if "all" in mods else any(hits)):
            return False
    return True


def search_matches(event, search):
    if isinstance(search, list):  # list of maps = OR
        return any(map_matches(event, item) for item in search)
    return map_matches(event, search)


TOKEN = re.compile(r"\(|\)|1 of [\w*]+|all of [\w*]+|\band\b|\bor\b|\bnot\b|[\w*]+")


def evaluate(condition, results):
    tokens = TOKEN.findall(condition)
    pos = [0]

    def peek():
        return tokens[pos[0]] if pos[0] < len(tokens) else None

    def take():
        pos[0] += 1
        return tokens[pos[0] - 1]

    def group(tok):
        quant, pattern = tok.split(" of ")
        names = [n for n in results if fnmatch.fnmatchcase(n, pattern)]
        hits = [results[n] for n in names]
        return all(hits) if quant == "all" else any(hits)

    def atom():
        tok = take()
        if tok == "(":
            val = expr()
            take()  # ")"
            return val
        if tok == "not":
            return not atom()
        if " of " in tok:
            return group(tok)
        return results[tok]

    def term():
        val = atom()
        while peek() == "and":
            take()
            rhs = atom()
            val = val and rhs
        return val

    def expr():
        val = term()
        while peek() == "or":
            take()
            rhs = term()
            val = val or rhs
        return val

    return expr()


def rule_matches(rule, event):
    det = rule["detection"]
    results = {k: search_matches(event, v)
               for k, v in det.items() if k != "condition"}
    return evaluate(det["condition"], results)


def main(rules_dir="rules", tests_dir="tests"):
    rules = {}
    for path in pathlib.Path(rules_dir).rglob("*.yml"):
        rule = yaml.safe_load(path.read_text())
        rules[rule["id"]] = rule
    tp = fp = fn = tn = failures = 0
    for test_path in sorted(pathlib.Path(tests_dir).glob("*.yml")):
        suite = yaml.safe_load(test_path.read_text())
        rule = rules[suite["rule_id"]]
        print("%s (%s)" % (rule["title"], rule["status"]))
        for case in suite["cases"]:
            fired = rule_matches(rule, case["event"])
            want = case["expect"]
            if want == "out_of_scope":
                print("  SKIP %-34s fired=%s owner=%s"
                      % (case["name"], fired, case["owner"]))
                continue
            if want == "known_gap":  # malicious, accepted blind spot
                fn += not fired
                failures += fired  # gap closed: update the test
                print("  GAP  %-34s fired=%s accepted=%s"
                      % (case["name"], fired, case["owner"]))
                continue
            ok = fired == (want == "match")
            failures += not ok
            tp += fired and want == "match"
            fp += fired and want == "no_match"
            fn += (not fired) and want == "match"
            tn += (not fired) and want == "no_match"
            print("  %s %-34s expect=%-8s fired=%s"
                  % ("PASS" if ok else "FAIL", case["name"], want, fired))
    precision = tp / (tp + fp) if tp + fp else float("nan")
    recall = tp / (tp + fn) if tp + fn else float("nan")
    print("TP=%d FP=%d FN=%d TN=%d precision=%.2f recall=%.2f failures=%d"
          % (tp, fp, fn, tn, precision, recall, failures))
    return 1 if failures else 0


if __name__ == "__main__":
    sys.exit(main(*sys.argv[1:]))

3. Run python3 tools/run_tests.py against version 1. Expected output (executed; exit code 1):

Scheduled Task Created With An Action In A User-Writable Folder (experimental)
  PASS logon task runs from Users\Public  expect=match    fired=True
  FAIL action in a roaming profile        expect=match    fired=False
  FAIL renamed copy of schtasks           expect=match    fired=False
  FAIL dash switches                      expect=match    fired=False
  FAIL unexpanded environment variable    expect=match    fired=False
  FAIL action under ProgramData           expect=match    fired=False
  PASS hides behind the updater path      expect=match    fired=True
  PASS vendor updater in Program Files    expect=no_match fired=False
  PASS administrator lists tasks          expect=no_match fired=False
  PASS chat app per-user updater          expect=no_match fired=False
  GAP  malware at the updater path        fired=False accepted=filter_chatapp
  SKIP task registered from PowerShell    fired=False owner=rule e71410f2
  SKIP task repointed with /change        fired=False owner=rule e71410f2
TP=2 FP=0 FN=6 TN=3 precision=1.00 recall=0.25 failures=5

4. Replace the detection block with version 2. It accepts the name compiled into the executable’s header (OriginalFileName), adds \AppData\, \ProgramData\ and environment variables, and uses windash because “Windows command line flags can often be indicated by both characters”^[46]. Its filter for the updater the SOC reported checks two fields, parent process and action path; an endswith on the whole command line would hide the “hides behind the updater path” case and miss this updater’s quoted /tr. The path lists are a teaching subset: production rules add folders such as \Perflogs\, \Downloads\ and \Desktop\ (compare the SigmaHQ rules^[26]); forward-slash spellings (C:/Users/Public/) match neither rule and are an accepted gap in the strategy.

Example status: executed on 2026-10-17.

detection:
  selection_img:
    - Image|endswith: '\schtasks.exe'
    - OriginalFileName: 'schtasks.exe'
  selection_create:
    CommandLine|contains|windash: '/create'
  selection_path:
    CommandLine|contains:
      - '\Users\Public\'
      - '\AppData\'
      - '\ProgramData\'
      - '\Temp\'
      - '%Public%'
      - '%AppData%'
      - '%LocalAppData%'
      - '%Temp%'
      - '%TMP%'
  filter_chatapp:
    ParentImage|endswith: '\AppData\Local\ChatApp\Update.exe'
    CommandLine|contains: '\AppData\Local\ChatApp\Update.exe'
  condition: all of selection_* and not filter_chatapp

5. Run the tests again. Expected output (executed; exit code 0):

Scheduled Task Created With An Action In A User-Writable Folder (experimental)
  PASS logon task runs from Users\Public  expect=match    fired=True
  PASS action in a roaming profile        expect=match    fired=True
  PASS renamed copy of schtasks           expect=match    fired=True
  PASS dash switches                      expect=match    fired=True
  PASS unexpanded environment variable    expect=match    fired=True
  PASS action under ProgramData           expect=match    fired=True
  PASS hides behind the updater path      expect=match    fired=True
  PASS vendor updater in Program Files    expect=no_match fired=False
  PASS administrator lists tasks          expect=no_match fired=False
  PASS chat app per-user updater          expect=no_match fired=False
  GAP  malware at the updater path        fired=False accepted=filter_chatapp
  SKIP task registered from PowerShell    fired=False owner=rule e71410f2
  SKIP task repointed with /change        fired=False owner=rule e71410f2
TP=7 FP=0 FN=1 TN=3 precision=1.00 recall=0.88 failures=0

6. See the trade-off. Remove and not filter_chatapp and rerun: on 2026-10-17 the result was TP=7 FP=1 FN=0 TN=2 precision=0.88 recall=1.00 failures=2. Without the filter the updater alerts (in production, a benign positive); with it, malware written to that path is invisible, and the known-gap test records the decision. Restore the filter.

7. Cover the out-of-scope cases with rule B. Save the rule on events 4698 and 4702 and its tests, then run a third time.

Example status: executed on 2026-10-17.

title: Scheduled Task Registered Or Updated To Run From A User-Writable Folder
id: e71410f2-d333-4f3d-bb72-343d74241685
status: experimental
description: >-
  Detects any scheduled task registration or update (schtasks, PowerShell,
  COM or API) whose action runs from a user-writable folder, using Security
  events 4698 (task created) and 4702 (task updated).
references:
  - https://attack.mitre.org/detectionstrategies/DET0441/
author: Detection team
date: 2026-10-16
tags:
  - attack.t1053.005
logsource:
  product: windows
  service: security
  definition: Requires the audit subcategory "Audit Other Object Access Events"
detection:
  selection_created:
    EventID: 4698
    TaskContent|contains:
      - '\Users\Public\'
      - '\AppData\'
      - '\ProgramData\'
      - '\Temp\'
      - '%Public%'
      - '%AppData%'
      - '%LocalAppData%'
      - '%Temp%'
      - '%TMP%'
  selection_updated:
    EventID: 4702
    TaskContentNew|contains:
      - '\Users\Public\'
      - '\AppData\'
      - '\ProgramData\'
      - '\Temp\'
      - '%Public%'
      - '%AppData%'
      - '%LocalAppData%'
      - '%Temp%'
      - '%TMP%'
  condition: 1 of selection_*
falsepositives:
  - Per-user software updaters that install under the user profile
level: medium

rule_id: e71410f2-d333-4f3d-bb72-343d74241685
cases:
  - name: task registered from PowerShell
    expect: match
    event:
      EventID: 4698
      TaskName: '\Updater'
      TaskContent: '<Exec><Command>C:\Users\Public\u.exe</Command></Exec>'
  - name: action as an environment variable
    expect: match
    event:
      EventID: 4698
      TaskName: '\Upd'
      TaskContent: '<Exec><Command>%PUBLIC%\u.exe</Command></Exec>'
  - name: existing task repointed to TMP
    expect: match
    event:
      EventID: 4702
      TaskName: '\Microsoft\StartListener'
      TaskContentNew: '<Exec><Command>%TMP%\listener.exe</Command></Exec>'
  - name: vendor task in Program Files
    expect: no_match
    event:
      EventID: 4698
      TaskName: '\Vendor\Update'
      TaskContent: '<Exec><Command>C:\Program Files\Vendor\update.exe</Command></Exec>'
  - name: vendor update in Program Files
    expect: no_match
    event:
      EventID: 4702
      TaskName: '\Vendor\Update'
      TaskContentNew: '<Exec><Command>C:\Program Files\Vendor\update.exe</Command></Exec>'
  - name: inline encoded PowerShell action
    expect: known_gap
    owner: backlog DR-0142
    event:
      EventID: 4698
      TaskName: '\Sync'
      TaskContent: >-
        <Exec><Command>powershell.exe</Command>
        <Arguments>-enc BASE64_PAYLOAD</Arguments></Exec>

Expected output of the third run (executed; last lines, exit code 0):

Scheduled Task Registered Or Updated To Run From A User-Writable Folder (experimental)
  PASS task registered from PowerShell    expect=match    fired=True
  PASS action as an environment variable  expect=match    fired=True
  PASS existing task repointed to TMP     expect=match    fired=True
  PASS vendor task in Program Files       expect=no_match fired=False
  PASS vendor update in Program Files     expect=no_match fired=False
  GAP  inline encoded PowerShell action   fired=False accepted=backlog DR-0142
TP=10 FP=0 FN=2 TN=5 precision=1.00 recall=0.83 failures=0

What to notice. Event 4698 exists only if “Audit Other Object Access Events” is enabled^[47], and Sigma’s definition is documentation, not configuration^[24]: put the dependency in the request and a health check. Rule B ignores which tool registered the task, provided Task Scheduler logs it, and sees a task repointed with schtasks /change or the API through event 4702, whose new definition is in TaskContentNew, not TaskContent^[48]. A task written straight into the registry (needs SYSTEM and a service restart) “can bypass the generation of scheduled task creation event logs, such as 4698 and 106”^[49]; look for it in DET0441’s Sysmon registry events^[18]. Validation. Break one positive fixture and confirm the run fails. Troubleshooting. KeyError: a test’s rule_id matches no rule id. Teardown. Keep the folder for Lab 3.

Objective. Validate every pull request, deploy merges in shadow mode, and promote only from a reviewed release tag on main. Prerequisites. A private GitHub repository with the Lab 2 folder; a SIEM API with a least-privileged token per environment; your own tools/deploy.py adapter (for Splunk, around the saved/searches REST endpoint; not shown). Cost and risk. CI minutes; a bad production deploy can flood the SOC.

Example status: syntax-checked, not executed (needs GitHub, a SIEM and Python 3.10 or later for sigma-cli 3.1.0). Action majors are current as of October 2026^[50]; pin each action to a full commit SHA in production.

name: detections
on:
  pull_request:
    paths: ["rules/**", "tests/**", "tools/**", ".github/workflows/**"]
  push:
    branches: [main]
    tags: ["release-*"]
permissions:
  contents: read
jobs:
  validate:
    runs-on: ubuntu-24.04
    steps:
      - uses: actions/checkout@v7      # pin to a full commit SHA in production
      - uses: actions/setup-python@v7
        with:
          python-version: "3.12"
      - name: Install the pinned toolchain
        run: >-
          python -m pip install "sigma-cli==3.1.0" "pysigma==1.5.1"
          "pysigma-backend-splunk==2.1.0" "pysigma-pipeline-sysmon==2.0.0"
          "pysigma-pipeline-windows==2.0.1" "pyyaml==6.0.3"
      - name: House rules and unit tests
        run: |
          python tools/lint_rules.py
          python tools/run_tests.py
      - name: Sigma validators (fail on issues, not only on errors)
        run: |
          mkdir -p build
          sigma check --fail-on-issues --junitxml build/sigma-check.xml rules/
      - name: Convert for Splunk
        run: >-
          sigma convert -t splunk -p sysmon -p windows-logsources
          -f savedsearches -o build/savedsearches.conf rules/
      - uses: actions/upload-artifact@v7
        with:
          name: converted
          path: build/
  deploy-staging:
    if: github.ref == 'refs/heads/main'
    needs: validate
    runs-on: ubuntu-24.04
    environment: detections-staging      # deployment branch rule: main only
    steps:
      - uses: actions/checkout@v7
      - uses: actions/download-artifact@v8
        with:
          name: converted
          path: build/
      - name: Deploy in shadow mode (no notable events, no paging)
        env:
          SIEM_TOKEN: ${{ secrets.SIEM_TOKEN }}
        run: python tools/deploy.py --mode shadow build/savedsearches.conf
  deploy-production:
    if: startsWith(github.ref, 'refs/tags/release-')
    needs: validate
    runs-on: ubuntu-24.04
    environment: detections-production   # tag rule release-*, required reviewer
    steps:
      - uses: actions/checkout@v7
        with:
          fetch-depth: 0                 # all branches, for the next step
      - name: Refuse a tag whose commit is not on main
        run: git merge-base --is-ancestor "$GITHUB_SHA" origin/main
      - uses: actions/download-artifact@v8
        with:
          name: converted
          path: build/
      - name: Promote to alerting
        env:
          SIEM_TOKEN: ${{ secrets.SIEM_TOKEN }}
        run: python tools/deploy.py --mode alert build/savedsearches.conf

Why each line is there.

• sigma check passes on validation issues unless you add --fail-on-issues^[51].
• paths includes the workflow folder, so a pin bump runs validate instead of shipping untested or, if validate is a required check, never merging.
• Conversion follows the sigma-cli README^[16]. The Sysmon pipeline rewrites only generic categories such as process_creation; windows-logsources adds rule B’s Windows Channel condition^[52]. Check the converted rule B for it.
• Who can reach each SIEM token. The validate job runs unreviewed pull-request code: contents: read, no environment. Keep each token only as an environment secret, never a repository or organisation secret: such secrets “are only available to workflow jobs that reference the environment”^[53], on any branch unless restricted. Limit detections-staging to main with a deployment branch rule, and protect main with required review and status checks; the if: lines are convenience, not control.
• What makes a release reviewed. A deployment tag rule matches only the name in GITHUB_REF^[53]; any writer can push release-x on any commit unless a tag ruleset restricts creating and updating release-* to maintainers^[54]. The ancestry step refuses tags outside main. On detections-production, require reviewers, prevent self-review and disallow the default administrator bypass; a job awaiting approval cannot read the environment’s secrets.
• Plan limits (October 2026). In private repositories, environments, their secrets and branch and tag rules need GitHub Pro, Team or Enterprise^[55]; rulesets need Pro, Team or Enterprise Cloud^[54]; required reviewers need Enterprise^[53][55], and self-hosted runners are not isolated^[53]. Each token may only create and update detections in one app or workspace.

Expected output. A pull request runs only validate; a merge to main runs deploy-staging; a release-* tag on main waits for a reviewer, then deploys; a tag elsewhere fails the ancestry step. Validation. A broken fixture fails the pull request and deploys nothing; a branch workflow naming detections-staging is refused. Troubleshooting. ResolutionImpossible from pip: one package moved to pySigma 2 early; keep pins together. Teardown. Delete the test tag and staging searches; revoke lab tokens.

Objective. Turn “we detect scheduled-task persistence” into evidence. Prerequisites. An authorised, isolated lab host sending telemetry to staging; the SOC informed. Status. A worksheet, nothing executed. Its Expected output, Validation and Teardown lines are the acceptance criteria:

Behaviour:        T1053.005 Scheduled Task (persistence)
Implementations:  1 schtasks.exe   2 PowerShell cmdlet   3 COM/API   4 XML import
                  5 direct registry write (needs SYSTEM)
                  6 existing task repointed (schtasks /change or the API)
Expected signal:  impl 1 -> rules A and B;  impl 2-4 -> rule B only (4698);
                  impl 5 -> neither: known gap (look at Sysmon 13/14, DET0441);
                  impl 6 -> rule B only (4702)
Test source:      Atomic Red Team test chosen by GUID (GUIDs do not change)
Harmless payload: copy of a built-in program; no network, no credentials
Expected output:  each implementation raises its expected alert within 15 min
Validation:       alert and raw event fall inside the run's window, per GUID
Evidence kept:    alert IDs, raw events, test GUID, time window, engineer
Teardown:         delete the task and the copied file; confirm with a query
Stop condition:   any unexpected impact on the host or the SOC queue

Atomic Red Team tests are designed to run “in five minutes or less”, many with cleanup commands^[56]; select them by GUID because GUIDs “are guaranteed to not change”^[57]; the detected-or-missed record per implementation feeds validated coverage.

1. Sourced: the European Commission’s security operations centre. Before OpenTide, threat information went into spreadsheets whose “rows could not be linked to the rules they deployed”, analyst Claus Houmann recalls. Since September 2022, Remi Seguy’s team has used it to “manage hundreds of threat vectors, each linked to detection objectives and deployed detection rules”; Seguy says “Every rule is validated, greatly reducing the risk of errors”^[12]. Lesson: traceability was the problem; tooling followed. Caveat: the Commission’s own account; no independent evaluation found.

Measure the library, not people; the formulas are the Atlas’s proposals. Precision uses SOC Ops dispositions: benign positives count against it like false positives, since both cost analyst time, but need a different fix (context or a scoped filter); undetermined alerts are reported separately. Teams judging logic, not analyst load, may count benign positives as correct: Magonia classes expected activity as a true positive with “No Business Impact”, defines precision as “how often the rule fires correctly” and adds a separate impact rate^[58]. Label each report’s convention.

Test completeness, rule health and filter debt lead; validated coverage lags but is closest to the truth; lead time and change failure rate come from DevOps. The sponsor’s 2026 summary reports 59% of teams track false-positive rates but only 14% prioritise reducing them^[7].

Worked calculation (illustrative numbers). In September a team has 120 production rules.

• Rule A: 40 alerts, 6 true, 30 benign and 4 false positives. Precision = 6 ÷ (6 + 30 + 4) = 15%: noisy-rule review.
• 84 rules have a positive and a negative test. Test completeness = 84 ÷ 120 = 70%.
• Emulation in 90 days detected 18 of 30 prioritised implementations. Validated coverage = 18 ÷ 30 = 60%; a tag heatmap shows 100%.
• 114 rules passed health checks. Rule health = 114 ÷ 120 = 95%.
• 3 of 50 deployments were rolled back within a week. Change failure rate = 3 ÷ 50 = 6%.

Set targets per rule class: a rule that pages on-call may need precision above 50%; a low-severity rule feeding a correlation can tolerate less.

• The repository is sensitive. It lists what you watch and, through filters and known gaps, what you do not. Palantir notes the “operational security considerations around publicly acknowledging and documenting internal alerts”^[6]; the Atlas adds: keep it private, use code owners, and require review for filter changes.
• Pipeline credentials are scoped per environment (Lab 3). ATT&CK’s Defense Impairment tactic covers attempts to “break security mechanisms, pipelines, and tooling”^[33], yours included.
• Test data. Prefer synthetic or sanitised samples; real events carry user and host names (PrivacyOps).
• Auditability. Git history shows who changed which detection and why. Report against NIST CSF 2.0 outcomes DE.CM and DE.AE^[59]; CSF is voluntary unless mandated; sector monitoring duties belong to ComplianceOps.

Telemetry: Microsoft warns that Sysmon ProcessAccess events “can generate significant amounts of logging”^[60]; requests state their data needs, and SecDataOps decides ingest. Compute: long look-backs, correlations and real-time rules consume SIEM capacity. People: SANS’s 2025 summary lists high false-positive rates in vendor detection tools (64%) among the most-cited technology challenges, and 71% cite resource and time constraints as their primary obstacle^[3]. A useful unit is analyst minutes per true positive: (alerts × average triage minutes) ÷ true positives. Rule A’s 40 alerts at 6 minutes for 6 true positives cost 40 minutes each. Retirement is the cheapest capacity.

• Sampled review: each month, check five random production rules for requirement, strategy, tests, owner and review date.
• Deployed equals repository: a nightly job reports drift between platforms and Git (console edits, disabled rules).
• Gap register: known-gap tests and expired filters appear in the quarterly coverage report.
• Independent validation: an annual exercise by a red team tests implementations the detection team did not choose.

Situation (illustrative). Rule B (events 4698 and 4702) usually raises 2–5 alerts a week; the weekly review notices eleven silent days.

1. Did the deployed rule run? Check the rule-run log and diff the deployed query with Git. In Microsoft Sentinel, with health monitoring on, the SentinelHealth table records for each run “if they succeeded or failed, and if they failed, why”, and shows auto-disabled rules and skipped windows^[61]. Result: all runs succeeded; the query matches Git.
2. Is the data arriving? Count events 4698 and 4702 per day and host for 14 days. Result: it stopped on 80% of hosts on one day.
3. What changed? Check the change calendar (SecOps). Result: a Group Policy change rebuilt the workstation audit policy without “Audit Other Object Access Events”, the subcategory that produces both events^[47][48].
4. Contain the blind spot. Open a data-gap incident with SecDataOps and the policy owner; meanwhile raise rule A’s priority, knowing it sees only the schtasks.exe path.
5. Hunt the gap window. Ask ThreatOps to hunt the eleven days with DET0441’s other telemetry (Sysmon process and registry events)^[18].
6. Restore and verify through the deployed rule. After the fix, repeat the Lab 4 validation on one lab host; the alert must come from the production rule, not a fixture.
7. Prevent recurrence. Alert when a host sends Security events but no 4698 for 72 hours, and on failed or skipped runs; add the audit subcategory to the rule’s dependencies and to the audit-policy change checklist.

Exit criteria. Runs healthy; 4698 on all monitored hosts; validation alert from the production rule; gap hunt closed.

flowchart TB
  R["Silent rule: ran, and matches Git?"]
  B["Raw events arriving on time?"]
  D["Fixture matches live fields?"]
  F["Canary fires through the deployed rule?"]
  Q["Quiet is real: record it"]
  RX["Fix the run or redeploy"]
  C["Data gap or delay: SecDataOps"]
  E["Fix mapping or rule"]
  R -->|"yes"| B
  B -->|"yes"| D
  D -->|"yes"| F
  F -->|"yes"| Q
  R -->|"no"| RX
  B -->|"no"| C
  D -->|"no"| E
  F -->|"no"| E
  RX ~~~ C
  C ~~~ E
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  class R,B,D,F decision
  class Q core
  class RX,C,E risk

• “We cover 80% of ATT&CK.” A tag count, not coverage; report implementations and validation evidence^[30].
• Console hot-fixes. A night-time exclusion in the SIEM UI never reaches Git; the next deploy erases it.
• Tests written from the rule. Fixtures derived from the query pass by construction; derive them from the threat and real benign activity.
• Importing a pack wholesale. SigmaHQ’s 3,000+ rules^[10] and vendor content are starting points to adapt and test one by one; enabling everything buys noise and false confidence.
• Indicator rules as detections. Hash and IP matches sit on the two lowest levels of the Pyramid of Pain^[8]; give them expiry dates and behavioural replacements.
• Myth: detection-as-code needs a big platform. Labs 1–3 need Git, two scripts, a CI runner and one API.

Adopt the full practice when you write or change custom detections, more than one person edits them, the SOC reports noise or misses, or nobody can answer “what do we detect?” with evidence; multi-platform and multi-tenant teams gain most. Use a light version with mainly vendor or MDR content: custom rules and exclusions in Git, filter reviews with the provider, periodic emulation.

Do not start here when:

• The data is missing. Start with SecDataOps; SANS’s 2025 summary says 47% of respondents report inadequate access to the data feeds needed^[3].
• Nobody works the alerts. Better detections become better-ignored alerts; start with SOC Ops.
• Prevention basics are missing. The same effort may reduce more risk in VulnerabilityOps or IAMOps.
• The platform has no content API. Keep the repository as documentation and reconcile by export.

Hidden responsibilities: someone must own the toolchain (pins, upgrades, remapping), health checks and the emulation lab.

flowchart TB
  Q1["Custom detections?"]
  Q2["Telemetry parsed?"]
  Q3["Alerts triaged?"]
  Q4["Several platforms?"]
  L1["Light version"]
  L2["Fix data first"]
  L3["Fix triage first"]
  L4["Native queries as code"]
  L5["Sigma or threat model"]
  Q1 -->|"no"| L1
  Q1 -->|"yes"| Q2
  Q2 -->|"no"| L2
  Q2 -->|"yes"| Q3
  Q3 -->|"no"| L3
  Q3 -->|"yes"| Q4
  Q4 -->|"no"| L4
  Q4 -->|"yes"| L5
  L1 ~~~ L2
  L2 ~~~ L3
  L3 ~~~ L4
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  class Q1,Q2,Q3,Q4 decision
  class L1,L4,L5 core
  class L2,L3 risk

For tool choices, score candidates 1–5 against weights you can defend; an example for a single-SIEM team:

This is the Atlas’s model, not a standard. It parallels Elastic’s five DEBMM tiers^[4] and the Detection Engineering Maturity Matrix^[11], with exit criteria you can check in a repository.

Elastic warns that rushing tiers “without proper budget and staff can lead to burnout”^[4]. Reach level 3 for your top threats before level 2 everywhere.

Quick wins: visible exclusions and evidence-based tuning. Dependencies: SecDataOps (data), SOC Ops (dispositions), ThreatOps (priorities), platform owners (API tokens).

Pre-merge checklist (every pull request)

• Request linked; threat, assets and data stated
• Strategy document updated, blind spots included
• At least one positive and one negative test; known gaps written as tests
• Technique IDs checked against the current ATT&CK release
• Filters have reason, owner and expiry
• CI green: lint, tests, sigma check --fail-on-issues or native validator, conversion
• Reviewer is not the author

Promotion checklist (staging to production)

• Staging period completed; volume within the estimate; matches sampled with an analyst
• Runbook linked
• Severity agreed with the SOC lead
• Health checks defined (rule runs, data presence, canary)
• Release tag created by a maintainer on main

Monthly health review

• Silent rules investigated (runs, data or logic)
• Expired filters renewed with evidence or removed
• Rules past their review date renewed or deprecated
• Deployed-versus-repository drift is zero
• Toolchain pins reviewed

Quarterly governance review

• Coverage by implementation, with evidence and gap register, reviewed with risk owners; mappings current

Detection request (intake form)

Example status: template.

Request ID:        DR-____
Requested by:      (team, person)           Date:
Source:            intel report / hunt / incident / audit / SOC
Behaviour:         (plain words + ATT&CK technique or sub-technique)
Implementations:   (which ways of doing it must be detected)
Assets in scope:   (systems, criticality)
Data needed:       (log source, event IDs, fields; available? Y/N)
Evidence:          (hunt query, incident timeline, report link)
Expected volume:   (alerts per week, best guess)
Response:          (what the SOC should do; runbook exists? Y/N)
Priority:          (impact x likelihood, with reasons)
Decision:          accepted / parked (data gap) / rejected   By:      Date:

Filter record (one per exclusion)

Example status: template.

Filter ID:     FLT-____        Applies to rules:  (IDs)
Reason:        (alert samples, analyst evidence)
Logic:         (fields and values excluded)
Blind spot:    (what an attacker could hide behind this filter)
Known-gap test: (test file and case name)
Owner:         (person)        Expiry: YYYY-MM-DD   Reviewed by:

Retirement record

Example status: template.

Rule ID / title:   (id, title)               Status -> deprecated
Reason:            replaced / obsolete technique / data source retired /
                   cost exceeds value (precision, analyst minutes per TP)
Replacement:       (rule IDs, in Sigma "related" with type)
Coverage impact:   (implementations no longer detected, if any)
Approved by:       (detection lead)  SOC informed: (date)

Sigma, native queries, or a threat-to-rule model

Is DetectionOps the same as detection engineering? In practice yes: the label names detection engineering run as a DevOps-style loop^[2]. Hire under the standard names.

Do we need Sigma? No. It helps with several platforms or community content; one platform can keep native queries as code with the same gates.

How many tests does a rule need? One that must fire and one that must not^[27], a known-gap test per accepted blind spot, and a real regression sample when you can^[43].

Our vendor claims 90% ATT&CK coverage. Is that coverage? It is a mapping. Ask which implementations, on which assets, at what precision, validated when^[30].

Should detection content be open source? Generic rules often, yes; your filters, gaps and priorities, no, since they map your blind spots^[6].

• ATT&CK detection strategies (established since v18). v20 lists 703 Enterprise detection strategies and 1,770 analytics^[29].
• Coverage measurement (maturing). Summiting the Pyramid 5.0 (September 2026) added a Detection Coverage Calculator^[15]; Sigma has an stp tag namespace for robustness scores^[34].
• Correlation as code (uneven support). Sigma correlation rules express counts, distinct values and temporal order^[62]; test conversion per backend.

Situation (illustrative). A 1,500-person company runs Microsoft Sentinel; two engineers maintain about 150 KQL analytics in the portal. The CISO wants detection-as-code and has heard Sigma makes rules portable; a SIEM migration is “possible within three years”.

Option A: everything in Sigma, KQL generated. Attractive: portability, community rules, validators, a hedge against migration. Costs: every rule must survive conversion; KQL features the team relies on (watchlist joins, functions, enrichment) may not map; analysts debug queries they did not write; and the toolchain is in flux (pinning caution above).

Option B: KQL as the source of truth, as code. Deploy the analytics rules through Sentinel’s repository connection^[40], with the same tests, review, staging and metrics. Microsoft now recommends custom detections for new rules^[44]: a dated assumption (October 2026) to revisit, not a reason to rewrite 150 working analytics. Cost: lock-in; a migration means rewriting.

Deciding assumptions. Migration probability and timing; the share of rules using platform-specific features; whether you will share or consume Sigma content; capacity to own a conversion toolchain.

Reasoning. With one platform, two engineers and an uncertain migration, the expensive part of any migration is re-validating behaviour on new telemetry, not syntax. Requirements, tests and strategy documents survive it; generated queries help little if field semantics change. So Option B now, with cheap portability: keep requirements, strategies and test cases platform-neutral, and author in Sigma only simple single-event rules and community imports. Revisit if a migration is scheduled, a second platform appears, or a partner requires Sigma. “Sigma everywhere” optimises for a migration that may never come and pays daily in conversion gaps.

After this page you can explain the label, write a Sigma rule with positive, negative and known-gap tests, design a staged pipeline with scoped credentials, measure precision, coverage and rule health, and decide how much of the practice to adopt.

“DetectionOps” is a small label for a large idea: treat each detection as a tested, owned and measured product rather than a saved query. Tools matter less than that contract, above all tests for what you chose not to see. Keep it, and “what do we detect, and how do we know?” has an answer better than a heatmap.

Sources were opened on 2026-10-16 or 2026-10-17; survey figures come from summary pages, not full reports. Ledger: research/detectionops-ledger.md.

Changelog

• 2026-10-16: first version (labs 1–2 executed, lab 3 syntax-checked); term status overlapping synonym: the label comes from one project, and the practice has better-known names.
• 2026-10-17: revised after independent review: Lab 3 token reach and plan limits; Lab 2 evasion cases and an anchored filter (re-run); SigmaHQ starting points; Microsoft rule types; SOC Ops’ dispositions; rule-run and ingestion-delay checks; latency KPI; registry-only tasks as a known gap.
• 2026-10-17, second review pass: rule B extended to event 4702 (TaskContentNew) with fixtures and Lab 4 implementation 6; \ProgramData\ and %TMP% in both rules, path lists declared a teaching subset; Labs 1–2 re-run; workflow paths covers itself; about 110 words of restatement cut.
• 2026-10-17, post-score polish: benign-positive precision convention (Magonia); lab output ≤ 90 characters (Labs 1–2 re-run); tooling signpost; footnotes re-checked; CSF wording; restatements cut.