SecDataOps

SecDataOps is the discipline of engineering security telemetry as a product. It decides which logs to collect, gets them out of the systems that produce them, parses and normalises them to a common schema, enriches them, routes them to the storage tier that fits their use, keeps them as long as investigations and obligations require, and measures all of this. The output is evidence that is complete, parsed, timely and affordable, ready for DetectionOps, SOC Ops, ThreatOps and ResponseOps.

The name is not standardised, so this page states its reading: SecDataOps is data engineering for defenders. It is an editorial distinction of this Atlas that SecDataOps builds data for security while DataSecOps secures business data; neither label has an industry definition.

Who it is for: security data engineers; engineers and architects of the SIEM (security information and event management platform) and the SOC (security operations centre); detection engineers whose fields are missing; and leaders paying an ingestion bill they cannot explain. Practitioners: start at the reference implementation; leaders: the decision framework and judgement call.

Security teams rarely lack data; they lack usable data. The Australian Signals Directorate’s Australian Cyber Security Centre (ASD’s ACSC), writing with the US Cybersecurity and Infrastructure Security Agency (CISA) and others in 2025, saw teams “preferring to collect and centralise too much data, which makes it more difficult to develop effective rules and filters”, while incomplete data produces false negatives.^[3] Query’s 2026 white paper describes retention “driven by licensing pressure rather than investigative needs”.^[4] Three failures recur:

• Silence. A source stops sending and nobody notices until an investigation needs it. ASD: “The absence of logs from systems should be a concern to organisations.”^[3]
• Too short a memory. Mandiant’s 2026 report on its 2025 investigations puts the global median dwell time at 14 days, but 122 days for espionage and North Korean IT-worker cases, and warns that with threats such as BRICKSTORM “achieving dwell times of nearly 400 days, standard 90-day log retention policies leave organizations completely blind to the initial access vector and the full scope of the intrusion”.^[5] ASD agrees: “Default log retention periods are often insufficient.”^[2]
• The field is missing, or costs extra. The US State Department found the 2023 Exchange Online intrusion because it had bought the premium audit tier exposing the MailItemsAccessed log and had built a custom rule on it; the Cyber Safety Review Board (CSRB) adds that “Just purchasing the additional logging alone would not have been enough”, and recommended access logs “without additional charges, with a minimum default retention of six months”.^[6] In July 2023, after the intrusion, Microsoft announced email-access logs and 180-day default retention for its standard audit tier.^[7]

The cost is paid during an incident, when “what did the attacker touch?” meets a source never onboarded, a parser that dropped the field, or a retention window that ended last month.

• Not DataSecOps. DataSecOps protects business data; the two meet where a data platform’s access history becomes a log source.
• Not detection engineering or the SOC. Rules belong to DetectionOps, triage to SOC Ops; DetectionOps names a data gap, SecDataOps owns the fix.
• Not application observability or generic DataOps. ObservabilityOps shares the collectors and DataOps teaches tests and contracts; security telemetry adds its own schemas, coverage against attacker techniques, forensic retention, and an adversary who benefits from gaps.
• Not a product category. Forrester calls the tools “data pipeline management (DPM)”;^[8] the practice exists without them.

Misconceptions: that a SIEM licence includes log management (it includes storage and search, not coverage or quality), and that “ingest everything” is safer than selection.^[3]

The practice is older than the name. NIST’s 2023 draft log-management guide defines log management as “the process for generating, transmitting, storing, accessing, and disposing of log data”;^[1] M-21-31 (2021) set US federal logging maturity tiers (EL0 “Not Effective” to EL3 “Advanced”) and retention minimums;^[9] and a 2024 publication led by ASD’s ACSC with CISA, the FBI, the NSA and partners from seven other countries set a baseline for “event logging best practices”.^[2]

The compound label is recent, and its users disagree on its scope:

• In March 2023 Jonathan Rau described it as “the process of integrating data into the entire security life cycle”.^[10]
• Tenzir, a pipeline vendor, calls itself “the pioneer in SecDataOps”^[11] and defines the term as applying “DataOps principles to security”:^[12] a vendor’s argument, not an adopted standard.
• Query, a federated-search vendor, has run a “SecDataOps Cast” podcast since 2024 whose description asks: “What is SecDataOps? Who should care about it? Wait, wasn’t it a throwaway joke?”^[13] Its 2025 workshop posts^[14] and its 2026 white paper use the long form “Security Data Operations (SDO)”, which the white paper defines as “the discipline of managing security data intentionally from creation through retirement, with the explicit goal of improving security outcomes”.^[4]
• Forrester (November 2024) calls the tool category “data pipeline management (DPM)” and notes such tools “are not new” to enterprise data teams.^[8]

So the label is emerging: real and vendor-led, with definitions from whole-lifecycle data integration to pipeline engineering. This page takes the telemetry-engineering reading; the whole-lifecycle reading overlaps SecOps and SOC Ops.

The mental model is the evidence supply chain: producers → collection → pipeline (parse, normalise, enrich, redact, route) → tiers → consumers (detections, hunts, investigations, audits). Each link has an owner, a contract and a health signal; because an adversary benefits from any gap, monitoring the chain is a security control.

flowchart TB
  SRC["`Log sources: endpoints, identity,
  cloud, network, SaaS, data platforms`"]:::neutral
  SD["`SecDataOps
  onboard · parse · normalise · enrich
  route · tier · retain · measure`"]:::core
  DET["`DetectionOps
  tested detections`"]:::neutral
  SOC["`SOC Ops
  triage`"]:::neutral
  RESP["`ResponseOps
  investigation, forensics`"]:::neutral
  THR["`ThreatOps
  hunting`"]:::neutral
  OBS["`ObservabilityOps
  application telemetry`"]:::decision
  DSO["`DataSecOps
  protecting business data`"]:::decision
  DO["`DataOps
  pipeline methods`"]:::decision
  SRC --> SD
  SD --> DET --> SOC
  SD --> RESP
  SD --> THR
  DET -. "data gaps" .-> SD
  OBS -. "same collectors" .- SD
  DSO -. "access logs are a source" .-> SD
  DO -. "tests, contracts" .-> SD
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029

This is the Atlas’s proposed charter, not a published standard. The nearest official baseline is the ASD-led guidance’s “four key factors”: a logging policy, centralised access and correlation, secure storage and integrity, and a detection strategy.^[2]

Mission. Give defenders complete, parsed, timely, trustworthy and affordable evidence, and know which of those five words is false for which source.

1. Every source is a product: owner, contract, cost, consumer. No consumer, no ingest.
2. Silence is an incident, with a detection-time target and a runbook.
3. Keep the raw, publish the normalised. Query-time parsing preserves the source format; ingest-time normalisation queries faster; Microsoft documents both for Sentinel.^[20]
4. One schema, versioned, with room for the unmapped. OCSF reserves unmapped for fields with no place in the class.^[21]
5. Tier by use. ASD recommends “a centralised event logging facility such as a secured data lake” feeding select, processed logs to the SIEM.^[2]
6. Time is a field you must trust. UTC, ISO 8601 with milliseconds (ASD’s model: 2024-07-25T20:54:59.649Z),^[2] synchronised clocks, a zone designator;^[9] the event’s own time kept apart from the pipeline’s processing time.
7. Cost is a design input. Pre-process at source, forwarder or ingestion; name the decision each stream informs.^[3]
8. Measure the data, not only the bill. Coverage, parse rate, completeness, freshness and retention compliance are the service levels.

Non-goals. Writing detections, running triage, protecting business data, replacing the observability stack.

SecDataOps rarely starts as a team: one SIEM engineer with a spreadsheet in small organisations, a security data group between the SOC and platform teams in large ones. The 2025 ASD guidance names a SIEM System Owner who “should be a standing member of the organisation’s enterprise architecture group and/or change control board”.^[3]

Skills: one query language, one pipeline language, the chosen schema, and enough forensics to know which fields an investigator will ask for.

Day 0 (design): inventory sources and consumers; choose the schema; design tiers and retention; decide where pre-processing happens. Day 1 (onboard): per source, enable at the producer, collect, parse, map, test against fixtures, route, publish a canary, record in the register. Day 2 (operate): watch health, parse rate and completeness; handle producer changes; re-tier streams nobody queries; age data off. Most cost and nearly all failures live in Day 2, which SIEM projects usually leave unstaffed.

An enterprise-approved logging policy names what is collected, for how long, and when collection is reassessed.^[2] New-source requests arrive through one intake form (template). Producers’ changes pass the same change board as the pipeline’s.^[3] Reviews: monthly source health and cost per stream; quarterly coverage against ATT&CK data components; annually retention against obligations. Escalation: a silent high-criticality source pages the engineer and informs the SOC lead; a parse-rate breach opens a data incident; a retention breach goes to the compliance owner.

flowchart TB
  subgraph P["Producers"]
    P1["Endpoints: audit policy, Sysmon"]
    P2["Identity, cloud, SaaS APIs"]
    P3["Network, data platforms"]
  end
  C1["Collection: agents, forwarders, API pulls"]
  L1["`Parse and normalise
  (OCSF / ECS / ASIM / CIM / UDM)`"]
  L2["Enrich, redact, filter, route"]
  T1["`Hot tier: SIEM analytics
  days to months`"]:::core
  T2["`Warm tier: lake (Parquet)
  months to years`"]:::core
  T3["`Cold tier: archive
  obligations`"]:::neutral
  K1["Detections and alerts"]
  K2["Investigations, hunts"]
  K3["Audits, reports"]
  CP["`Control plane: register,
  contracts, parser tests,
  health monitor, cost model`"]:::decision
  CP -. "measures" .-> L2
  CP -. "measures" .-> T1
  P1 --> C1
  P2 --> C1
  P3 --> C1
  C1 --> L1 --> L2
  L2 --> T1 --> K1
  L2 --> T2 --> K2
  L2 --> T3 --> K3
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029

Trust boundaries. Producers are inside the adversary’s reach: attackers “are known to modify or delete local system event logs”, so ASD recommends aggregation, Transport Layer Security (TLS) 1.3 in transit, least-privilege access and a “separate or segmented network” for log storage.^[2] The pipeline drops fields, so its configuration is versioned and tested like detection code. Tiers differ in capability, not only cost: Azure Monitor’s Auxiliary plan has minimal ingestion cost but slower queries, no alerts and no restore (as of October 2026).^[37]

%%{init: {"flowchart": {"rankSpacing": 24}}}%%
flowchart TB
  A["`Request: consumer names
  the decision, fields
  and freshness`"]:::neutral
  B{"`Already in the
  coverage register?`"}:::decision
  K{"`Contract covers the
  fields and freshness?`"}:::decision
  L["`Close: point the consumer
  to the existing table`"]:::neutral
  C["`Write or amend the contract:
  owner, class, required fields,
  freshness, tier, retention`"]:::core
  E["`Enable at the producer;
  collect samples; write
  parser, mapping, fixtures`"]:::core
  F{"`Contract tests pass?
  parse rate, UTC,
  completeness`"}:::decision
  G["`Fix parser or renegotiate
  fields with the owner`"]:::risk
  H["`Route and tier; set retention;
  estimate cost per day`"]:::core
  I{"`Canary detection fires
  on live events?`"}:::decision
  J["`Publish in register; enable
  source-health monitor`"]:::core
  A --> B
  B -- "yes" --> K
  K -- "yes" --> L
  K -- "no" --> C
  B -- "no" --> C
  C --> E --> F
  F -- "yes" --> H --> I
  F -- "no" --> G --> E
  I -- "yes" --> J
  I -- "no" --> G
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029

Happy path. A detection engineer asks for Entra ID sign-ins with user, src_endpoint.ip, is_mfa and status within five minutes. The contract names OCSF Authentication (3002), whose required attributes include time, user and type_uid, with is_mfa, src_endpoint and status recommended.^[15] Exception path. The sample shows failed sign-ins without a source IP for one client type: fix the mapping if the field is elsewhere in the raw event, or record the exception in the contract so the rule’s blind spot is named.

Playbook: cutting a high-volume stream without losing evidence. (1) List the detections and investigation questions that use it, from rule definitions and 90 days of analyst queries. (2) Classify fields as needed-hot, needed-cold or unused. (3) Choose the lever: drop unused fields; summarise per minute for the hot tier; route the full stream to the lake; sample only if no detection depends on individual events. Cribl documents the same levers: “Always consider which data is mission-critical for security analysis in your SIEM.”^[38] (4) Replay a week of raw data; every dependent detection must still fire. (5) Record the decision in the contract. ASD places pre-processing “at source/host, at forwarder/replication, or at SIEM ingestion”; the further upstream, the cheaper and the less reversible.^[3]

Enrichment: what to add, and when. Context turns an IP address into a host, an owner and a verdict.

Ingest-time enrichment records what was true when the event happened, which an investigator needs months later; query-time enrichment stays current and cheap to change, which suits intelligence that improves later. That split is this Atlas’s reasoning, consistent with M-21-31 requiring DHCP lease information^[9] and ASD advising that a SIEM “should ingest threat intelligence information”.^[3] OCSF carries results in an enrichments array.^[41] Give each enrichment source a health check: when the CMDB export stops, events still arrive but lose their meaning.

Delivery guarantees. Pipelines lose evidence in buffers. In Vector, in-memory buffers “are not durable” and “push-based sources may not be able to retransmit their messages”; for durability the documentation recommends disk buffers.^[42] End-to-end acknowledgements make a source wait until sinks confirm delivery,^[43] but only sources that can acknowledge benefit, and Vector’s syslog source cannot.^[44] Retries and replays can deliver an event twice, so give each event a stable metadata.uid and deduplicate on it before counting, or a replayed day doubles the coverage numbers.

Regulations and directions set minimum retention and sometimes clock and reporting rules; everything else above is a recommendation or a vocabulary. Take legal advice on which regimes apply.

Representative tools per layer are in the building blocks; the forwarding-format trap is under cost. Selection risks: pipeline products (Cribl, Tenzir and others Forrester lists^[8]) hide dropped fields from consumers; ingest pricing shapes SIEM behaviour; a lake alone does not alert; and CISA’s Logging Made Easy (Wazuh, Elastic, ElastAlert) lost support on 2026-05-22.^[49] Choose on openness of format, the tests you can run in CI, and the cost of leaving.

Automate CI for parsers and contracts, the health monitor, and lifecycle enforcement.^[39] Keep humans on dropping a field or stream, shortening retention, and changing a schema version a consumer depends on. AI is a consumer of this work before it is a tool for it: Query argues that AI systems “depend on clean, well-structured, and accessible data”;^[4] evaluation is in AIOps for Security. Machine-written parsers pass the same fixture-and-canary gate as human ones.

An illustrative design for about 1,200 managed Windows endpoints, Entra ID, 42 AWS accounts, six firewalls, four DNS resolvers and a Snowflake warehouse: OCSF-normalised events, 90 days hot, a Parquet lake for 400 days, a cold archive for an 18-month obligation, and a source-health monitor. All values are placeholders.

Artefact 1: the coverage register, scored on DeTT&CT’s five dimensions,^[19] is the input to Lab 1.

Example status: JSON syntax-checked; this exact file was used in the executed run in Lab 1.

{
  "assessed": "2026-10-17",
  "desired_retention_days": 365,
  "sources": [
    {"id": "win-security", "name": "Windows Security log (4624/4625/4688/4697)",
     "ocsf_class": "authentication, process_activity", "devices_expected": 1200,
     "devices_reporting": 1104, "required_fields": 9, "fields_populated": 9,
     "median_lag_seconds": 45, "schema_mapped_fields_pct": 100, "retention_days": 365,
     "owner": "endpoint-platform"},
    {"id": "sysmon", "name": "Sysmon (1, 3, 11, 22)",
     "ocsf_class": "process_activity, network_activity, dns_activity",
     "devices_expected": 1200, "devices_reporting": 310, "required_fields": 7,
     "fields_populated": 6, "median_lag_seconds": 60, "schema_mapped_fields_pct": 85,
     "retention_days": 90, "owner": "endpoint-platform"},
    {"id": "entra-signin", "name": "Entra ID sign-in logs",
     "ocsf_class": "authentication", "devices_expected": 1, "devices_reporting": 1,
     "required_fields": 8, "fields_populated": 8, "median_lag_seconds": 150,
     "schema_mapped_fields_pct": 100, "retention_days": 400, "owner": "identity"},
    {"id": "cloudtrail-mgmt", "name": "CloudTrail management events (42 accounts)",
     "ocsf_class": "api_activity", "devices_expected": 42, "devices_reporting": 42,
     "required_fields": 6, "fields_populated": 6, "median_lag_seconds": 900,
     "schema_mapped_fields_pct": 100, "retention_days": 400, "owner": "cloud-platform"},
    {"id": "cloudtrail-s3-data",
     "name": "CloudTrail S3 data events (crown-jewel buckets)",
     "ocsf_class": "api_activity", "devices_expected": 12, "devices_reporting": 5,
     "required_fields": 6, "fields_populated": 6, "median_lag_seconds": 900,
     "schema_mapped_fields_pct": 100, "retention_days": 400, "owner": "cloud-platform"},
    {"id": "fw-allow", "name": "Perimeter firewall allowed flows",
     "ocsf_class": "network_activity", "devices_expected": 6, "devices_reporting": 6,
     "required_fields": 7, "fields_populated": 5, "median_lag_seconds": 20,
     "schema_mapped_fields_pct": 60, "retention_days": 30, "owner": "network"},
    {"id": "dns-resolver", "name": "Internal DNS resolver query logs",
     "ocsf_class": "dns_activity", "devices_expected": 4, "devices_reporting": 0,
     "required_fields": 5, "fields_populated": 0, "median_lag_seconds": 0,
     "schema_mapped_fields_pct": 0, "retention_days": 0, "owner": "network"},
    {"id": "snowflake-access", "name": "Snowflake ACCESS_HISTORY",
     "ocsf_class": "api_activity (custom mapping)", "devices_expected": 1,
     "devices_reporting": 1, "required_fields": 6, "fields_populated": 5,
     "median_lag_seconds": 10800, "schema_mapped_fields_pct": 70, "retention_days": 365,
     "owner": "data-platform", "licence_gate": "Snowflake Enterprise Edition or higher"}
  ]
}

cloudtrail-s3-data reports 5 of 12 crown-jewel buckets because data events are opt-in per resource and billed separately;^[23] snowflake-access needs Enterprise Edition or higher, lags three hours because ACCESS_HISTORY “may be up to 180 minutes” behind, and excludes failed queries,^[24] so a detection that needs failed attempts uses QUERY_HISTORY, whose latency is up to 45 minutes (both latencies as of October 2026).^[50]

Artefact 2: the telemetry contract, one file per source, versioned next to the parser.

Example status: YAML syntax-checked; a proposal of this Atlas, not a standard format.

# contracts/entra-signin.yaml
source_id: entra-signin
owner: identity-team@example.com
producer: "Microsoft Entra ID sign-in logs (interactive and non-interactive)"
collection: "vendor connector, API pull every minute"
schema:
  standard: OCSF
  class_uid: 3002            # Authentication
  versions:                  # pin per destination; re-test on upgrade
    siem: "1.9.0"            # the SIEM's OCSF mapping
    lake: "1.3.0"            # Security Lake custom sources: 1.3 or earlier (October 2026)
required_fields:             # this organisation's detection contract, beyond OCSF's own required list
  - time                     # UTC, ISO 8601, millisecond precision
  - user.name
  - src_endpoint.ip
  - status_id
  - is_mfa
  - auth_protocol_id
service_levels:
  freshness_p95_seconds: 300
  parse_success_rate_min: 0.995
  field_completeness_min: 0.98
  max_silence_seconds: 900   # page after 15 minutes without events in business hours
tiering:                     # days count from ingestion; tiers overlap, they do not add up
  hot_days: 90               # SIEM analytics tier
  lake_days: 400             # Parquet lake, queryable
  cold_days: 548             # archive for the 18-month obligation
enrichment:                  # added at ingest, so values are those true at event time
  - "src_endpoint.location: GeoIP database, version recorded"
  - "user department and manager: HR directory export, refreshed daily"
delivery: "at least once; duplicates removed on metadata.uid"
canary:
  description: "A test-account sign-in from a lab IP must appear normalised within freshness_p95_seconds"
  schedule: "hourly"
known_gaps:
  - "Legacy-protocol failures carry no src_endpoint.ip for one client type; completeness measured excluding them"
cost:
  estimated_gb_per_day: 2.5
  justification: "Password spraying, impossible travel, MFA-fatigue detections; identity investigations"
changelog:
  - "2026-10-17 first version"

Artefact 3: the pipeline. A Vector configuration that receives firewall syslog over TLS, maps each line to OCSF Network Activity (4001), routes by use, and sends anything it cannot parse to a quarantine file. In Vector, remap runs a VRL program per event; with drop_on_error, drop_on_abort and reroute_dropped, a failed or aborted event leaves through a dropped output in its original form.^[27] route sends unmatched events to <name>._unmatched.^[28] Vector 0.59.0 was current on 2026-10-06.^[51]

Example status: YAML syntax-checked by the validator; not executed (Vector is not installed on the authoring host). The VRL was written from the documentation and needs a run against your samples (Lab 2).

# vector.yaml: firewall flows over TLS syslog -> OCSF Network Activity 1.3.0 -> hot, lake, quarantine
secret:
  local:
    type: directory
    path: /etc/vector/secrets        # one file per secret, readable only by the vector user
    remove_trailing_whitespace: true # a final newline would otherwise become part of the token

sources:
  fw_syslog:
    type: syslog
    mode: tcp
    address: "10.0.5.10:6514"        # the collector's logging interface, not 0.0.0.0
    permit_origin: ["10.0.9.0/28"]   # the six firewalls only
    tls:
      enabled: true
      crt_file: /etc/vector/tls/collector.crt
      key_file: /etc/vector/tls/collector.key
      ca_file: /etc/vector/tls/firewall-ca.crt
      verify_certificate: true       # firewalls must present client certificates from this CA

transforms:
  fw_parse:
    type: remap
    inputs: ["fw_syslog"]
    drop_on_error: true
    drop_on_abort: true
    reroute_dropped: true            # failures go to fw_parse.dropped, not to /dev/null
    source: |
      # Accept exactly: "allow|deny src=<ip> dst=<ip> dport=<n> proto=<name> bytes=<n>"
      p, err = parse_regex(.message, r'^(?P<act>allow|deny) src=(?P<src>[0-9a-fA-F:.]+) dst=(?P<dst>[0-9a-fA-F:.]+) dport=(?P<dport>\d{1,5}) proto=(?P<proto>[a-z]+) bytes=(?P<bytes>\d+)$')
      if err != null { abort }       # unknown shape or action: quarantine, never "deny"
      event_time = timestamp!(.timestamp)            # from the syslog header
      uid = sha2(encode_json([.host, .timestamp, .message]))  # stable id for deduplication
      raw = .message
      . = {}
      .metadata = {
        "version": "1.3.0",          # the lake's limit for custom sources; see the contract
        "uid": uid,
        "processed_time": to_unix_timestamp(now(), unit: "milliseconds"),
        "product": { "name": "example-fw", "vendor_name": "Example" }
      }
      .time = to_unix_timestamp(event_time, unit: "milliseconds")
      .class_uid = 4001              # Network Activity
      .category_uid = 4
      .activity_id = 6               # Traffic
      .type_uid = 400106
      .severity_id = 1
      .action_id = if p.act == "allow" { 1 } else { 2 }   # 1 Allowed, 2 Denied
      .disposition_id = .action_id   # 1 Allowed, 2 Blocked
      .src_endpoint = { "ip": p.src }
      .dst_endpoint = { "ip": p.dst, "port": to_int!(p.dport) }
      .connection_info = { "protocol_name": p.proto }
      .traffic = { "bytes": to_int!(p.bytes) }
      .raw_data = raw                # keep the original line
      .timestamp = event_time        # for the sinks' date templates; not encoded

  fw_route:
    type: route
    inputs: ["fw_parse"]
    route:
      hot: '.action_id == 2 || .dst_endpoint.port == 3389 || .dst_endpoint.port == 22'
      lake: '.action_id == 1'

sinks:
  siem_hot:
    type: http
    inputs: ["fw_route.hot"]
    uri: "https://siem.example.com/ingest/ocsf"
    encoding:
      codec: json
      except_fields: ["timestamp"]
    auth:
      strategy: bearer
      token: "SECRET[local.siem_token]"
    buffer:
      type: disk
      max_size: 10737418240          # 10 GiB rides out a SIEM outage; then back-pressure
      when_full: block
  lake_staging:
    type: aws_s3
    inputs: ["fw_route.lake"]
    region: "eu-west-1"
    bucket: "example-security-lake-staging"
    key_prefix: "staging/example-fw/eventDay=%Y%m%d/"   # UTC day of the event time
    compression: gzip
    encoding:
      codec: json
      except_fields: ["timestamp"]
    buffer:
      type: disk
      max_size: 10737418240
      when_full: block
    # IAM: s3:PutObject on this prefix (plus s3:ListBucket for the healthcheck); no delete
  quarantine:
    type: file
    inputs: ["fw_parse.dropped", "fw_route._unmatched"]
    path: "/var/lib/vector/quarantine/%Y-%m-%d.log"
    encoding:
      codec: json

Four choices make this safe to copy. Strict parsing: VRL’s parse_key_value accepts a bare word as a key by default and to_int(null) returns 0,^[52] so a lenient key-value parse would turn the line garbage into a “deny” to port 0; the anchored pattern aborts on anything unexpected, including an unknown action such as drop. Event time: time comes from the syslog header, which Vector replaces with arrival time only when the header does not parse;^[44] a device timestamp in the payload, where one exists, should take precedence. The pipeline’s clock goes to metadata.processed_time, and action_id and disposition_id carry the verdict as OCSF enums.^[53] An authenticated listener: TLS with client certificates and an origin allow-list, which the firewalls must support; a plaintext port lets anyone who reaches it inject “events”. The token comes from Vector’s secrets backend, which its documentation recommends over environment variables.^[54] Durable buffers: the syslog source cannot acknowledge, so disk buffers carry outages; when full, block pushes back on the firewalls, which then buffer or drop by their own rules.^[42]

The lake path is staging: Security Lake requires Parquet, one OCSF event class per source and, for custom sources, OCSF 1.3 or earlier when checked,^[40] so this stream emits 1.3.0 and a separate, tested job converts each day to Parquet under the eventDay partition. The quarantine file is the point of the design: a parser error becomes something you can count, replay and fix.

Artefact 4: producer configuration. A minimal Sysmon configuration: process creation and DNS queries for all processes, network connections only for the ports investigations ask about, and no image loads, which Sysmon says “should be configured carefully”.^[22] Every DNS query is the volume risk: measure it on a pilot group and exclude known-benign processes first.

Example status: structure follows the Sysmon documentation’s examples; not executed.

<!-- schemaversion: use the value that "sysmon -? config" prints for your binary -->
<Sysmon schemaversion="4.82">
  <HashAlgorithms>sha256</HashAlgorithms>
  <EventFiltering>
    <ProcessCreate onmatch="exclude" />
    <DnsQuery onmatch="exclude" />
    <NetworkConnect onmatch="include">
      <DestinationPort>22</DestinationPort>
      <DestinationPort>445</DestinationPort>
      <DestinationPort>3389</DestinationPort>
    </NetworkConnect>
    <ImageLoad onmatch="include" />  <!-- an empty include logs no image loads -->
  </EventFiltering>
</Sysmon>

Artefact 5: lifecycle as policy. Security Lake stores data “indefinitely using the S3 Standard storage class” unless retention is configured per Region.^[39] AWS’s example uses ONEZONE_IA, a single-Availability-Zone class AWS recommends for data you can re-create; evidence cannot be, so this policy uses STANDARD_IA, which AWS recommends for an only copy.^[55]

Example status: bash syntax-checked; not executed (no AWS account here); placeholders throughout.

# Move lake objects to Standard-IA after 60 days; expire them after 400 days (one Region).
# Not ONEZONE_IA: a single Availability Zone is no home for evidence you cannot re-create.
aws securitylake update-data-lake \
  --configurations '[{"region": "eu-west-1",
    "lifecycleConfiguration": {"expiration": {"days": 400},
      "transitions": [{"days": 60, "storageClass": "STANDARD_IA"}]}}]' \
  --meta-store-manager-role-arn "arn:aws:iam::123456789012:role/AmazonSecurityLakeMetaStoreManagerV2"

• Objective: turn a source inventory into DeTT&CT-style quality scores^[19] and a fix-first list, then check a sample of normalised events for parse rate, field completeness and silence.
• Prerequisites: Python 3.8 or later, standard library only; 45 minutes. Cost and risk: none (illustrative data).

Steps

1. Save the register above as coverage.json and the script below as coverage_score.py; run python3 coverage_score.py coverage.json.

Example status: executed on 2026-10-17 with the system Python 3; output reproduced below.

"""Score a log-source coverage register with DeTT&CT-style data quality dimensions (0-5).

Usage: python3 coverage_score.py coverage.json
Dimensions follow the DeTT&CT wiki (device completeness, data field completeness, timeliness,
consistency, retention); the bands below are this Atlas's reading of those 0-5 scales, not DeTT&CT's code.
Standard library only.
"""
import json
import sys


def band(pct):
    """Map a percentage to the DeTT&CT 1-5 completeness bands (0 = unknown / not applicable)."""
    if pct <= 0:
        return 0
    if pct < 26:
        return 1
    if pct < 51:
        return 2
    if pct < 76:
        return 3
    if pct < 100:
        return 4
    return 5


def timeliness(lag_seconds, reporting):
    """5 = available right away, 3 = acceptable, 1 = long delay (DeTT&CT uses 1/3/5)."""
    if not reporting:
        return 0
    if lag_seconds <= 120:
        return 5
    if lag_seconds <= 3600:
        return 3
    return 1


def consistency(mapped_pct, reporting):
    if not reporting:
        return 0
    if mapped_pct >= 100:
        return 5
    if mapped_pct > 50:
        return 3
    return 1


def score(register):
    desired = register["desired_retention_days"]
    rows = []
    for s in register["sources"]:
        reporting = s["devices_reporting"] > 0
        dev = band(100.0 * s["devices_reporting"] / s["devices_expected"]) if s["devices_expected"] else 0
        fld = band(100.0 * s["fields_populated"] / s["required_fields"]) if reporting else 0
        tim = timeliness(s["median_lag_seconds"], reporting)
        con = consistency(s["schema_mapped_fields_pct"], reporting)
        ret = band(100.0 * min(s["retention_days"], desired) / desired) if reporting else 0
        total = dev + fld + tim + con + ret
        rows.append((s["id"], dev, fld, tim, con, ret, total, s["owner"]))
    return rows


def main(path):
    with open(path, encoding="utf-8") as f:
        register = json.load(f)
    rows = score(register)
    print(f"{'source':20} {'dev':>3} {'fld':>3} {'tim':>3} {'con':>3} {'ret':>3} {'sum':>4}  owner")
    for r in rows:
        print(f"{r[0]:20} {r[1]:>3} {r[2]:>3} {r[3]:>3} {r[4]:>3} {r[5]:>3} {r[6]:>4}/25  {r[7]}")
    silent = [r[0] for r in rows if r[1] == 0]
    weakest = sorted(rows, key=lambda r: r[6])[:3]
    print("\nsilent sources:", ", ".join(silent) or "none")
    print("fix first:", ", ".join(f"{r[0]} ({r[6]}/25)" for r in weakest))


if __name__ == "__main__":
    main(sys.argv[1] if len(sys.argv) > 1 else "coverage.json")

Actual output:

source               dev fld tim con ret  sum  owner
win-security           4   5   5   5   5   24/25  endpoint-platform
sysmon                 1   4   5   3   1   14/25  endpoint-platform
entra-signin           5   5   3   5   5   23/25  identity
cloudtrail-mgmt        5   5   3   5   5   23/25  cloud-platform
cloudtrail-s3-data     2   5   3   5   5   20/25  cloud-platform
fw-allow               5   3   5   3   1   17/25  network
dns-resolver           0   0   0   0   0    0/25  network
snowflake-access       5   4   1   3   5   18/25  data-platform

silent sources: dns-resolver
fix first: dns-resolver (0/25), sysmon (14/25), fw-allow (17/25)

2. Save the script below as silent_source.py. Run python3 silent_source.py --demo to write the twelve sample records to events.jsonl (six Windows authentication events, one of them a parser failure and one without a source IP; five firewall flows, two of them parser failures and one without a destination IP; one DNS query at 07:58), then run python3 silent_source.py events.jsonl. Three of the records look like this:

{"source": "win-security", "time": "2026-10-16T09:00:12Z", "class_uid": 3002, "user": ...}
{"source": "win-security", "time": "2026-10-16T09:02:55Z", ..., "parse_status": "error"}
{"source": "dns-resolver", "time": "2026-10-16T07:58:00Z", "class_uid": 4003, ...}

Example status: executed on 2026-10-17 (--demo, then the check); output reproduced below.

"""Detection-readiness check over a sample of normalised events.

Usage: python3 silent_source.py --demo    (writes the twelve sample records to events.jsonl)
       python3 silent_source.py events.jsonl [--now 2026-10-16T09:05:00Z]
For each source: event count, parse success rate, required-field completeness for the class, and
the silence gap against the source's expected cadence. A record that reached the sample from the
quarantine carries "parse_status": "error", which is this page's convention, not an OCSF attribute
(OCSF's `unmapped` is for source fields that have no place in the class).
Standard library only; illustrative data.
"""
import json
import sys
from datetime import datetime, timezone

# Required fields per OCSF class for THIS organisation's detection contract (not OCSF's own list).
CONTRACT = {
    3002: ["time", "user.name", "src_endpoint.ip", "status_id"],                # Authentication
    4001: ["time", "src_endpoint.ip", "dst_endpoint.ip", "dst_endpoint.port"],  # Network Activity
    4003: ["time", "query.hostname", "src_endpoint.ip"],                        # DNS Activity
}
# Expected maximum silence per source in seconds (from the coverage register).
MAX_SILENCE = {"win-security": 300, "fw-allow": 60, "dns-resolver": 120}


def demo_records():
    """Twelve illustrative records: two Windows and three firewall records are deliberately bad."""
    def auth(t, user, ip, status=1):
        rec = {"class_uid": 3002, "user": {"name": user}, "status_id": status}
        if ip:
            rec["src_endpoint"] = {"ip": ip}
        return ("win-security", t, rec)

    def flow(t, src, dst, port):
        rec = {"class_uid": 4001, "src_endpoint": {"ip": src}, "dst_endpoint": {"port": port}}
        if dst:  # a record without dst_endpoint.ip is an intended completeness failure
            rec["dst_endpoint"]["ip"] = dst
        return ("fw-allow", t, rec)

    def failed(source, t, class_uid):
        rec = {"class_uid": class_uid, "raw_data": "... unparsed", "parse_status": "error"}
        return (source, t, rec)

    rows = [
        auth("09:00:12", "a.khan", "10.1.4.22"),
        auth("09:00:40", "b.lee", "10.1.4.31", status=2),
        auth("09:01:05", "c.ortiz", "10.1.4.40"),
        auth("09:01:30", "d.wu", None),  # legacy-protocol failure without a source IP
        auth("09:02:10", "a.khan", "10.1.4.22"),
        failed("win-security", "09:02:55", 3002),
        flow("08:59:30", "10.1.9.7", "142.250.74.78", 443),
        failed("fw-allow", "08:59:45", 4001),
        flow("09:00:02", "10.1.9.7", None, 53),
        flow("09:00:03", "10.1.9.8", "151.101.1.69", 443),
        failed("fw-allow", "09:00:04", 4001),
        ("dns-resolver", "07:58:00", {"class_uid": 4003,
                                      "query": {"hostname": "updates.example.com"},
                                      "src_endpoint": {"ip": "10.1.4.22"}}),
    ]
    for source, t, rec in rows:
        yield {"source": source, "time": f"2026-10-16T{t}Z", **rec}


def get(ev, dotted):
    cur = ev
    for part in dotted.split("."):
        if not isinstance(cur, dict) or part not in cur:
            return None
        cur = cur[part]
    return cur


def parse_time(s):
    return datetime.strptime(s, "%Y-%m-%dT%H:%M:%SZ").replace(tzinfo=timezone.utc)


def main(path, now):
    per = {}
    with open(path, encoding="utf-8") as f:
        for line in f:
            ev = json.loads(line)
            s = per.setdefault(ev["source"], {"n": 0, "parsed": 0, "complete": 0, "last": None})
            s["n"] += 1
            if ev.get("parse_status") != "error":
                s["parsed"] += 1
                req = CONTRACT.get(ev.get("class_uid"), ["time"])
                if all(get(ev, r) is not None for r in req):
                    s["complete"] += 1
            t = parse_time(ev["time"])
            s["last"] = t if s["last"] is None or t > s["last"] else s["last"]
    print(f"{'source':14} {'events':>6} {'parse%':>7} {'complete%':>9} {'silence_s':>9}  state")
    for name, s in per.items():
        parse_rate = 100.0 * s["parsed"] / s["n"]
        complete = 100.0 * s["complete"] / s["parsed"] if s["parsed"] else 0.0
        silence = int((now - s["last"]).total_seconds())
        state = "SILENT" if silence > MAX_SILENCE.get(name, 3600) else "ok"
        print(f"{name:14} {s['n']:>6} {parse_rate:>7.1f} {complete:>9.1f} {silence:>9}  {state}")


if __name__ == "__main__":
    args = sys.argv[1:]
    if "--demo" in args:
        with open("events.jsonl", "w", encoding="utf-8") as out:
            for rec in demo_records():
                out.write(json.dumps(rec) + "\n")
        print("wrote 12 records to events.jsonl")
        sys.exit(0)
    now = datetime(2026, 10, 16, 9, 5, 0, tzinfo=timezone.utc)
    if "--now" in args:
        now = parse_time(args[args.index("--now") + 1])
    files = [a for a in args if a.endswith(".jsonl")]
    main(files[0] if files else "events.jsonl", now)

Actual output:

source         events  parse% complete% silence_s  state
win-security        6    83.3      80.0       125  ok
fw-allow            5    60.0      66.7       296  SILENT
dns-resolver        1   100.0     100.0      4020  SILENT

• Expected output: dns-resolver at 0/25; the DNS source silent for over an hour; the firewall’s parse rate at 60%.
• Validation: set Sysmon’s devices_reporting to 1,200 and re-run step 1: its device score becomes 5. Delete the two firewall failure records from events.jsonl and re-run step 2: the firewall’s parse rate becomes 100.0.
• Troubleshooting: a KeyError means a register row lacks a field; a ValueError on time means a timestamp is not YYYY-MM-DDTHH:MM:SSZ, which is itself the lesson.
• Teardown: delete the four files.

• Objective: run the reference Vector configuration on synthetic firewall lines, confirm hot/lake routing and event time, and prove that malformed lines and unknown actions land in quarantine.
• Prerequisites: a Linux VM where you may install Vector 0.59.0 or later;^[51] bash; 90 minutes. Cost and risk: none with file sinks; never point the http sink at a production SIEM.

Steps

1. Copy vector.yaml to lab.yaml. For the lab only: remove the secret and tls blocks; set address to 127.0.0.1:6514 and permit_origin to ["127.0.0.1/32"]; replace the three sinks with file sinks writing /tmp/lab/hot.log, /tmp/lab/lake.log and /tmp/lab/quarantine.log. Run vector validate lab.yaml. Expected: validation succeeds.
2. Start vector --config lab.yaml in one terminal. Expected: the syslog source starts listening on 127.0.0.1:6514.
3. From a second terminal, send four RFC 5424 lines with a fixed header time:

for msg in "allow src=10.1.4.22 dst=142.250.74.78 dport=443 proto=tcp bytes=8812" \
           "deny src=203.0.113.9 dst=10.1.4.22 dport=3389 proto=tcp bytes=0" \
           "garbage" \
           "drop src=203.0.113.9 dst=10.1.4.22 dport=22 proto=tcp bytes=0"; do
  printf '<134>1 2026-10-16T09:00:00.000Z fw1 fw - - - %s\n' "$msg" > /dev/tcp/127.0.0.1/6514
done

Expected: no errors in Vector’s log. 4. Run wc -l /tmp/lab/*.log. Expected: hot.log 1 (the deny), lake.log 1 (the allow), quarantine.log 2 (garbage and the drop line). 5. Run head -1 /tmp/lab/lake.log. Expected: class_uid 4001, action_id 1, raw_data holding the original line, and time equal to 2026-10-16T09:00:00Z in epoch milliseconds (1792141200000), not the time you ran the lab.

• Validation: the line counts are 1, 1 and 2, and time is earlier than metadata.processed_time.
• Troubleshooting: a VRL compile error names the line; nothing arriving means permit_origin does not include 127.0.0.1/32; an empty quarantine file means reroute_dropped is not true.
• Teardown: stop Vector; remove /tmp/lab and lab.yaml.
• Status: not executed by the author (Vector not installed on the authoring host); YAML and bash syntax-checked.

• Objective: convert one day of events to Parquet in Security Lake’s layout, register a custom source, query coverage.
• Prerequisites: an AWS sandbox with Security Lake enabled and the permissions the custom-source guide lists;^[40] a Parquet library; Athena; half a day. Cost and risk: Security Lake, S3, Glue and Athena charges.

Steps

1. Write one day of OCSF 1.3.0 events as zstd-compressed Parquet to ext/<source>/region=<region>/accountId=external/eventDay=<YYYYMMDD>/, one event class per source, ordered by time.^[40] Expected: objects whose eventDay matches each record’s UTC event time.
2. Check them with AWS’s OCSF validation tool. Expected: no schema errors.
3. Register the custom source and wait for the Glue crawler. Expected: a new table in the Security Lake Glue database.
4. Run the coverage query below in Athena. Expected: one row per day under account external, with the record count you wrote.

Example status: SQL not executed (no AWS account here); illustrative names.

-- Events per account and day, last 7 days. Rows under accountId=external share one
-- account, so a gap shows as a missing day; external_<id> names show per-account gaps.
SELECT accountid, eventday, COUNT(*) AS events
FROM amazon_security_lake_glue_db_eu_west_1.amazon_security_lake_table_eu_west_1_ext_example_fw
WHERE eventday >= date_format(current_date - interval '7' day, '%Y%m%d')
GROUP BY accountid, eventday
ORDER BY accountid, eventday;

• Validation: COUNT(*) equals the records converted.
• Troubleshooting: no table → check the prefix and the source role’s read access; validation fails → attributes newer than the OCSF version Security Lake accepts for custom sources.^[40]
• Teardown: delete the custom source, the S3 objects and the Glue table.
• Status: not executed by the author; documented from the AWS user guide.

Three illustrative scenarios built on sourced facts.

1. The mailbox logs you did not pay for. Context: a hosted mail platform; sign-ins collected, mailbox item access not. Method: check the provider’s current defaults first: Microsoft changed them in 2023,^[7] and its mailbox-auditing page now marks MailItemsAccessed as audited by default, with new default actions “subject to the user having the appropriate license”.^[56] Record the stream as collected, licence-gated or not emitted, with the date checked; price any licence against the detection it enables, remembering that State also needed a custom rule;^[6] onboard with a canary that reads a test mailbox daily. Pitfall: treating “we have sign-in logs” as “we have mailbox logs”.

2. Snowflake access history as a security source. Context: UNC5537 stole data from Snowflake customers with infostealer credentials, no MFA and no allow lists, leaving a recognisable SQL sequence (CREATE TEMPORARY STAGE, COPY INTO, GET).^[57] Method: onboard ACCESS_HISTORY and QUERY_HISTORY with a custom OCSF mapping and the contract terms noted under the reference implementation; the detection runs on QUERY_HISTORY, whose latency of up to 45 minutes^[50] suits exfiltration detection, not session blocking. DataSecOps owns the preventive controls. Success: replaying the sequence in a test warehouse alerts in time.

3. Cutting the firewall bill without blinding the hunt. Context: allowed-flow logs are the largest, least-queried stream. Method: summarise allowed flows per minute for the hot tier, route full flows to the lake for 400 days, replay a week; ASD supports hot and cold tiering^[2] and pre-processing at the forwarder.^[3] Trade-off: hunts move to slower lake queries. Pitfall: dropping instead of routing; see the judgement call.

Gigabytes are a cost, not a service level; the service levels are the five words of the mission.

Worked calculation (illustrative). The firewall allowed-flow stream is 400 GB/day. Illustrative unit costs: hot tier 1.00 per GB ingested with 90 days included; lake 0.025 per GB-month. Everything hot: 400 × 1.00 = 400 per day, retained 90 days. Proposed: per-minute summaries to hot (about 8 GB/day), 8 per day; raw flows in the lake for 400 days, 400 GB/day × 400 days = 160,000 GB × 0.025 ≈ 4,000 per month ≈ 133 per day. Managed lakes and low-cost tiers often add two lines (Azure Monitor’s Basic and Auxiliary plans bill queries separately^[37]): say 0.05 per GB processed at ingestion, 20 per day, and ten 2 TB hunts a month at 0.005 per GB scanned, about 3 per day. Total ≈ 164 per day against 400, with raw retention up from 90 to 400 days. The numbers are placeholders; the method is to price each stream per tier at steady state, ingestion and query charges included. Azure Monitor’s summary rules, which aggregate data as it arrives, are one native form of the per-minute summary (as of October 2026).^[37]

• Integrity. Apply ASD’s transport and storage advice from trust boundaries, including cryptographic verification;^[2] a forged event is an attack on the evidence, hence the authenticated listener in the reference pipeline. Security Lake does not support S3 Object Lock (as of October 2026),^[39] so an immutable copy needs another mechanism: signed daily checksums kept in another account (RFC 3227 suggests checksums and signatures for evidence^[58]), or a copy in a separate bucket under S3 Object Lock compliance mode, which no user can overwrite or delete while it is locked.^[59]
• Time. M-21-31 requires a GPS or NIST-authenticated time baseline, and public NTP pools “shall only be used as an option of last resort”;^[9] RFC 3227 asks collectors to “Note the difference between the system clock and UTC”.^[58] Keep event time and observed time apart, as the OpenTelemetry log model does.^[60]
• Privacy. Redact in the pipeline with allow-listed keys and blocked patterns. In the OTel Collector’s redaction processor (alpha for logs, October 2026) an empty allowed_keys list allows no attributes, and matches are masked or hashed; the README recommends HMAC over MD5, SHA-1 or SHA-3 for low-entropy values such as IP addresses.^[29] Use hmac-sha256 with a managed key: an unkeyed hash of a username is reversed by hashing guesses. Lawful basis belongs to PrivacyOps.
• Compliance and least privilege. Map each obligation (M-21-31 for US federal agencies,^[9] CERT-In’s 180 days within India,^[46] sector regimes this page does not enumerate) to a tier setting, and let ComplianceOps collect the evidence. The pipeline’s identities read every log: give them write-only access to sinks and no delete on archives, and audit their activity, as Sentinel’s data lake does by default.^[36]

Cost levers in order of leverage: what the producer emits (CloudTrail data events are opt-in and billed^[23]); the forwarding format (WEF’s default “Rendered Text” doubles or triples event size, so switch subscriptions to “Events”^[25]); where you pre-process;^[3] the tier;^[37] and retention per tier (Azure Monitor includes 31 days in the ingestion price, so shortening below that saves nothing^[61]). Capacity: Microsoft plans roughly 3,000 events per second per WEF collector and points estates above 50,000 events per second at a data lake.^[25] Sustainability is mostly storage: Parquet with zstd, which Security Lake prefers for custom sources,^[40] stores and scans less than JSON, and expiring what no obligation needs is the greenest option.

Monthly, stop a lab source and time the alert. Quarterly, pick five detections at random and prove their data path end to end. Annually, audit tier settings against the policy and replay a past incident’s timeline to confirm the data would still answer it. Risk indicators: rising quarantine counts, a register unchanged for a quarter, a contract without a canary.

Engineering root cause and prevention only: the shift response is SOC Ops’ silent-source runbook, and a rule that stops firing while its data still arrives is DetectionOps’ silent-rule runbook.

%%{init: {"flowchart": {"rankSpacing": 24}}}%%
flowchart TB
  A["`Alert: dns-resolver silent
  for 12 minutes (threshold 2)`"]:::risk
  Z{"`Other sources still
  arriving?`"}:::decision
  Y["`Platform incident: monitor,
  SIEM or collector down`"]:::risk
  B{"`Collector receiving
  anything from the hosts?`"}:::decision
  C["`Producer side: service, log
  setting, disk, agent version`"]:::core
  D{"`Pipeline accepting
  and parsing?`"}:::decision
  E["`Pipeline side: quarantine
  count, sink errors, certs`"]:::core
  K{"`Events reach the index
  the monitor reads?`"}:::decision
  L["`Sink, index or monitor
  query: fix and re-check`"]:::core
  M["`False alarm: quiet, not
  silent; tune the threshold`"]:::neutral
  F["`Restore and backfill
  from producer buffer if any`"]:::core
  G["`Declare the gap window;
  hunt it with firewall DNS, EDR`"]:::core
  I["`Post-incident: threshold,
  monitor, change-board review`"]:::neutral
  A --> Z
  Z -- "no" --> Y
  Z -- "yes" --> B
  B -- "no" --> C --> F
  B -- "yes" --> D
  D -- "no" --> E --> F
  D -- "yes" --> K
  K -- "no" --> L --> F
  K -- "yes" --> M --> I
  F --> G --> I
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029

1. Confirm the silence is real: other sources still arrive, so the monitor itself is healthy.
2. Split producer from pipeline. Here an operating-system upgrade reset query logging on two hosts and rotated a client certificate on the other two: producer changes that bypassed the change board.^[3]
3. Restore and backfill. Re-enable logging; replay local query logs with their original timestamps.
4. Declare the gap in the register and to the SOC lead; DetectionOps treats it as a known blind spot.
5. Hunt the gap with firewall DNS flows and Sysmon DNS events (ID 22^[22]) for the highest-risk hosts.
6. Prevent. Put the logging setting in the configuration baseline and the upgrade on the change calendar; review the threshold; re-run the monthly silence test.

Adopt deliberate SecDataOps when any of these holds: more than a handful of sources; a bill growing faster than the estate; detections blocked on data; an incident where evidence was missing; a retention obligation.

Do not create a function or buy a pipeline product when one SIEM with vendor connectors covers the estate and the bill is not a problem (keep the register and a health query anyway); when the SOC cannot consume better data (fix SOC Ops first); when an MDR (managed detection and response) provider runs detection, whose contract should then carry this page’s coverage, retention and health terms; or when producers do not emit the events at all, which is hardening work for CloudSecOps and EndpointOps. Hidden responsibilities: one more system to run, schema versions forever, an on-call for silent sources.

flowchart TB
  A["`Stream: who uses it,
  for which decision?`"]:::neutral
  B{"`Does a detection fire
  on individual events?`"}:::decision
  I{"`Volume too high
  for the budget?`"}:::decision
  J["`Trim unused fields, filter
  known-benign; copy raw to lake`"]:::core
  C["`Hot tier, individual events
  (SIEM analytics)`"]:::core
  D{"`Needed for investigations
  or hunts within months?`"}:::decision
  E["`Lake tier (Parquet),
  summary to hot if useful`"]:::core
  F{"`Retention obligation
  or forensic value?`"}:::decision
  G["`Cold archive,
  integrity-protected`"]:::core
  H["`Do not collect; record
  the decision in the register`"]:::risk
  A --> B
  B -- "yes" --> I
  I -- "no" --> C
  I -- "yes" --> J --> C
  B -- "no" --> D
  D -- "yes" --> E
  D -- "no" --> F
  F -- "yes" --> G
  F -- "no" --> H
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029

For the platform choice, weigh five criteria with your own weights: steady-state cost including query charges; openness of format and schema; operational burden; detection latency (a lake alone does not alert); and migration risk. The architecture table summarises the trade-offs.

This is the Atlas’s model, in the spirit of M-21-31’s maturity tiers^[9] but not a reproduction and not an official standard.

• Days 1–30 (see). Build the register from the SIEM’s source list, the asset inventory and the cloud accounts; score every source with the Lab 1 method; draft the retention policy; agree one canonical schema. Quick win: a health monitor on the ten most critical sources.
• Days 31–60 (contract). Contracts, fixtures, CI and canaries for those ten; price each stream per tier.
• Days 61–90 (tier). Re-tier the three largest streams with replay checks; set lake lifecycle rules; run the first monthly silence test; report coverage, completeness, freshness and cost per stream as the baseline.

Dependencies: a detection lead who states field needs, platform owners who will change audit settings, a budget owner.

Quick start

• Every source in a register with an owner, a criticality and a tier
• One canonical schema chosen; version pinned per source
• Health monitor with a silence threshold on every high-criticality source
• Raw events kept in the cheapest tier for every normalised stream

Onboarding preflight

• Consumer named the decision, fields and freshness target
• Producer emits the fields (audit policy, data events, licence tier checked)
• Contract written; fixtures from real samples; parse-rate and completeness tests pass
• Timestamps in UTC with zone designator; event time and observed time both kept
• Tier, retention and cost per day recorded; redaction applied where needed
• Canary fires on live events; register updated

Routine review (monthly) and governance audit (quarterly)

• Silence test on a lab source; detection time recorded against the runbook
• Quarantine counts and cost per stream compared with last month
• Tier retention matches policy; log-store access reviewed; archive deletes audited
• Coverage scored against ATT&CK data components; gaps prioritised with the detection lead
• Chain-of-custody steps for exporting evidence documented^[58]

Source onboarding request (intake form)

Source name:              (system, vendor, version)
Requested by / consumer:  (team; the decision this data informs)
Detections or questions:  (rule names, investigation questions, obligations)
Required fields:          (schema field names; which are mandatory for the use)
Freshness needed:         (p95 seconds; business hours only?)
Expected volume:          (events/s, GB/day; growth)
Producer owner:           (team; change process for audit settings)
Sensitive content:        (personal data, secrets; redaction needs)
Retention need:           (days; obligation or investigative; jurisdiction)
Proposed tier:            (hot / lake / cold; summary to hot?)
Cost estimate per day:    (by tier; who pays)
Canary:                   (how we will prove it works on live events)

Retention policy, one page. Periods count from ingestion and run at the same time: a 90-day hot copy mirrored into a 400-day lake keeps data for 400 days, not 490.

ADR stub for a tiering change

ADR-00N: Route <stream> raw events to the lake; keep per-minute summaries hot
Status: proposed | accepted | superseded
Context: volume, cost per day today, detections and hunts that use the stream (list)
Decision: tiers, retention per tier, pre-processing rule, schema version
Consequences: query latency for hunts; summary job to maintain; expected cost
Validation: replay of <N> days through the new pipeline; detections that must still fire
Rollback: restore the previous route; raw data still in the lake, so no evidence is lost
Owner / date / review date

Source-health runbook card

Source: <id>        Owner: <team>        Criticality: <0-3>
Expected cadence: <events per minute; business hours / always>
Silence threshold: <seconds>   Page: <who>   Inform: SOC lead
First checks: collector connection; producer service; certificate expiry; quarantine count
Backfill: <where the producer buffers; how long; replay command>
Alternative sources for the gap: <list>

Is SecDataOps a real discipline or a marketing term? Both. A few vendors and writers use the name, with differing definitions (origins). The work it names is documented by NIST, OMB, ASD and CISA as log management and event logging.^[1][9][2] Use the practice; be careful with the label.

Which schema should we pick? The one your main consumers query in: ASIM on Sentinel, CIM on Splunk, UDM on Google SecOps; OCSF for an open lake; ECS/OTel if the estate is observability-first. One canonical schema, pinned per source.

How long should we keep logs? Long enough for your investigations and obligations. Start from the obligations in the methods table and ASD’s starting point there, then test the result against the dwell times under the problem.

Our SIEM bill is the problem; is a pipeline product the answer? Often part of it: Forrester notes the immediate value is reducing log sizes and ingest costs.^[8] Do the register and contracts first; buy if engineering time is the constraint.

How do we know a source is silent before an analyst does? A health monitor comparing last-event time with a per-source cadence threshold, tested monthly by stopping a lab source; ASD explicitly asks for such processes.^[3]

• Schema convergence. ECS is merging into OTel semantic conventions^[32] while OCSF, a Linux Foundation project,^[47] releases several times a year;^[30] mapping between them will be a recurring task.
• Detection strategies as data requirements. ATT&CK v18’s detection strategies, analytics and data components give DetectionOps and SecDataOps one vocabulary for “what data would this detection need”.^[17]
• Default audit logging from cloud providers. The CSRB recommended free access logs with six months’ retention,^[6] and Microsoft extended email-access logging to its standard tier in 2023.^[7] Providers that have not moved still gate sources by licence, so registers need a licence-gated state that is re-checked.
• AI on security data. Vendors position clean data as the prerequisite;^[4] evaluating the AI belongs to AIOps for Security.

• Decide: one canonical schema; retention by log class; the first three streams to re-tier; the register’s owner.
• Measure: coverage, parse rate, completeness, freshness p95, silent-source detection time, retention compliance, cost per stream.
• Next actions: run Lab 1 on your real estate this week; stop a lab source and time the alert.

Assumptions. Ingest-priced SIEM; a lake can be enabled within the quarter; the analysts’ “sometimes” cannot be quantified because the SIEM’s query audit log was never retained (a finding in itself). For a US federal agency the choice is already made: M-21-31 lists firewall permit events among required logs, so Option A is not available.^[9]

Decision process. Utility first: no rule fires on individual allowed flows, so the hot tier does not need them. But an exfiltration path is by definition an allowed path (reasoning, not a measured statistic), so the decision framework sends the stream to the lake. Cost second: the worked calculation prices the lake path at well under half the hot path while retaining more. Reversibility third: Option A destroys evidence that cannot be recreated; Option B is undone by promoting from the lake.

Why the attractive choice is wrong. Option A delivers the exact number asked for, this quarter, with no new system. Its cost stays invisible until an investigation needs flows that were never stored; ASD’s framing, that retention must accommodate the organisation’s mean time to detect threats,^[3] applies.

Recommendation. Option B, if a one-week replay shows no detection regresses and a hunt over the lake answers an exfiltration question within an agreed time. If the lake cannot be enabled this quarter, summarise now and route raw flows to cold object storage rather than dropping them. Uncertainty: real usage is unknown, so turn on SIEM query auditing first.

After this page you should be able to explain why the label is emerging and which guidance the practice rests on; build and score a coverage register; design onboarding with contracts, fixtures, canaries and a health monitor; decide a stream’s tier and retention; and diagnose a silent source or a parse-rate drop and prevent its return.

Before anyone can detect, hunt, investigate or prove, someone must have made sure the evidence exists, is parsed, arrives on time, can be trusted and can be paid for. Governments have written that duty down for years as log management; vendors have lately called it SecDataOps. The name may not last. The work will, because every other security function inherits its failures.

The craft reduces to habits that are easy to state and hard to keep: treat each source as a product with a contract; parse strictly and keep the raw; tier by use, with retention set by how long intrusions actually last; notice silence before an adversary benefits from it; and measure the data alongside the bill. Do those, and the cost of the SIEM becomes a set of decisions rather than a surprise.

All sources were opened on 2026-10-17; the evidence ledger (research/secdataops-ledger.md) maps claims to sources, records the archive copies used and lists what could not be fetched.

Further reading: Query’s Phase 1 discovery guide (Neal Bridges, 2025-07-01), an interview-led assessment method.

Changelog.

• 2026-10-17: first edition. Term status confirmed as emerging after opening the vendor and author sources; Lab 1 scripts executed locally; Vector, Sysmon, AWS CLI and Athena examples syntax-checked or documented but not executed.
• 2026-10-17 (r1 revision): after an independent review, the reference pipeline parses strictly, keeps event time, listens on authenticated TLS and emits OCSF 1.3.0 for the lake; M-21-31 retention scope, the CSRB case and Microsoft’s 2023 logging change corrected; enrichment and delivery guarantees added; acronyms defined; Lab 1 fixture made reproducible; three diagrams fixed; restatements cut to keep the body within the length band.
• 2026-10-17, post-score polish (reviews/secdataops-polish.md): AU-11 and M-21-31 quotations made verbatim; zstd wording, Entra lag, Lab 3 query and lifecycle storage class corrected; integrity options, “as of” dates and acronyms added; FAQ repeats trimmed; figures 2 and 3 uncrossed, 3 and 4 shortened; Lab 1 re-executed with unchanged output.
• 2026-10-17, polish check (reviews/secdataops-polish-check.md): DeTT&CT credited to its authors and sponsor.