--- title: "SecDataOps" slug: secdataops category: security term_status: emerging evidence_status: verified summary: "SecDataOps is the engineering and governance of security telemetry: onboarding log sources, parsing and normalising them to a schema, enriching, routing and tiering them, and retaining them so that detections, investigations and audits have complete, trustworthy and affordable evidence." audience: [Security data engineers, SIEM and detection engineers, SOC architects, Security leaders, Beginners] last_verified: 2026-10-17 key_takeaways: - "The label is young and vendor-led; the practice is old and government-documented as log management, event logging and SIEM pre-processing." - "Treat every log source as a product with a contract: owner, schema, required fields, freshness, retention tier and a canary detection." - "Silence is the worst failure: build the monitor that notices a source stopped before an incident does." - "Tier by use, not by habit: hot for detections, a lake for investigations and hunting, cold for obligations; 90 days is rarely enough for intrusions." - "Normalise to one shared schema, such as the open-source OCSF, but keep the raw event; measure parse rate and field completeness, not just gigabytes." - "Cost is a design input: pre-process at source, forwarder or ingestion, and justify each stream by the decisions it informs." - "Parse strictly and keep the event's own time: an unrecognised line belongs in quarantine, not in the SIEM as a guessed 'deny'." related: [datasecops, detectionops, soc-ops, dataops, observabilityops, responseops, threatops, aiops-for-security, complianceops, cloudsecops] --- ## What SecDataOps is ### In 30 seconds **SecDataOps is the discipline of engineering security telemetry as a product.** It decides which logs to collect, gets them out of the systems that produce them, parses and normalises them to a common schema, enriches them, routes them to the storage tier that fits their use, keeps them as long as investigations and obligations require, and measures all of this. The output is evidence that is complete, parsed, timely and affordable, ready for [DetectionOps](../detectionops/), [SOC Ops](../soc-ops/), [ThreatOps](../threatops/) and [ResponseOps](../responseops/). The name is not standardised, so this page states its reading: SecDataOps is **data engineering for defenders**. It is an **editorial distinction** of this Atlas that SecDataOps builds *data for security* while [DataSecOps](../datasecops/) *secures business data*; neither label has an industry definition. Who it is for: security data engineers; engineers and architects of the SIEM (security information and event management platform) and the SOC (security operations centre); detection engineers whose fields are missing; and leaders paying an ingestion bill they cannot explain. Practitioners: start at the [reference implementation](#reference-implementation); leaders: the [decision framework](#decision-framework) and [judgement call](#a-judgement-call-drop-the-firewall-allow-logs-or-tier-them). ### At a glance | Item | Summary | |---|---| | Goal | Complete, parsed, timely, trustworthy and affordable security telemetry for every decision defenders make | | Scope (this page) | Source coverage, collection, parsing and normalisation to a shared [schema](#glossary-and-mental-model), enrichment, redaction, routing, delivery guarantees, SIEM and lake tiers, retention, data-quality contracts and metrics, cost of ingest | | Prerequisites | What a SIEM does ([SOC Ops](../soc-ops/)); basic JSON and one query language | | Key outputs | Coverage register, a telemetry contract per source, versioned parsers, a tiering and retention policy, a source-health monitor, a cost model per stream | | Term status | **Emerging.** In public use since at least March 2023 by a few vendors and writers with differing definitions; no standards body or certification defines it. Governments document the practice as log management and event logging.[^nist-800-92r1][^asd-logging] | ### The problem SecDataOps solves Security teams rarely lack data; they lack *usable* data. The Australian Signals Directorate's Australian Cyber Security Centre (ASD's ACSC), writing with the US Cybersecurity and Infrastructure Security Agency (CISA) and others in 2025, saw teams "preferring to collect and centralise too much data, which makes it more difficult to develop effective rules and filters", while incomplete data produces false negatives.[^asd-siem] Query's 2026 white paper describes retention "driven by licensing pressure rather than investigative needs".[^query-wp] Three failures recur: - **Silence.** A source stops sending and nobody notices until an investigation needs it. ASD: "The absence of logs from systems should be a concern to organisations."[^asd-siem] - **Too short a memory.** Mandiant's 2026 report on its 2025 investigations puts the global median dwell time at 14 days, but 122 days for espionage and North Korean IT-worker cases, and warns that with threats such as BRICKSTORM "achieving dwell times of nearly 400 days, standard 90-day log retention policies leave organizations completely blind to the initial access vector and the full scope of the intrusion".[^mtrends-2026] ASD agrees: "Default log retention periods are often insufficient."[^asd-logging] - **The field is missing, or costs extra.** The US State Department found the 2023 Exchange Online intrusion because it had bought the premium audit tier exposing the `MailItemsAccessed` log and had built a custom rule on it; the Cyber Safety Review Board (CSRB) adds that "Just purchasing the additional logging alone would not have been enough", and recommended access logs "without additional charges, with a minimum default retention of six months".[^csrb] In July 2023, after the intrusion, Microsoft announced email-access logs and 180-day default retention for its standard audit tier.[^ms-logging-2023] The cost is paid during an incident, when "what did the attacker touch?" meets a source never onboarded, a parser that dropped the field, or a retention window that ended last month. ### What SecDataOps is not - **Not DataSecOps.** [DataSecOps](../datasecops/) protects business data; the two meet where a data platform's access history becomes a log source. - **Not detection engineering or the SOC.** Rules belong to [DetectionOps](../detectionops/), triage to [SOC Ops](../soc-ops/); DetectionOps names a data gap, SecDataOps owns the fix. - **Not application observability or generic DataOps.** [ObservabilityOps](../observabilityops/) shares the collectors and [DataOps](../dataops/) teaches tests and contracts; security telemetry adds its own schemas, coverage against attacker techniques, forensic retention, and an adversary who benefits from gaps. - **Not a product category.** Forrester calls the tools "data pipeline management (DPM)";[^forrester-dpm] the practice exists without them. Misconceptions: that a SIEM licence includes log management (it includes storage and search, not coverage or quality), and that "ingest everything" is safer than selection.[^asd-siem] ## Where the label comes from and where it sits ### Origins and evolution of the label The practice is older than the name. NIST's 2023 draft log-management guide defines log management as "the process for generating, transmitting, storing, accessing, and disposing of log data";[^nist-800-92r1] M-21-31 (2021) set US federal logging maturity tiers (EL0 "Not Effective" to EL3 "Advanced") and retention minimums;[^omb-m2131] and a 2024 publication led by ASD's ACSC with CISA, the FBI, the NSA and partners from seven other countries set a baseline for "event logging best practices".[^asd-logging] The compound label is recent, and its users disagree on its scope: - In March 2023 Jonathan Rau described it as "the process of integrating data into the entire security life cycle".[^rau-2023] - Tenzir, a pipeline vendor, calls itself "the pioneer in SecDataOps"[^tenzir-ga] and defines the term as applying "DataOps principles to security":[^tenzir-blog] a vendor's argument, not an adopted standard. - Query, a federated-search vendor, has run a "SecDataOps Cast" podcast since 2024 whose description asks: "What is SecDataOps? Who should care about it? Wait, wasn't it a throwaway joke?"[^query-cast] Its 2025 workshop posts[^query-phase2] and its 2026 white paper use the long form "Security Data Operations (SDO)", which the white paper defines as "the discipline of managing security data intentionally from creation through retirement, with the explicit goal of improving security outcomes".[^query-wp] - Forrester (November 2024) calls the tool category "data pipeline management (DPM)" and notes such tools "are not new" to enterprise data teams.[^forrester-dpm] So the label is **emerging**: real and vendor-led, with definitions from whole-lifecycle data integration to pipeline engineering. This page takes the telemetry-engineering reading; the whole-lifecycle reading overlaps [SecOps](../secops/) and [SOC Ops](../soc-ops/). ### Glossary and mental model | Term | Meaning | Why it matters | |---|---|---| | SIEM | The platform that stores, searches and alerts on security events | The usual hot tier and largest bill | | Coverage | Share of expected producers actually reporting | A rule that sees 300 of 1,200 laptops is a quarter of a rule | | Schema | Shared field names and types: OCSF (Open Cybersecurity Schema Framework, Linux Foundation); ECS (Elastic Common Schema), merging into OpenTelemetry (OTel) conventions; vendor models ASIM (Microsoft Sentinel), CIM (Splunk) and UDM (Google Security Operations) | Pick what your consumers query in | | Parse / normalise | Turning a raw event into typed schema fields, e.g. OCSF `Authentication` (class 3002)[^ocsf-auth] | Rules work across vendors only on normalised fields | | Pipeline | Software that parses, enriches, redacts and routes between collection and storage, e.g. Vector, programmed in VRL (Vector Remap Language) | Where cost is controlled and fields are lost | | Tiers | Hot: SIEM storage that alerts; warm: a lake of Parquet (open columnar) files queried in minutes; cold: an archive restored for audits | Where most cost and much risk lives | | Telemetry contract | One source's agreement: owner, schema, required fields, freshness, tier, canary | Makes "the data is broken" measurable | | Data component (MITRE ATT&CK) | What a source must show to detect a technique in ATT&CK, a public knowledge base of adversary tactics and techniques: 120 across domains, 106 in Enterprise; "data sources" deprecated in v18 (October 2025)[^attack-dc][^attack-v18] | Coverage against adversary behaviour | | DeTT&CT scoring | Detect Tactics, Techniques & Combat Threats, an open-source framework by Marcus Bakker and Ruben Bouman, sponsored by Rabobank:[^dettect] 0–5 scoring on device completeness, field completeness, timeliness, consistency and retention[^dettect-dq] | An open rubric ([Lab 1](#hands-on-labs)) | The mental model is **the evidence supply chain**: producers → collection → pipeline (parse, normalise, enrich, redact, route) → tiers → consumers (detections, hunts, investigations, audits). Each link has an owner, a contract and a health signal; because an adversary benefits from any gap, monitoring the chain is a security control. ### How SecDataOps relates to other Ops ::: figure Where SecDataOps sits among its neighbours ```mermaid flowchart TB SRC["`Log sources: endpoints, identity, cloud, network, SaaS, data platforms`"]:::neutral SD["`SecDataOps onboard · parse · normalise · enrich route · tier · retain · measure`"]:::core DET["`DetectionOps tested detections`"]:::neutral SOC["`SOC Ops triage`"]:::neutral RESP["`ResponseOps investigation, forensics`"]:::neutral THR["`ThreatOps hunting`"]:::neutral OBS["`ObservabilityOps application telemetry`"]:::decision DSO["`DataSecOps protecting business data`"]:::decision DO["`DataOps pipeline methods`"]:::decision SRC --> SD SD --> DET --> SOC SD --> RESP SD --> THR DET -. "data gaps" .-> SD OBS -. "same collectors" .- SD DSO -. "access logs are a source" .-> SD DO -. "tests, contracts" .-> SD classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029 classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029 ``` Text equivalent: Log sources feed SecDataOps, which supplies DetectionOps (whose detections feed SOC Ops), ResponseOps and ThreatOps; DetectionOps returns data-gap requests. ObservabilityOps shares collectors, DataSecOps hands over access logs, and DataOps contributes tests and contracts. ::: | Neighbouring Ops | Shared scope | Unique focus there | |---|---|---| | [DataSecOps](../datasecops/) | Data-platform audit logs | The data is the protected object: policies, masking, access audit | | [DetectionOps](../detectionops/) | Field requirements; Sigma `logsource`; fixtures | Detection as code, tests, tuning | | [SOC Ops](../soc-ops/) | SIEM in daily use; source-health tickets | Tiers, triage, escalation, handover | | [ResponseOps](../responseops/) and [ThreatOps](../threatops/) | Retention and clock sync; enrichment and hunt data needs | Containment and evidence; intelligence and hunting | | [DataOps](../dataops/) and [ObservabilityOps](../observabilityops/) | Pipelines, tests, contracts; collectors and redaction | Business data pipelines; reliability telemetry | | [ComplianceOps](../complianceops/) and [CloudSecOps](../cloudsecops/) | Retention obligations; cloud audit logs as sources | Control evidence; cloud posture | ## Principles and operating model ### Charter and principles This is the Atlas's proposed charter, not a published standard. The nearest official baseline is the ASD-led guidance's "four key factors": a logging policy, centralised access and correlation, secure storage and integrity, and a detection strategy.[^asd-logging] **Mission.** Give defenders complete, parsed, timely, trustworthy and affordable evidence, and know which of those five words is false for which source. 1. **Every source is a product**: owner, contract, cost, consumer. No consumer, no ingest. 2. **Silence is an incident**, with a detection-time target and a runbook. 3. **Keep the raw, publish the normalised.** Query-time parsing preserves the source format; ingest-time normalisation queries faster; Microsoft documents both for Sentinel.[^asim] 4. **One schema, versioned, with room for the unmapped.** OCSF reserves `unmapped` for fields with no place in the class.[^ocsf-understanding] 5. **Tier by use.** ASD recommends "a centralised event logging facility such as a secured data lake" feeding select, processed logs to the SIEM.[^asd-logging] 6. **Time is a field you must trust.** UTC, ISO 8601 with milliseconds (ASD's model: `2024-07-25T20:54:59.649Z`),[^asd-logging] synchronised clocks, a zone designator;[^omb-m2131] the event's own time kept apart from the pipeline's processing time. 7. **Cost is a design input.** Pre-process at source, forwarder or ingestion; name the decision each stream informs.[^asd-siem] 8. **Measure the data, not only the bill.** Coverage, parse rate, completeness, freshness and retention compliance are the service levels. **Non-goals.** Writing detections, running triage, protecting business data, replacing the observability stack. ### Fundamentals: the building blocks | Layer | Representative verified options | Note | |---|---|---| | Producers | Windows audit policy and Sysmon;[^sysmon] CloudTrail management and data events;[^cloudtrail-data] Snowflake `ACCESS_HISTORY`[^snowflake-ah] | The field exists only if the producer emits it | | Collection | Windows Event Forwarding;[^wef] Fluent Bit;[^fluentbit] OpenTelemetry Collector; vendor connectors | WEF's buffer is the local event log, overwritten silently[^wef] | | Pipeline | Vector `remap` (VRL) and `route`;[^vector-remap][^vector-route] Fluent Bit filters; OTel processors incl. redaction;[^otel-redaction] DPM products[^forrester-dpm] | Where cost is controlled and fields are lost | | Schema | OCSF 1.9.0;[^ocsf-releases] ECS 9.5.0 converging with OTel;[^ecs][^ecs-otel] ASIM;[^asim] Splunk CIM;[^splunk-cim] UDM[^udm] | Pick what your consumers query in | | Tiers | SIEM tables; Amazon Security Lake (Parquet, OCSF);[^seclake] Sentinel data lake (Parquet, 12 years);[^sentinel-lake] Azure Monitor Analytics/Basic/Auxiliary plans[^azmon-plans] | Retention and query capability differ by tier | | Control plane | Register, contracts, parser tests, health monitor, cost model | The part most teams lack | ### Roles, ownership and RACI SecDataOps rarely starts as a team: one SIEM engineer with a spreadsheet in small organisations, a security data group between the SOC and platform teams in large ones. The 2025 ASD guidance names a **SIEM System Owner** who "should be a standing member of the organisation's enterprise architecture group and/or change control board".[^asd-siem] | Activity | Security data engineer | Source owner | Detection engineer | SOC lead | CISO | |---|---|---|---|---|---| | Decide what to onboard | R | C | C (states fields) | C | A | | Enable audit policy at the producer | C | R/A | I | I | I | | Write and test parser and mapping | R/A | C (samples) | C (fixtures) | I | I | | Set tier and retention per stream | R | C (volume) | C | C | A | | Monitor source health | R/A | I | I | C | I | | Approve the ingest budget | C | I | I | C | A | Skills: one query language, one pipeline language, the chosen schema, and enough forensics to know which fields an investigator will ask for. ### Lifecycle: Day 0, Day 1, Day 2 **Day 0 (design):** inventory sources and consumers; choose the schema; design tiers and retention; decide where pre-processing happens. **Day 1 (onboard):** per source, enable at the producer, collect, parse, map, test against fixtures, route, publish a canary, record in the register. **Day 2 (operate):** watch health, parse rate and completeness; handle producer changes; re-tier streams nobody queries; age data off. Most cost and nearly all failures live in Day 2, which SIEM projects usually leave unstaffed. ### Operating model and governance An enterprise-approved logging policy names what is collected, for how long, and when collection is reassessed.[^asd-logging] New-source requests arrive through one intake form ([template](#templates)). Producers' changes pass the same change board as the pipeline's.[^asd-siem] Reviews: monthly source health and cost per stream; quarterly coverage against ATT&CK data components; annually retention against obligations. Escalation: a silent high-criticality source pages the engineer and informs the SOC lead; a parse-rate breach opens a data incident; a retention breach goes to the compliance owner. ## Architecture and workflow ### Reference architecture ::: figure Reference architecture: the evidence supply chain with a control plane ```mermaid flowchart TB subgraph P["Producers"] P1["Endpoints: audit policy, Sysmon"] P2["Identity, cloud, SaaS APIs"] P3["Network, data platforms"] end C1["Collection: agents, forwarders, API pulls"] L1["`Parse and normalise (OCSF / ECS / ASIM / CIM / UDM)`"] L2["Enrich, redact, filter, route"] T1["`Hot tier: SIEM analytics days to months`"]:::core T2["`Warm tier: lake (Parquet) months to years`"]:::core T3["`Cold tier: archive obligations`"]:::neutral K1["Detections and alerts"] K2["Investigations, hunts"] K3["Audits, reports"] CP["`Control plane: register, contracts, parser tests, health monitor, cost model`"]:::decision CP -. "measures" .-> L2 CP -. "measures" .-> T1 P1 --> C1 P2 --> C1 P3 --> C1 C1 --> L1 --> L2 L2 --> T1 --> K1 L2 --> T2 --> K2 L2 --> T3 --> K3 classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029 classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029 ``` Text equivalent: Producers (endpoints; identity, cloud and SaaS APIs; network and data platforms) feed collection, then a pipeline that parses, normalises, enriches, redacts, filters and routes to a hot SIEM tier for detections, a warm Parquet lake for investigations and hunts, and a cold archive for audits. A control plane (register, contracts, parser tests, health monitor, cost model) measures the pipeline and the hot tier. ::: **Trust boundaries.** Producers are inside the adversary's reach: attackers "are known to modify or delete local system event logs", so ASD recommends aggregation, Transport Layer Security (TLS) 1.3 in transit, least-privilege access and a "separate or segmented network" for log storage.[^asd-logging] The pipeline drops fields, so its configuration is versioned and tested like detection code. Tiers differ in capability, not only cost: Azure Monitor's Auxiliary plan has minimal ingestion cost but slower queries, no alerts and no restore (as of October 2026).[^azmon-plans] | Pattern | Strengths | Limits | Fits when | |---|---|---|---| | SIEM-only, everything hot | One tool, alerts on everything | Ingest-priced; short retention; over-collection trap[^asd-siem] | Small estate | | SIEM with native tiers | Sentinel mirrors analytics data to a Parquet lake tier with up to 12 years' retention (as of October 2026)[^sentinel-lake] | Tier limits must be learnt;[^azmon-plans] one vendor | Already on that SIEM | | Pipeline tool in front | Routing, reduction, redaction outside the SIEM licence[^forrester-dpm][^cribl-cost] | One more system; dropped fields invisible downstream | High volume, several destinations | | Security data lake with OCSF | Open format, cheap storage, many subscribers, lifecycle per Region[^seclake][^seclake-lifecycle] | Custom sources need OCSF 1.3 or earlier, in Parquet, strictly partitioned;[^seclake-custom] no detection engine alone | Multi-account cloud, long retention | | Federated query | No bulk data movement; the SIEM keeps real-time alerting[^query-phase2] | More engineering effort;[^query-phase2] speed and cost set by each store, and some tiers bill per query[^azmon-plans] | Many existing stores | ### End-to-end workflow: onboarding a log source ::: figure Onboarding a source, with the exception path ```mermaid %%{init: {"flowchart": {"rankSpacing": 24}}}%% flowchart TB A["`Request: consumer names the decision, fields and freshness`"]:::neutral B{"`Already in the coverage register?`"}:::decision K{"`Contract covers the fields and freshness?`"}:::decision L["`Close: point the consumer to the existing table`"]:::neutral C["`Write or amend the contract: owner, class, required fields, freshness, tier, retention`"]:::core E["`Enable at the producer; collect samples; write parser, mapping, fixtures`"]:::core F{"`Contract tests pass? parse rate, UTC, completeness`"}:::decision G["`Fix parser or renegotiate fields with the owner`"]:::risk H["`Route and tier; set retention; estimate cost per day`"]:::core I{"`Canary detection fires on live events?`"}:::decision J["`Publish in register; enable source-health monitor`"]:::core A --> B B -- "yes" --> K K -- "yes" --> L K -- "no" --> C B -- "no" --> C C --> E --> F F -- "yes" --> H --> I F -- "no" --> G --> E I -- "yes" --> J I -- "no" --> G classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029 classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029 classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029 ``` Text equivalent: A request names the decision, fields and freshness. If the source is registered and its contract covers them, the consumer is pointed to the existing table. Otherwise the engineer writes or amends the contract, enables the producer, writes parser, mapping and fixtures, and runs contract tests (parse rate, completeness, UTC); failures loop back to fix the parser or renegotiate fields. Passing data is routed, tiered and costed, and a canary must fire on live events before the source is published with its health monitor. ::: **Happy path.** A detection engineer asks for Entra ID sign-ins with `user`, `src_endpoint.ip`, `is_mfa` and `status` within five minutes. The contract names OCSF `Authentication` (3002), whose required attributes include `time`, `user` and `type_uid`, with `is_mfa`, `src_endpoint` and `status` recommended.[^ocsf-auth] **Exception path.** The sample shows failed sign-ins without a source IP for one client type: fix the mapping if the field is elsewhere in the raw event, or record the exception in the contract so the rule's blind spot is named. ## Practices, frameworks and tools ### Core practices and playbooks | Practice | Goal | Output | Verification | |---|---|---|---| | Coverage register | Know every source, owner, state, score | Register with DeTT&CT-style scores[^dettect-dq] | Review against ATT&CK data components[^attack-dc] | | Telemetry contract | Make quality measurable per source | Contract file in Git | Contract tests in CI on fixtures | | Parser as code | Reproducible normalisation | Versioned config, fixtures | CI: parse rate, completeness | | Source-health monitor | Detect silence and degradation | Alert with runbook | Monthly: stop a lab source, time the alert | | Pre-processing policy | Cut cost without cutting evidence | Per-stream rule: keep, trim, summarise, route, drop | Replay check that detections still fire | | Redaction at pipeline | Keep secrets and personal data out of hot tiers | Allow-listed keys, blocked patterns[^otel-redaction] | Planted-secret fixture never reaches the sink | **Playbook: cutting a high-volume stream without losing evidence.** (1) List the detections and investigation questions that use it, from rule definitions and 90 days of analyst queries. (2) Classify fields as needed-hot, needed-cold or unused. (3) Choose the lever: drop unused fields; summarise per minute for the hot tier; route the full stream to the lake; sample only if no detection depends on individual events. Cribl documents the same levers: "Always consider which data is mission-critical for security analysis in your SIEM."[^cribl-cost] (4) Replay a week of raw data; every dependent detection must still fire. (5) Record the decision in the contract. ASD places pre-processing "at source/host, at forwarder/replication, or at SIEM ingestion"; the further upstream, the cheaper and the less reversible.[^asd-siem] **Enrichment: what to add, and when.** Context turns an IP address into a host, an owner and a verdict. | Context | Typical source | Add at | If the source fails | |---|---|---|---| | Host and owner for an IP | DHCP leases, CMDB (configuration database), EDR (endpoint detection) inventory | Ingest: leases move within hours | An IP nobody can name; mark it unenriched, never guess | | User department and privilege | Directory or HR export | Ingest; query time for "who is it now" | Stale fields read as "unprivileged" | | Geolocation and network owner (ASN) | GeoIP database | Ingest, with the database version | Locations shift when the database updates | | Threat-intelligence match | Intelligence feeds | Query time, or re-run over old data | Ingest-only matching misses indicators published later | Ingest-time enrichment records what was true when the event happened, which an investigator needs months later; query-time enrichment stays current and cheap to change, which suits intelligence that improves later. That split is this Atlas's reasoning, consistent with M-21-31 requiring DHCP lease information[^omb-m2131] and ASD advising that a SIEM "should ingest threat intelligence information".[^asd-siem] OCSF carries results in an `enrichments` array.[^ocsf-enrichment] Give each enrichment source a health check: when the CMDB export stops, events still arrive but lose their meaning. **Delivery guarantees.** Pipelines lose evidence in buffers. In Vector, in-memory buffers "are not durable" and "push-based sources may not be able to retransmit their messages"; for durability the documentation recommends disk buffers.[^vector-buffers] End-to-end acknowledgements make a source wait until sinks confirm delivery,[^vector-e2e] but only sources that can acknowledge benefit, and Vector's syslog source cannot.[^vector-syslog] Retries and replays can deliver an event twice, so give each event a stable `metadata.uid` and deduplicate on it before counting, or a replayed day doubles the coverage numbers. ### Methods, standards and frameworks | Item | Type | What it gives SecDataOps | |---|---|---| | NIST SP 800-53 Rev. 5 AU-11 | Control catalogue | "Retain audit records for [Assignment: organization-defined time period]"[^nist-au11] | | OMB M-21-31 (2021) | Binding on US federal agencies (not national security systems) | Tiers EL0–EL3; criticality 0–3 sets which logs each tier must cover; minimum 12 months active plus 18 months cold for most log categories at every criticality; 72 hours for full packet capture; firewall permit and deny events; clock rules; minimum fields per event[^omb-m2131] | | ASD-led event logging (2024) and SIEM/SOAR (2025) guidance | Multinational guidance | Four key factors; JSON with consistent schemas; UTC/ISO 8601; tiering; integrity;[^asd-logging] for organisations new to detection, a retention starting point of one year for administrative and security logs and 90 days for informational logs, tailored to risk; pre-processing points[^asd-siem] | | CERT-In directions (India, 2022) | Binding directions | 180-day rolling retention within India; clocks synced to NTP (Network Time Protocol) servers of the National Informatics Centre or National Physical Laboratory; 6-hour reporting[^certin] | | OCSF (Linux Foundation, Apache 2.0) | Open schema | Categories, classes, objects, profiles, extensions; `type_uid = class_uid × 100 + activity_id`; SemVer;[^ocsf-site][^ocsf-understanding] 1.9.0 released 2026-08-03[^ocsf-releases] | | ATT&CK data components (v18+) | Open knowledge base | 106 Enterprise components (120 across all domains) tied to detection strategies and analytics[^attack-dc][^attack-v18] | | Sigma `logsource` | Open rule format | `category`/`product`/`service`; "any mismatches in logsource can render the detection ineffective"[^sigma-logsource] | Regulations and directions set minimum retention and sometimes clock and reporting rules; everything else above is a recommendation or a vocabulary. Take legal advice on which regimes apply. ### Tooling landscape Representative tools per layer are in the [building blocks](#fundamentals-the-building-blocks); the forwarding-format trap is under [cost](#cost-capacity-and-sustainability). Selection risks: pipeline products (Cribl, Tenzir and others Forrester lists[^forrester-dpm]) hide dropped fields from consumers; ingest pricing shapes SIEM behaviour; a lake alone does not alert; and CISA's Logging Made Easy (Wazuh, Elastic, ElastAlert) lost support on 2026-05-22.[^lme] Choose on openness of format, the tests you can run in CI, and the cost of leaving. ### Automation and AI Automate CI for parsers and contracts, the health monitor, and lifecycle enforcement.[^seclake-lifecycle] Keep humans on dropping a field or stream, shortening retention, and changing a schema version a consumer depends on. AI is a consumer of this work before it is a tool for it: Query argues that AI systems "depend on clean, well-structured, and accessible data";[^query-wp] evaluation is in [AIOps for Security](../aiops-for-security/). Machine-written parsers pass the same fixture-and-canary gate as human ones. ## SecDataOps in practice ### Reference implementation An illustrative design for about 1,200 managed Windows endpoints, Entra ID, 42 AWS accounts, six firewalls, four DNS resolvers and a Snowflake warehouse: OCSF-normalised events, 90 days hot, a Parquet lake for 400 days, a cold archive for an 18-month obligation, and a source-health monitor. All values are placeholders. **Artefact 1: the coverage register**, scored on DeTT&CT's five dimensions,[^dettect-dq] is the input to Lab 1. _Example status: JSON syntax-checked; this exact file was used in the executed run in Lab 1._ ```json { "assessed": "2026-10-17", "desired_retention_days": 365, "sources": [ {"id": "win-security", "name": "Windows Security log (4624/4625/4688/4697)", "ocsf_class": "authentication, process_activity", "devices_expected": 1200, "devices_reporting": 1104, "required_fields": 9, "fields_populated": 9, "median_lag_seconds": 45, "schema_mapped_fields_pct": 100, "retention_days": 365, "owner": "endpoint-platform"}, {"id": "sysmon", "name": "Sysmon (1, 3, 11, 22)", "ocsf_class": "process_activity, network_activity, dns_activity", "devices_expected": 1200, "devices_reporting": 310, "required_fields": 7, "fields_populated": 6, "median_lag_seconds": 60, "schema_mapped_fields_pct": 85, "retention_days": 90, "owner": "endpoint-platform"}, {"id": "entra-signin", "name": "Entra ID sign-in logs", "ocsf_class": "authentication", "devices_expected": 1, "devices_reporting": 1, "required_fields": 8, "fields_populated": 8, "median_lag_seconds": 150, "schema_mapped_fields_pct": 100, "retention_days": 400, "owner": "identity"}, {"id": "cloudtrail-mgmt", "name": "CloudTrail management events (42 accounts)", "ocsf_class": "api_activity", "devices_expected": 42, "devices_reporting": 42, "required_fields": 6, "fields_populated": 6, "median_lag_seconds": 900, "schema_mapped_fields_pct": 100, "retention_days": 400, "owner": "cloud-platform"}, {"id": "cloudtrail-s3-data", "name": "CloudTrail S3 data events (crown-jewel buckets)", "ocsf_class": "api_activity", "devices_expected": 12, "devices_reporting": 5, "required_fields": 6, "fields_populated": 6, "median_lag_seconds": 900, "schema_mapped_fields_pct": 100, "retention_days": 400, "owner": "cloud-platform"}, {"id": "fw-allow", "name": "Perimeter firewall allowed flows", "ocsf_class": "network_activity", "devices_expected": 6, "devices_reporting": 6, "required_fields": 7, "fields_populated": 5, "median_lag_seconds": 20, "schema_mapped_fields_pct": 60, "retention_days": 30, "owner": "network"}, {"id": "dns-resolver", "name": "Internal DNS resolver query logs", "ocsf_class": "dns_activity", "devices_expected": 4, "devices_reporting": 0, "required_fields": 5, "fields_populated": 0, "median_lag_seconds": 0, "schema_mapped_fields_pct": 0, "retention_days": 0, "owner": "network"}, {"id": "snowflake-access", "name": "Snowflake ACCESS_HISTORY", "ocsf_class": "api_activity (custom mapping)", "devices_expected": 1, "devices_reporting": 1, "required_fields": 6, "fields_populated": 5, "median_lag_seconds": 10800, "schema_mapped_fields_pct": 70, "retention_days": 365, "owner": "data-platform", "licence_gate": "Snowflake Enterprise Edition or higher"} ] } ``` `cloudtrail-s3-data` reports 5 of 12 crown-jewel buckets because data events are opt-in per resource and billed separately;[^cloudtrail-data] `snowflake-access` needs Enterprise Edition or higher, lags three hours because `ACCESS_HISTORY` "may be up to 180 minutes" behind, and excludes failed queries,[^snowflake-ah] so a detection that needs failed attempts uses `QUERY_HISTORY`, whose latency is up to 45 minutes (both latencies as of October 2026).[^snowflake-qh] **Artefact 2: the telemetry contract**, one file per source, versioned next to the parser. _Example status: YAML syntax-checked; a proposal of this Atlas, not a standard format._ ```yaml # contracts/entra-signin.yaml source_id: entra-signin owner: identity-team@example.com producer: "Microsoft Entra ID sign-in logs (interactive and non-interactive)" collection: "vendor connector, API pull every minute" schema: standard: OCSF class_uid: 3002 # Authentication versions: # pin per destination; re-test on upgrade siem: "1.9.0" # the SIEM's OCSF mapping lake: "1.3.0" # Security Lake custom sources: 1.3 or earlier (October 2026) required_fields: # this organisation's detection contract, beyond OCSF's own required list - time # UTC, ISO 8601, millisecond precision - user.name - src_endpoint.ip - status_id - is_mfa - auth_protocol_id service_levels: freshness_p95_seconds: 300 parse_success_rate_min: 0.995 field_completeness_min: 0.98 max_silence_seconds: 900 # page after 15 minutes without events in business hours tiering: # days count from ingestion; tiers overlap, they do not add up hot_days: 90 # SIEM analytics tier lake_days: 400 # Parquet lake, queryable cold_days: 548 # archive for the 18-month obligation enrichment: # added at ingest, so values are those true at event time - "src_endpoint.location: GeoIP database, version recorded" - "user department and manager: HR directory export, refreshed daily" delivery: "at least once; duplicates removed on metadata.uid" canary: description: "A test-account sign-in from a lab IP must appear normalised within freshness_p95_seconds" schedule: "hourly" known_gaps: - "Legacy-protocol failures carry no src_endpoint.ip for one client type; completeness measured excluding them" cost: estimated_gb_per_day: 2.5 justification: "Password spraying, impossible travel, MFA-fatigue detections; identity investigations" changelog: - "2026-10-17 first version" ``` **Artefact 3: the pipeline.** A Vector configuration that receives firewall syslog over TLS, maps each line to OCSF `Network Activity` (4001), routes by use, and sends anything it cannot parse to a quarantine file. In Vector, `remap` runs a VRL program per event; with `drop_on_error`, `drop_on_abort` and `reroute_dropped`, a failed or aborted event leaves through a `dropped` output in its original form.[^vector-remap] `route` sends unmatched events to `._unmatched`.[^vector-route] Vector 0.59.0 was current on 2026-10-06.[^vector-releases] _Example status: YAML syntax-checked by the validator; **not executed** (Vector is not installed on the authoring host). The VRL was written from the documentation and needs a run against your samples ([Lab 2](#hands-on-labs))._ ```yaml # vector.yaml: firewall flows over TLS syslog -> OCSF Network Activity 1.3.0 -> hot, lake, quarantine secret: local: type: directory path: /etc/vector/secrets # one file per secret, readable only by the vector user remove_trailing_whitespace: true # a final newline would otherwise become part of the token sources: fw_syslog: type: syslog mode: tcp address: "10.0.5.10:6514" # the collector's logging interface, not 0.0.0.0 permit_origin: ["10.0.9.0/28"] # the six firewalls only tls: enabled: true crt_file: /etc/vector/tls/collector.crt key_file: /etc/vector/tls/collector.key ca_file: /etc/vector/tls/firewall-ca.crt verify_certificate: true # firewalls must present client certificates from this CA transforms: fw_parse: type: remap inputs: ["fw_syslog"] drop_on_error: true drop_on_abort: true reroute_dropped: true # failures go to fw_parse.dropped, not to /dev/null source: | # Accept exactly: "allow|deny src= dst= dport= proto= bytes=" p, err = parse_regex(.message, r'^(?Pallow|deny) src=(?P[0-9a-fA-F:.]+) dst=(?P[0-9a-fA-F:.]+) dport=(?P\d{1,5}) proto=(?P[a-z]+) bytes=(?P\d+)$') if err != null { abort } # unknown shape or action: quarantine, never "deny" event_time = timestamp!(.timestamp) # from the syslog header uid = sha2(encode_json([.host, .timestamp, .message])) # stable id for deduplication raw = .message . = {} .metadata = { "version": "1.3.0", # the lake's limit for custom sources; see the contract "uid": uid, "processed_time": to_unix_timestamp(now(), unit: "milliseconds"), "product": { "name": "example-fw", "vendor_name": "Example" } } .time = to_unix_timestamp(event_time, unit: "milliseconds") .class_uid = 4001 # Network Activity .category_uid = 4 .activity_id = 6 # Traffic .type_uid = 400106 .severity_id = 1 .action_id = if p.act == "allow" { 1 } else { 2 } # 1 Allowed, 2 Denied .disposition_id = .action_id # 1 Allowed, 2 Blocked .src_endpoint = { "ip": p.src } .dst_endpoint = { "ip": p.dst, "port": to_int!(p.dport) } .connection_info = { "protocol_name": p.proto } .traffic = { "bytes": to_int!(p.bytes) } .raw_data = raw # keep the original line .timestamp = event_time # for the sinks' date templates; not encoded fw_route: type: route inputs: ["fw_parse"] route: hot: '.action_id == 2 || .dst_endpoint.port == 3389 || .dst_endpoint.port == 22' lake: '.action_id == 1' sinks: siem_hot: type: http inputs: ["fw_route.hot"] uri: "https://siem.example.com/ingest/ocsf" encoding: codec: json except_fields: ["timestamp"] auth: strategy: bearer token: "SECRET[local.siem_token]" buffer: type: disk max_size: 10737418240 # 10 GiB rides out a SIEM outage; then back-pressure when_full: block lake_staging: type: aws_s3 inputs: ["fw_route.lake"] region: "eu-west-1" bucket: "example-security-lake-staging" key_prefix: "staging/example-fw/eventDay=%Y%m%d/" # UTC day of the event time compression: gzip encoding: codec: json except_fields: ["timestamp"] buffer: type: disk max_size: 10737418240 when_full: block # IAM: s3:PutObject on this prefix (plus s3:ListBucket for the healthcheck); no delete quarantine: type: file inputs: ["fw_parse.dropped", "fw_route._unmatched"] path: "/var/lib/vector/quarantine/%Y-%m-%d.log" encoding: codec: json ``` Four choices make this safe to copy. **Strict parsing:** VRL's `parse_key_value` accepts a bare word as a key by default and `to_int(null)` returns 0,[^vrl-functions] so a lenient key-value parse would turn the line `garbage` into a "deny" to port 0; the anchored pattern aborts on anything unexpected, including an unknown action such as `drop`. **Event time:** `time` comes from the syslog header, which Vector replaces with arrival time only when the header does not parse;[^vector-syslog] a device timestamp in the payload, where one exists, should take precedence. The pipeline's clock goes to `metadata.processed_time`, and `action_id` and `disposition_id` carry the verdict as OCSF enums.[^ocsf-network] **An authenticated listener:** TLS with client certificates and an origin allow-list, which the firewalls must support; a plaintext port lets anyone who reaches it inject "events". The token comes from Vector's secrets backend, which its documentation recommends over environment variables.[^vector-secrets] **Durable buffers:** the syslog source cannot acknowledge, so disk buffers carry outages; when full, `block` pushes back on the firewalls, which then buffer or drop by their own rules.[^vector-buffers] The lake path is staging: Security Lake requires Parquet, one OCSF event class per source and, for custom sources, OCSF 1.3 or earlier when checked,[^seclake-custom] so this stream emits 1.3.0 and a separate, tested job converts each day to Parquet under the `eventDay` partition. The quarantine file is the point of the design: a parser error becomes something you can count, replay and fix. **Artefact 4: producer configuration.** A minimal Sysmon configuration: process creation and DNS queries for all processes, network connections only for the ports investigations ask about, and no image loads, which Sysmon says "should be configured carefully".[^sysmon] Every DNS query is the volume risk: measure it on a pilot group and exclude known-benign processes first. _Example status: structure follows the Sysmon documentation's examples; **not executed**._ ```xml sha256 22 445 3389 ``` **Artefact 5: lifecycle as policy.** Security Lake stores data "indefinitely using the S3 Standard storage class" unless retention is configured per Region.[^seclake-lifecycle] AWS's example uses `ONEZONE_IA`, a single-Availability-Zone class AWS recommends for data you can re-create; evidence cannot be, so this policy uses `STANDARD_IA`, which AWS recommends for an only copy.[^s3-classes] _Example status: bash syntax-checked; **not executed** (no AWS account here); placeholders throughout._ ```bash # Move lake objects to Standard-IA after 60 days; expire them after 400 days (one Region). # Not ONEZONE_IA: a single Availability Zone is no home for evidence you cannot re-create. aws securitylake update-data-lake \ --configurations '[{"region": "eu-west-1", "lifecycleConfiguration": {"expiration": {"days": 400}, "transitions": [{"days": 60, "storageClass": "STANDARD_IA"}]}}]' \ --meta-store-manager-role-arn "arn:aws:iam::123456789012:role/AmazonSecurityLakeMetaStoreManagerV2" ``` ### Hands-on labs #### Lab 1 (beginner): score a coverage register and find the silent source - **Objective:** turn a source inventory into DeTT&CT-style quality scores[^dettect-dq] and a fix-first list, then check a sample of normalised events for parse rate, field completeness and silence. - **Prerequisites:** Python 3.8 or later, standard library only; 45 minutes. **Cost and risk:** none (illustrative data). **Steps** 1. Save the register above as `coverage.json` and the script below as `coverage_score.py`; run `python3 coverage_score.py coverage.json`. _Example status: **executed** on 2026-10-17 with the system Python 3; output reproduced below._ ```python """Score a log-source coverage register with DeTT&CT-style data quality dimensions (0-5). Usage: python3 coverage_score.py coverage.json Dimensions follow the DeTT&CT wiki (device completeness, data field completeness, timeliness, consistency, retention); the bands below are this Atlas's reading of those 0-5 scales, not DeTT&CT's code. Standard library only. """ import json import sys def band(pct): """Map a percentage to the DeTT&CT 1-5 completeness bands (0 = unknown / not applicable).""" if pct <= 0: return 0 if pct < 26: return 1 if pct < 51: return 2 if pct < 76: return 3 if pct < 100: return 4 return 5 def timeliness(lag_seconds, reporting): """5 = available right away, 3 = acceptable, 1 = long delay (DeTT&CT uses 1/3/5).""" if not reporting: return 0 if lag_seconds <= 120: return 5 if lag_seconds <= 3600: return 3 return 1 def consistency(mapped_pct, reporting): if not reporting: return 0 if mapped_pct >= 100: return 5 if mapped_pct > 50: return 3 return 1 def score(register): desired = register["desired_retention_days"] rows = [] for s in register["sources"]: reporting = s["devices_reporting"] > 0 dev = band(100.0 * s["devices_reporting"] / s["devices_expected"]) if s["devices_expected"] else 0 fld = band(100.0 * s["fields_populated"] / s["required_fields"]) if reporting else 0 tim = timeliness(s["median_lag_seconds"], reporting) con = consistency(s["schema_mapped_fields_pct"], reporting) ret = band(100.0 * min(s["retention_days"], desired) / desired) if reporting else 0 total = dev + fld + tim + con + ret rows.append((s["id"], dev, fld, tim, con, ret, total, s["owner"])) return rows def main(path): with open(path, encoding="utf-8") as f: register = json.load(f) rows = score(register) print(f"{'source':20} {'dev':>3} {'fld':>3} {'tim':>3} {'con':>3} {'ret':>3} {'sum':>4} owner") for r in rows: print(f"{r[0]:20} {r[1]:>3} {r[2]:>3} {r[3]:>3} {r[4]:>3} {r[5]:>3} {r[6]:>4}/25 {r[7]}") silent = [r[0] for r in rows if r[1] == 0] weakest = sorted(rows, key=lambda r: r[6])[:3] print("\nsilent sources:", ", ".join(silent) or "none") print("fix first:", ", ".join(f"{r[0]} ({r[6]}/25)" for r in weakest)) if __name__ == "__main__": main(sys.argv[1] if len(sys.argv) > 1 else "coverage.json") ``` Actual output: ```text source dev fld tim con ret sum owner win-security 4 5 5 5 5 24/25 endpoint-platform sysmon 1 4 5 3 1 14/25 endpoint-platform entra-signin 5 5 3 5 5 23/25 identity cloudtrail-mgmt 5 5 3 5 5 23/25 cloud-platform cloudtrail-s3-data 2 5 3 5 5 20/25 cloud-platform fw-allow 5 3 5 3 1 17/25 network dns-resolver 0 0 0 0 0 0/25 network snowflake-access 5 4 1 3 5 18/25 data-platform silent sources: dns-resolver fix first: dns-resolver (0/25), sysmon (14/25), fw-allow (17/25) ``` 2. Save the script below as `silent_source.py`. Run `python3 silent_source.py --demo` to write the twelve sample records to `events.jsonl` (six Windows authentication events, one of them a parser failure and one without a source IP; five firewall flows, two of them parser failures and one without a destination IP; one DNS query at 07:58), then run `python3 silent_source.py events.jsonl`. Three of the records look like this: ```text {"source": "win-security", "time": "2026-10-16T09:00:12Z", "class_uid": 3002, "user": ...} {"source": "win-security", "time": "2026-10-16T09:02:55Z", ..., "parse_status": "error"} {"source": "dns-resolver", "time": "2026-10-16T07:58:00Z", "class_uid": 4003, ...} ``` _Example status: **executed** on 2026-10-17 (`--demo`, then the check); output reproduced below._ ```python """Detection-readiness check over a sample of normalised events. Usage: python3 silent_source.py --demo (writes the twelve sample records to events.jsonl) python3 silent_source.py events.jsonl [--now 2026-10-16T09:05:00Z] For each source: event count, parse success rate, required-field completeness for the class, and the silence gap against the source's expected cadence. A record that reached the sample from the quarantine carries "parse_status": "error", which is this page's convention, not an OCSF attribute (OCSF's `unmapped` is for source fields that have no place in the class). Standard library only; illustrative data. """ import json import sys from datetime import datetime, timezone # Required fields per OCSF class for THIS organisation's detection contract (not OCSF's own list). CONTRACT = { 3002: ["time", "user.name", "src_endpoint.ip", "status_id"], # Authentication 4001: ["time", "src_endpoint.ip", "dst_endpoint.ip", "dst_endpoint.port"], # Network Activity 4003: ["time", "query.hostname", "src_endpoint.ip"], # DNS Activity } # Expected maximum silence per source in seconds (from the coverage register). MAX_SILENCE = {"win-security": 300, "fw-allow": 60, "dns-resolver": 120} def demo_records(): """Twelve illustrative records: two Windows and three firewall records are deliberately bad.""" def auth(t, user, ip, status=1): rec = {"class_uid": 3002, "user": {"name": user}, "status_id": status} if ip: rec["src_endpoint"] = {"ip": ip} return ("win-security", t, rec) def flow(t, src, dst, port): rec = {"class_uid": 4001, "src_endpoint": {"ip": src}, "dst_endpoint": {"port": port}} if dst: # a record without dst_endpoint.ip is an intended completeness failure rec["dst_endpoint"]["ip"] = dst return ("fw-allow", t, rec) def failed(source, t, class_uid): rec = {"class_uid": class_uid, "raw_data": "... unparsed", "parse_status": "error"} return (source, t, rec) rows = [ auth("09:00:12", "a.khan", "10.1.4.22"), auth("09:00:40", "b.lee", "10.1.4.31", status=2), auth("09:01:05", "c.ortiz", "10.1.4.40"), auth("09:01:30", "d.wu", None), # legacy-protocol failure without a source IP auth("09:02:10", "a.khan", "10.1.4.22"), failed("win-security", "09:02:55", 3002), flow("08:59:30", "10.1.9.7", "142.250.74.78", 443), failed("fw-allow", "08:59:45", 4001), flow("09:00:02", "10.1.9.7", None, 53), flow("09:00:03", "10.1.9.8", "151.101.1.69", 443), failed("fw-allow", "09:00:04", 4001), ("dns-resolver", "07:58:00", {"class_uid": 4003, "query": {"hostname": "updates.example.com"}, "src_endpoint": {"ip": "10.1.4.22"}}), ] for source, t, rec in rows: yield {"source": source, "time": f"2026-10-16T{t}Z", **rec} def get(ev, dotted): cur = ev for part in dotted.split("."): if not isinstance(cur, dict) or part not in cur: return None cur = cur[part] return cur def parse_time(s): return datetime.strptime(s, "%Y-%m-%dT%H:%M:%SZ").replace(tzinfo=timezone.utc) def main(path, now): per = {} with open(path, encoding="utf-8") as f: for line in f: ev = json.loads(line) s = per.setdefault(ev["source"], {"n": 0, "parsed": 0, "complete": 0, "last": None}) s["n"] += 1 if ev.get("parse_status") != "error": s["parsed"] += 1 req = CONTRACT.get(ev.get("class_uid"), ["time"]) if all(get(ev, r) is not None for r in req): s["complete"] += 1 t = parse_time(ev["time"]) s["last"] = t if s["last"] is None or t > s["last"] else s["last"] print(f"{'source':14} {'events':>6} {'parse%':>7} {'complete%':>9} {'silence_s':>9} state") for name, s in per.items(): parse_rate = 100.0 * s["parsed"] / s["n"] complete = 100.0 * s["complete"] / s["parsed"] if s["parsed"] else 0.0 silence = int((now - s["last"]).total_seconds()) state = "SILENT" if silence > MAX_SILENCE.get(name, 3600) else "ok" print(f"{name:14} {s['n']:>6} {parse_rate:>7.1f} {complete:>9.1f} {silence:>9} {state}") if __name__ == "__main__": args = sys.argv[1:] if "--demo" in args: with open("events.jsonl", "w", encoding="utf-8") as out: for rec in demo_records(): out.write(json.dumps(rec) + "\n") print("wrote 12 records to events.jsonl") sys.exit(0) now = datetime(2026, 10, 16, 9, 5, 0, tzinfo=timezone.utc) if "--now" in args: now = parse_time(args[args.index("--now") + 1]) files = [a for a in args if a.endswith(".jsonl")] main(files[0] if files else "events.jsonl", now) ``` Actual output: ```text source events parse% complete% silence_s state win-security 6 83.3 80.0 125 ok fw-allow 5 60.0 66.7 296 SILENT dns-resolver 1 100.0 100.0 4020 SILENT ``` - **Expected output:** `dns-resolver` at 0/25; the DNS source silent for over an hour; the firewall's parse rate at 60%. - **Validation:** set Sysmon's `devices_reporting` to 1,200 and re-run step 1: its device score becomes 5. Delete the two firewall failure records from `events.jsonl` and re-run step 2: the firewall's parse rate becomes 100.0. - **Troubleshooting:** a `KeyError` means a register row lacks a field; a `ValueError` on time means a timestamp is not `YYYY-MM-DDTHH:MM:SSZ`, which is itself the lesson. - **Teardown:** delete the four files. #### Lab 2 (intermediate): run the pipeline and prove nothing is lost - **Objective:** run the reference Vector configuration on synthetic firewall lines, confirm hot/lake routing and event time, and prove that malformed lines and unknown actions land in quarantine. - **Prerequisites:** a Linux VM where you may install Vector 0.59.0 or later;[^vector-releases] bash; 90 minutes. **Cost and risk:** none with `file` sinks; never point the `http` sink at a production SIEM. **Steps** 1. Copy `vector.yaml` to `lab.yaml`. For the lab only: remove the `secret` and `tls` blocks; set `address` to `127.0.0.1:6514` and `permit_origin` to `["127.0.0.1/32"]`; replace the three sinks with `file` sinks writing `/tmp/lab/hot.log`, `/tmp/lab/lake.log` and `/tmp/lab/quarantine.log`. Run `vector validate lab.yaml`. *Expected:* validation succeeds. 2. Start `vector --config lab.yaml` in one terminal. *Expected:* the syslog source starts listening on 127.0.0.1:6514. 3. From a second terminal, send four RFC 5424 lines with a fixed header time: ```bash for msg in "allow src=10.1.4.22 dst=142.250.74.78 dport=443 proto=tcp bytes=8812" \ "deny src=203.0.113.9 dst=10.1.4.22 dport=3389 proto=tcp bytes=0" \ "garbage" \ "drop src=203.0.113.9 dst=10.1.4.22 dport=22 proto=tcp bytes=0"; do printf '<134>1 2026-10-16T09:00:00.000Z fw1 fw - - - %s\n' "$msg" > /dev/tcp/127.0.0.1/6514 done ``` *Expected:* no errors in Vector's log. 4. Run `wc -l /tmp/lab/*.log`. *Expected:* `hot.log` 1 (the deny), `lake.log` 1 (the allow), `quarantine.log` 2 (`garbage` and the `drop` line). 5. Run `head -1 /tmp/lab/lake.log`. *Expected:* `class_uid` 4001, `action_id` 1, `raw_data` holding the original line, and `time` equal to 2026-10-16T09:00:00Z in epoch milliseconds (1792141200000), not the time you ran the lab. - **Validation:** the line counts are 1, 1 and 2, and `time` is earlier than `metadata.processed_time`. - **Troubleshooting:** a VRL compile error names the line; nothing arriving means `permit_origin` does not include 127.0.0.1/32; an empty quarantine file means `reroute_dropped` is not `true`. - **Teardown:** stop Vector; remove `/tmp/lab` and `lab.yaml`. - **Status:** **not executed** by the author (Vector not installed on the authoring host); YAML and bash syntax-checked. #### Lab 3 (advanced): register a custom source in Amazon Security Lake and query coverage - **Objective:** convert one day of events to Parquet in Security Lake's layout, register a custom source, query coverage. - **Prerequisites:** an AWS sandbox with Security Lake enabled and the permissions the custom-source guide lists;[^seclake-custom] a Parquet library; Athena; half a day. **Cost and risk:** Security Lake, S3, Glue and Athena charges. **Steps** 1. Write one day of OCSF 1.3.0 events as zstd-compressed Parquet to `ext//region=/accountId=external/eventDay=/`, one event class per source, ordered by time.[^seclake-custom] *Expected:* objects whose `eventDay` matches each record's UTC event time. 2. Check them with AWS's OCSF validation tool. *Expected:* no schema errors. 3. Register the custom source and wait for the Glue crawler. *Expected:* a new table in the Security Lake Glue database. 4. Run the coverage query below in Athena. *Expected:* one row per day under account `external`, with the record count you wrote. _Example status: SQL **not executed** (no AWS account here); illustrative names._ ```sql -- Events per account and day, last 7 days. Rows under accountId=external share one -- account, so a gap shows as a missing day; external_ names show per-account gaps. SELECT accountid, eventday, COUNT(*) AS events FROM amazon_security_lake_glue_db_eu_west_1.amazon_security_lake_table_eu_west_1_ext_example_fw WHERE eventday >= date_format(current_date - interval '7' day, '%Y%m%d') GROUP BY accountid, eventday ORDER BY accountid, eventday; ``` - **Validation:** `COUNT(*)` equals the records converted. - **Troubleshooting:** no table → check the prefix and the source role's read access; validation fails → attributes newer than the OCSF version Security Lake accepts for custom sources.[^seclake-custom] - **Teardown:** delete the custom source, the S3 objects and the Glue table. - **Status:** **not executed** by the author; documented from the AWS user guide. ### Use cases and scenarios Three illustrative scenarios built on sourced facts. **1. The mailbox logs you did not pay for.** *Context:* a hosted mail platform; sign-ins collected, mailbox item access not. *Method:* check the provider's current defaults first: Microsoft changed them in 2023,[^ms-logging-2023] and its mailbox-auditing page now marks `MailItemsAccessed` as audited by default, with new default actions "subject to the user having the appropriate license".[^ms-audit-mailboxes] Record the stream as collected, licence-gated or not emitted, with the date checked; price any licence against the detection it enables, remembering that State also needed a custom rule;[^csrb] onboard with a canary that reads a test mailbox daily. *Pitfall:* treating "we have sign-in logs" as "we have mailbox logs". **2. Snowflake access history as a security source.** *Context:* UNC5537 stole data from Snowflake customers with infostealer credentials, no MFA and no allow lists, leaving a recognisable SQL sequence (`CREATE TEMPORARY STAGE`, `COPY INTO`, `GET`).[^unc5537] *Method:* onboard `ACCESS_HISTORY` and `QUERY_HISTORY` with a custom OCSF mapping and the contract terms noted under the [reference implementation](#reference-implementation); the detection runs on `QUERY_HISTORY`, whose latency of up to 45 minutes[^snowflake-qh] suits exfiltration detection, not session blocking. [DataSecOps](../datasecops/) owns the preventive controls. *Success:* replaying the sequence in a test warehouse alerts in time. **3. Cutting the firewall bill without blinding the hunt.** *Context:* allowed-flow logs are the largest, least-queried stream. *Method:* summarise allowed flows per minute for the hot tier, route full flows to the lake for 400 days, replay a week; ASD supports hot and cold tiering[^asd-logging] and pre-processing at the forwarder.[^asd-siem] *Trade-off:* hunts move to slower lake queries. *Pitfall:* dropping instead of routing; see the [judgement call](#a-judgement-call-drop-the-firewall-allow-logs-or-tier-them). ## Measure, govern, pay ### Metrics and KPIs Gigabytes are a cost, not a service level; the service levels are the five words of the mission. | KPI | Definition / formula | Data source | Pitfall | |---|---|---|---| | Device coverage | devices reporting ÷ devices expected, per source | Register vs asset inventory | Counting agents installed, not events received | | Parse success rate | events without a parser error ÷ events received | Quarantine counts | Measuring after the drop | | Field completeness | events with all contract-required fields ÷ parsed events | Contract tests on live samples | Using OCSF's required list instead of your rule's needs | | Freshness (p95 lag) | 95th percentile of observed time − event time | Pipeline, both timestamps kept | Clock skew read as lag | | Silent-source detection time | last event → alert, per source | Health monitor | Thresholds guessed, not measured | | Cost per stream per day | GB/day × tier unit cost (+ per-query cost on tiers that charge it[^azmon-plans]) | Billing, pipeline volume | Averages hide the stream that matters | **Worked calculation (illustrative).** The firewall allowed-flow stream is 400 GB/day. Illustrative unit costs: hot tier 1.00 per GB ingested with 90 days included; lake 0.025 per GB-month. Everything hot: 400 × 1.00 = **400 per day**, retained 90 days. Proposed: per-minute summaries to hot (about 8 GB/day), 8 per day; raw flows in the lake for 400 days, 400 GB/day × 400 days = 160,000 GB × 0.025 ≈ 4,000 per month ≈ 133 per day. Managed lakes and low-cost tiers often add two lines (Azure Monitor's Basic and Auxiliary plans bill queries separately[^azmon-plans]): say 0.05 per GB processed at ingestion, 20 per day, and ten 2 TB hunts a month at 0.005 per GB scanned, about 3 per day. Total ≈ **164 per day** against 400, with raw retention up from 90 to 400 days. The numbers are placeholders; the method is to price each stream per tier at steady state, ingestion and query charges included. Azure Monitor's summary rules, which aggregate data as it arrives, are one native form of the per-minute summary (as of October 2026).[^azmon-plans] ### Quality, security, privacy and compliance - **Integrity.** Apply ASD's transport and storage advice from [trust boundaries](#reference-architecture), including cryptographic verification;[^asd-logging] a forged event is an attack on the evidence, hence the authenticated listener in the reference pipeline. Security Lake does not support S3 Object Lock (as of October 2026),[^seclake-lifecycle] so an immutable copy needs another mechanism: signed daily checksums kept in another account (RFC 3227 suggests checksums and signatures for evidence[^rfc3227]), or a copy in a separate bucket under S3 Object Lock compliance mode, which no user can overwrite or delete while it is locked.[^s3-object-lock] - **Time.** M-21-31 requires a GPS or NIST-authenticated time baseline, and public NTP pools "shall only be used as an option of last resort";[^omb-m2131] RFC 3227 asks collectors to "Note the difference between the system clock and UTC".[^rfc3227] Keep event time and observed time apart, as the OpenTelemetry log model does.[^otel-logs] - **Privacy.** Redact in the pipeline with allow-listed keys and blocked patterns. In the OTel Collector's redaction processor (alpha for logs, October 2026) an empty `allowed_keys` list allows no attributes, and matches are masked or hashed; the README recommends HMAC over MD5, SHA-1 or SHA-3 for low-entropy values such as IP addresses.[^otel-redaction] Use `hmac-sha256` with a managed key: an unkeyed hash of a username is reversed by hashing guesses. Lawful basis belongs to [PrivacyOps](../privacyops/). - **Compliance and least privilege.** Map each obligation (M-21-31 for US federal agencies,[^omb-m2131] CERT-In's 180 days within India,[^certin] sector regimes this page does not enumerate) to a tier setting, and let [ComplianceOps](../complianceops/) collect the evidence. The pipeline's identities read every log: give them write-only access to sinks and no delete on archives, and audit their activity, as Sentinel's data lake does by default.[^sentinel-lake] ### Cost, capacity and sustainability Cost levers in order of leverage: what the producer emits (CloudTrail data events are opt-in and billed[^cloudtrail-data]); the forwarding format (WEF's default "Rendered Text" doubles or triples event size, so switch subscriptions to "Events"[^wef]); where you pre-process;[^asd-siem] the tier;[^azmon-plans] and retention per tier (Azure Monitor includes 31 days in the ingestion price, so shortening below that saves nothing[^azmon-retention]). Capacity: Microsoft plans roughly 3,000 events per second per WEF collector and points estates above 50,000 events per second at a data lake.[^wef] Sustainability is mostly storage: Parquet with zstd, which Security Lake prefers for custom sources,[^seclake-custom] stores and scans less than JSON, and expiring what no obligation needs is the greenest option. ### Assurance of the operating model Monthly, stop a lab source and time the alert. Quarterly, pick five detections at random and prove their data path end to end. Annually, audit tier settings against the policy and replay a past incident's timeline to confirm the data would still answer it. Risk indicators: rising quarantine counts, a register unchanged for a quarter, a contract without a canary. ## When things go wrong ### Common failures and troubleshooting | Symptom | Likely cause | Diagnosis | Remediation | Prevention | |---|---|---|---|---| | A source goes quiet; nothing alerts | Agent update, certificate expiry, audit policy reset, token rotation | Last-event time vs cadence; collector's own log | Restore; backfill from the producer's buffer | Health monitor per source[^asd-siem] | | Parse rate drops after a vendor release | Log format changed | Diff a quarantined event against a fixture | Update mapping; re-parse from raw | Keep raw; fixtures per format version | | Rule fires on test data, not in production | Field renamed or empty; Sigma `logsource` mismatch[^sigma-logsource] | Contract test on live samples | Fix mapping or producer setting | Canary before publishing | | Investigator hits a 90-day cliff | Retention left at the tool default | Tier settings vs policy | Extend tier or add a lake; the past is gone | A retention policy by log class ([template](#templates)) | ### Worked runbook: the DNS resolver logs went silent Engineering root cause and prevention only: the shift response is SOC Ops' [silent-source runbook](../soc-ops/#worked-runbook-a-critical-log-source-goes-silent-overnight), and a rule that stops firing while its data still arrives is DetectionOps' [silent-rule runbook](../detectionops/#worked-runbook-the-rule-that-went-silent). ::: figure Runbook: silent source ```mermaid %%{init: {"flowchart": {"rankSpacing": 24}}}%% flowchart TB A["`Alert: dns-resolver silent for 12 minutes (threshold 2)`"]:::risk Z{"`Other sources still arriving?`"}:::decision Y["`Platform incident: monitor, SIEM or collector down`"]:::risk B{"`Collector receiving anything from the hosts?`"}:::decision C["`Producer side: service, log setting, disk, agent version`"]:::core D{"`Pipeline accepting and parsing?`"}:::decision E["`Pipeline side: quarantine count, sink errors, certs`"]:::core K{"`Events reach the index the monitor reads?`"}:::decision L["`Sink, index or monitor query: fix and re-check`"]:::core M["`False alarm: quiet, not silent; tune the threshold`"]:::neutral F["`Restore and backfill from producer buffer if any`"]:::core G["`Declare the gap window; hunt it with firewall DNS, EDR`"]:::core I["`Post-incident: threshold, monitor, change-board review`"]:::neutral A --> Z Z -- "no" --> Y Z -- "yes" --> B B -- "no" --> C --> F B -- "yes" --> D D -- "no" --> E --> F D -- "yes" --> K K -- "no" --> L --> F K -- "yes" --> M --> I F --> G --> I classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029 classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029 classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029 ``` Text equivalent: The alert reports the DNS resolver source silent for 12 minutes against a 2-minute threshold. If no other source arrives either, it is a platform incident. Otherwise ask in turn whether the collector receives events from the resolver hosts (if not, the producer side), whether the pipeline parses them (if not, quarantine, sink errors, certificates) and whether they reach the index the monitor reads (if not, the sink, index or monitor query); if they do, it is a false alarm and the threshold needs tuning. After a real gap: restore and backfill, declare the gap window, hunt it with firewall DNS and EDR data, and review threshold, monitor and change board. ::: 1. **Confirm the silence is real:** other sources still arrive, so the monitor itself is healthy. 2. **Split producer from pipeline.** Here an operating-system upgrade reset query logging on two hosts and rotated a client certificate on the other two: producer changes that bypassed the change board.[^asd-siem] 3. **Restore and backfill.** Re-enable logging; replay local query logs with their original timestamps. 4. **Declare the gap** in the register and to the SOC lead; [DetectionOps](../detectionops/) treats it as a known blind spot. 5. **Hunt the gap** with firewall DNS flows and Sysmon DNS events (ID 22[^sysmon]) for the highest-risk hosts. 6. **Prevent.** Put the logging setting in the configuration baseline and the upgrade on the change calendar; review the threshold; re-run the monthly silence test. ### Anti-patterns and myths | Anti-pattern or myth | Why it fails | Do instead | |---|---|---| | "Ingest everything, decide later" | Cost scales with volume; over-collection makes effective rules harder[^asd-siem] | A named decision per stream | | "Normalised, so drop the raw" | Parsers are wrong; OCSF deprecates attributes between releases[^ocsf-releases] | Raw in the cheapest tier; normalise forward | | "The SIEM vendor handles log management" | SIEMs search what arrives; coverage and quality are yours | Name an owner; run the control plane | | "90 days is standard" | Long-dwell intrusions outlive it[^mtrends-2026] | Retention set from dwell-time evidence and obligations | | "One schema fits every consumer" | Vendor models are not portable; OTel and ECS are still converging[^ecs-otel] | One canonical schema, pinned; map others from it | ## Deciding and adopting ### When to use and when not to use Adopt deliberate SecDataOps when any of these holds: more than a handful of sources; a bill growing faster than the estate; detections blocked on data; an incident where evidence was missing; a retention obligation. Do not create a function or buy a pipeline product when one SIEM with vendor connectors covers the estate and the bill is not a problem (keep the register and a health query anyway); when the SOC cannot consume better data (fix [SOC Ops](../soc-ops/) first); when an MDR (managed detection and response) provider runs detection, whose contract should then carry this page's coverage, retention and health terms; or when producers do not emit the events at all, which is hardening work for [CloudSecOps](../cloudsecops/) and [EndpointOps](../endpointops/). Hidden responsibilities: one more system to run, schema versions forever, an on-call for silent sources. ### Decision framework ::: figure Tiering decision for one stream ```mermaid flowchart TB A["`Stream: who uses it, for which decision?`"]:::neutral B{"`Does a detection fire on individual events?`"}:::decision I{"`Volume too high for the budget?`"}:::decision J["`Trim unused fields, filter known-benign; copy raw to lake`"]:::core C["`Hot tier, individual events (SIEM analytics)`"]:::core D{"`Needed for investigations or hunts within months?`"}:::decision E["`Lake tier (Parquet), summary to hot if useful`"]:::core F{"`Retention obligation or forensic value?`"}:::decision G["`Cold archive, integrity-protected`"]:::core H["`Do not collect; record the decision in the register`"]:::risk A --> B B -- "yes" --> I I -- "no" --> C I -- "yes" --> J --> C B -- "no" --> D D -- "yes" --> E D -- "no" --> F F -- "yes" --> G F -- "no" --> H classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029 classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029 classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029 ``` Text equivalent: Start from who uses the stream and for which decision. If a detection fires on individual events, keep them hot, first trimming unused fields and filtering known-benign events (raw copy to the lake) when volume exceeds the budget. Otherwise use the lake if investigations or hunts need it within months, a cold integrity-protected archive if only an obligation or forensic value remains, and if nothing does, do not collect, and record the decision. ::: For the platform choice, weigh five criteria with your own weights: steady-state cost including query charges; openness of format and schema; operational burden; detection latency (a lake alone does not alert); and migration risk. The [architecture table](#reference-architecture) summarises the trade-offs. ### Maturity model This is the Atlas's model, in the spirit of M-21-31's maturity tiers[^omb-m2131] but not a reproduction and not an official standard. | Level | Observable behaviour | Exit criteria | |---|---|---| | 1 Reactive | Sources added after incidents; nobody owns the list; retention is the tool default | A register with an owner per source | | 2 Inventoried | Register exists; onboarding ad hoc; silence noticed by analysts | Health monitor on critical sources; retention policy written | | 3 Contracted | Contracts, fixtures and canaries for critical sources; tiering policy applied | Coverage scored against ATT&CK data components; cost per stream known | | 4 Measured | Quality and cost metrics reviewed on cadence; pre-processing policy in force | Replay testing after pipeline changes; schema versions pinned | | 5 Optimising | Utility drives tiering; producer changes flow through the change board | A practice to keep, not a destination | ### Adoption roadmap: 30/60/90 days - **Days 1–30 (see).** Build the register from the SIEM's source list, the asset inventory and the cloud accounts; score every source with the Lab 1 method; draft the retention policy; agree one canonical schema. Quick win: a health monitor on the ten most critical sources. - **Days 31–60 (contract).** Contracts, fixtures, CI and canaries for those ten; price each stream per tier. - **Days 61–90 (tier).** Re-tier the three largest streams with replay checks; set lake lifecycle rules; run the first monthly silence test; report coverage, completeness, freshness and cost per stream as the baseline. Dependencies: a detection lead who states field needs, platform owners who will change audit settings, a budget owner. ## Toolkit ### Checklists **Quick start** - [ ] Every source in a register with an owner, a criticality and a tier - [ ] One canonical schema chosen; version pinned per source - [ ] Health monitor with a silence threshold on every high-criticality source - [ ] Raw events kept in the cheapest tier for every normalised stream **Onboarding preflight** - [ ] Consumer named the decision, fields and freshness target - [ ] Producer emits the fields (audit policy, data events, licence tier checked) - [ ] Contract written; fixtures from real samples; parse-rate and completeness tests pass - [ ] Timestamps in UTC with zone designator; event time and observed time both kept - [ ] Tier, retention and cost per day recorded; redaction applied where needed - [ ] Canary fires on live events; register updated **Routine review (monthly) and governance audit (quarterly)** - [ ] Silence test on a lab source; detection time recorded against the runbook - [ ] Quarantine counts and cost per stream compared with last month - [ ] Tier retention matches policy; log-store access reviewed; archive deletes audited - [ ] Coverage scored against ATT&CK data components; gaps prioritised with the detection lead - [ ] Chain-of-custody steps for exporting evidence documented[^rfc3227] ### Templates **Source onboarding request (intake form)** ```text Source name: (system, vendor, version) Requested by / consumer: (team; the decision this data informs) Detections or questions: (rule names, investigation questions, obligations) Required fields: (schema field names; which are mandatory for the use) Freshness needed: (p95 seconds; business hours only?) Expected volume: (events/s, GB/day; growth) Producer owner: (team; change process for audit settings) Sensitive content: (personal data, secrets; redaction needs) Retention need: (days; obligation or investigative; jurisdiction) Proposed tier: (hot / lake / cold; summary to hot?) Cost estimate per day: (by tier; who pays) Canary: (how we will prove it works on live events) ``` **Retention policy, one page.** Periods count from ingestion and run at the same time: a 90-day hot copy mirrored into a 400-day lake keeps data for 400 days, not 490. | Log class | Examples | Hot (days) | Lake (days) | Cold (days) | Basis | |---|---|---|---|---|---| | Identity, privileged access, cloud control plane | Sign-ins, role changes, management events | 90 | 400 | 548 | Investigation need; dwell-time evidence;[^mtrends-2026] obligation | | Endpoint process and DNS | 4688, Sysmon 1/22 | 30 | 400 | 0 | Investigation need | | Network allowed flows | Firewall allow, VPC flow logs | summary only | 400 | 0 | Hunting; volume | **ADR stub for a tiering change** ```text ADR-00N: Route raw events to the lake; keep per-minute summaries hot Status: proposed | accepted | superseded Context: volume, cost per day today, detections and hunts that use the stream (list) Decision: tiers, retention per tier, pre-processing rule, schema version Consequences: query latency for hunts; summary job to maintain; expected cost Validation: replay of days through the new pipeline; detections that must still fire Rollback: restore the previous route; raw data still in the lake, so no evidence is lost Owner / date / review date ``` **Source-health runbook card** ```text Source: Owner: Criticality: <0-3> Expected cadence: Silence threshold: Page: Inform: SOC lead First checks: collector connection; producer service; certificate expiry; quarantine count Backfill: Alternative sources for the gap: ``` ### Deep comparisons | Option | Trade-off summary | |---|---| | Ingest-time vs query-time normalisation | Ingest-time queries faster; query-time preserves the source format and fixes apply retroactively[^asim] | | Pipeline product vs collector-native | Products add routing, reduction, redaction and a UI;[^forrester-dpm] collector-native is free and testable as code but costs engineering time | | OCSF vs ECS/OTel vs vendor models | OCSF is Linux Foundation-governed with a Security Lake ecosystem;[^ocsf-site] ECS/OTel suit observability-first estates;[^ecs-otel] vendor models are least portable | ### FAQ **Is SecDataOps a real discipline or a marketing term?** Both. A few vendors and writers use the name, with differing definitions ([origins](#origins-and-evolution-of-the-label)). The work it names is documented by NIST, OMB, ASD and CISA as log management and event logging.[^nist-800-92r1][^omb-m2131][^asd-logging] Use the practice; be careful with the label. **Which schema should we pick?** The one your main consumers query in: ASIM on Sentinel, CIM on Splunk, UDM on Google SecOps; OCSF for an open lake; ECS/OTel if the estate is observability-first. One canonical schema, pinned per source. **How long should we keep logs?** Long enough for your investigations and obligations. Start from the obligations in the [methods table](#methods-standards-and-frameworks) and ASD's starting point there, then test the result against the dwell times under [the problem](#the-problem-secdataops-solves). **Our SIEM bill is the problem; is a pipeline product the answer?** Often part of it: Forrester notes the immediate value is reducing log sizes and ingest costs.[^forrester-dpm] Do the register and contracts first; buy if engineering time is the constraint. **How do we know a source is silent before an analyst does?** A health monitor comparing last-event time with a per-source cadence threshold, tested monthly by stopping a lab source; ASD explicitly asks for such processes.[^asd-siem] ## Going further ### Advanced topics and emerging direction - **Schema convergence.** ECS is merging into OTel semantic conventions[^ecs-otel] while OCSF, a Linux Foundation project,[^ocsf-site] releases several times a year;[^ocsf-releases] mapping between them will be a recurring task. - **Detection strategies as data requirements.** ATT&CK v18's detection strategies, analytics and data components give DetectionOps and SecDataOps one vocabulary for "what data would this detection need".[^attack-v18] - **Default audit logging from cloud providers.** The CSRB recommended free access logs with six months' retention,[^csrb] and Microsoft extended email-access logging to its standard tier in 2023.[^ms-logging-2023] Providers that have not moved still gate sources by licence, so registers need a licence-gated state that is re-checked. - **AI on security data.** Vendors position clean data as the prerequisite;[^query-wp] evaluating the AI belongs to [AIOps for Security](../aiops-for-security/). ### Learning path | Stage | Do | Outcome | |---|---|---| | Prerequisites | [SOC Ops](../soc-ops/) and the [DataOps](../dataops/) glossary; one query language; JSON | You know what consumers need | | 1-day primer | The ASD-led event logging guidance[^asd-logging] and "Understanding OCSF";[^ocsf-understanding] Lab 1 | You can score a register and explain a schema | | 1-week practice | Lab 2 on your samples; a contract for one real source; a silence monitor | A contracted, monitored source | | 30-day progression | Three sources with fixtures and canaries; a tiering proposal with a replay check | A control plane in miniature | | Advanced | Lab 3; DeTT&CT coverage against your threat model;[^dettect] a schema upgrade with regression tests | You can run Day 2 | | Portfolio ideas | A public OCSF mapping with fixtures; a silence-monitor tool | Evidence of the craft | ## Wrap-up ### Executive recap and cheat sheet - **Decide:** one canonical schema; retention by log class; the first three streams to re-tier; the register's owner. - **Measure:** coverage, parse rate, completeness, freshness p95, silent-source detection time, retention compliance, cost per stream. - **Next actions:** run Lab 1 on your real estate this week; stop a lab source and time the alert. ### A judgement call: drop the firewall allow logs, or tier them? ::: example Illustrative scenario The CISO must cut the SIEM bill by a third this quarter. Firewall allowed-flow logs are 40% of ingest; no rule references them; two analysts use them "sometimes". Option A: stop collecting allowed flows and keep denies, saving the full 40% now. Option B: summarise allowed flows per minute for the SIEM and route the raw flows to a lake with 400 days' retention, saving most of the SIEM cost but adding lake storage, a summary job and a slower query path. ::: **Assumptions.** Ingest-priced SIEM; a lake can be enabled within the quarter; the analysts' "sometimes" cannot be quantified because the SIEM's query audit log was never retained (a finding in itself). For a US federal agency the choice is already made: M-21-31 lists firewall permit events among required logs, so Option A is not available.[^omb-m2131] **Decision process.** Utility first: no rule fires on individual allowed flows, so the hot tier does not need them. But an exfiltration path is by definition an allowed path (reasoning, not a measured statistic), so the [decision framework](#decision-framework) sends the stream to the lake. Cost second: the [worked calculation](#metrics-and-kpis) prices the lake path at well under half the hot path while retaining more. Reversibility third: Option A destroys evidence that cannot be recreated; Option B is undone by promoting from the lake. **Why the attractive choice is wrong.** Option A delivers the exact number asked for, this quarter, with no new system. Its cost stays invisible until an investigation needs flows that were never stored; ASD's framing, that retention must accommodate the organisation's mean time to detect threats,[^asd-siem] applies. **Recommendation.** Option B, if a one-week replay shows no detection regresses and a hunt over the lake answers an exfiltration question within an agreed time. If the lake cannot be enabled this quarter, summarise now and route raw flows to cold object storage rather than dropping them. **Uncertainty:** real usage is unknown, so turn on SIEM query auditing first. ### Learning objectives and mastery check After this page you should be able to **explain** why the label is emerging and which guidance the practice rests on; **build** and score a coverage register; **design** onboarding with contracts, fixtures, canaries and a health monitor; **decide** a stream's tier and retention; and **diagnose** a silent source or a parse-rate drop and prevent its return. ::: answer 1. A detection engineer says a brute-force rule "works in testing but never fires in production". Which SecDataOps checks come first? Contract completeness on live samples (is `src_endpoint.ip` or `status_id` empty in production?), the Sigma `logsource` mapping (a mismatch "can render the detection ineffective"[^sigma-logsource]), device coverage, and the canary. Only then is it a rule problem. ::: ::: answer 2. Your register shows Sysmon on 310 of 1,200 endpoints with 90 days' retention. What is its device completeness score, and what is the first fix? 25.8%, below the Lab 1 script's 26% boundary, so band 1 (rounding first would give band 2: pick a rule and keep it). The first fix is deployment coverage, not retention: a rule that sees a quarter of the fleet is a quarter of a rule. ::: ::: answer 3. A platform team proposes to sample 1 in 10 Windows process-creation events to cut cost. What do you say? No: rules and investigations depend on individual process-creation events. Offer the other levers (trim unused fields, route full events to the lake, filter known-benign processes in the Sysmon configuration) and replay a week to prove detections still fire. ::: ::: answer 4. The CSRB recommended free access logs with six months' retention, and one provider changed its defaults in 2023. How should your register treat licence-gated sources? Give them an explicit state (collected, licence-gated or not emitted), the licence that unlocks them and the date checked, and re-check when a provider announces new defaults, as Microsoft did for email-access logs in 2023.[^ms-logging-2023] Collecting is not detecting: State also needed a custom rule on the log.[^csrb] ::: ::: answer 5. You normalise at ingest into OCSF tables and delete the raw. Six months later OCSF deprecates an attribute your mapping used. What went wrong? Without raw events you cannot re-parse; old records keep a deprecated shape forever. Keep raw in the cheapest tier, pin the schema version in the contract, and re-parse forward on upgrade.[^asim] ::: ### Conclusion Before anyone can detect, hunt, investigate or prove, someone must have made sure the evidence exists, is parsed, arrives on time, can be trusted and can be paid for. Governments have written that duty down for years as log management; vendors have lately called it SecDataOps. The name may not last. The work will, because every other security function inherits its failures. The craft reduces to habits that are easy to state and hard to keep: treat each source as a product with a contract; parse strictly and keep the raw; tier by use, with retention set by how long intrusions actually last; notice silence before an adversary benefits from it; and measure the data alongside the bill. Do those, and the cost of the SIEM becomes a set of decisions rather than a surprise. ## Sources and further reading All sources were opened on 2026-10-17; the evidence ledger (`research/secdataops-ledger.md`) maps claims to sources, records the archive copies used and lists what could not be fetched. [^tenzir-blog]: [The SecDataOps evolution: why security data needs its own pipeline](https://tenzir.com/blog/the-secdataops-revolution-why-security-data-needs-its-own-pipeline) — Tenzir (Matthias Vallentin), 2025-02-04. Checked 2026-10-17. [^tenzir-ga]: [Tenzir Platform Goes GA at Black Hat 2024](https://tenzir.com/company/press/tenzir-platform-goes-ga-at-black-hat-2024/) — Tenzir press release, 2024-08-06. Checked 2026-10-17. [^query-cast]: [SecDataOps Cast — About](https://podcast.query.ai/2367079/about) — Query.AI Inc., podcast since 2024. Checked 2026-10-17. [^rau-2023]: [Introducing The Complete Guide to SecDataOps and Vulnerability Management on AWS](https://securityboulevard.com/2023/03/introducing-the-complete-guide-to-secdataops-and-vulnerability-management-on-aws/) — Security Boulevard (Jonathan Rau), 2023-03-07. Checked 2026-10-17. [^query-phase2]: [The Security Data Operations Workshop: A Roadmap for Phase 2 – Analysis & Strategy](https://www.query.ai/resources/blogs/the-security-data-operations-workshop-a-roadmap-for-phase-2-analysis-strategy/) — Query (Neal Bridges), 2025-07-10. Checked 2026-10-17. [^query-wp]: [Security Data Operations Assessment & Strategy Guide (PDF)](https://www.query.ai/wp-content/uploads/2026/02/SDO-White-Paper.pdf) — Query, 2026 (undated document; URL path February 2026). Checked 2026-10-17. [^forrester-dpm]: [If You're Not Using Data Pipeline Management For Security And IT, You Need To](https://www.forrester.com/blogs/if-youre-not-using-data-pipeline-management-dpm-for-security-and-it-you-need-to/) — Forrester (A. Mellen et al.), 2024-11-12. Checked 2026-10-17. [^asd-logging]: [Best practices for event logging and threat detection (PDF)](https://www.cyber.gov.au/sites/default/files/2024-08/best-practices-for-event-logging-and-threat-detection.pdf) — ASD's ACSC with CISA, FBI, NSA and partners, 2024-08. Checked 2026-10-17 (archive copy). [^asd-siem]: [Implementing SIEM and SOAR platforms: practitioner guidance (PDF)](https://www.cyber.gov.au/sites/default/files/2025-05/implementing_siem_and_soar_platforms_-_practitioner_guidance.pdf) — ASD's ACSC with international partners, 2025-05. Checked 2026-10-17 (archive copy). [^omb-m2131]: [M-21-31: Improving the Federal Government's Investigative and Remediation Capabilities Related to Cybersecurity Incidents (PDF)](https://www.whitehouse.gov/wp-content/uploads/2021/08/M-21-31-Improving-the-Federal-Governments-Investigative-and-Remediation-Capabilities-Related-to-Cybersecurity-Incidents.pdf) — US Office of Management and Budget, 2021-08-27. Checked 2026-10-17. [^nist-800-92r1]: [Cybersecurity Log Management Planning Guide, SP 800-92 Rev. 1 (initial public draft)](https://csrc.nist.gov/pubs/sp/800/92/r1/ipd) — NIST (K. Scarfone, M. Souppaya), 2023-10-11. Checked 2026-10-17. [^nist-au11]: [NIST SP 800-53 Rev. 5, AU-11 Audit Record Retention](https://csf.tools/reference/nist-sp-800-53/r5/au/au-11/) — Rev. 5.2.0 control text as rendered by csf.tools, undated. Checked 2026-10-17. [^csrb]: [Review of the Summer 2023 Microsoft Exchange Online Intrusion (PDF)](https://www.cisa.gov/sites/default/files/2025-03/CSRBReviewOfTheSummer2023MEOIntrusion508.pdf) — Cyber Safety Review Board (CISA/DHS), 2024-03-20. Checked 2026-10-17. [^ms-logging-2023]: [Expanding cloud logging to give customers deeper security visibility](https://www.microsoft.com/en-us/security/blog/2023/07/19/expanding-cloud-logging-to-give-customers-deeper-security-visibility/) — Microsoft Security blog (Vasu Jakkal), 2023-07-19. Checked 2026-10-17. [^ms-audit-mailboxes]: [Manage mailbox auditing](https://learn.microsoft.com/en-us/purview/audit-mailboxes) — Microsoft Learn, updated 2026-06-19. Checked 2026-10-17. [^certin]: [Directions under section 70B(6) of the Information Technology Act, 2000 (PDF)](https://www.cert-in.org.in/PDF/CERT-In_Directions_70B_28.04.2022.pdf) — CERT-In, 2022-04-28. Checked 2026-10-17. [^rfc3227]: [RFC 3227: Guidelines for Evidence Collection and Archiving (BCP 55)](https://www.rfc-editor.org/rfc/rfc3227) — IETF, 2002-02. Checked 2026-10-17. [^mtrends-2026]: [M-Trends 2026: Data, Insights, and Strategies From the Frontlines](https://cloud.google.com/blog/topics/threat-intelligence/m-trends-2026) — Google Cloud blog (Mandiant, J. Kutscher), 2026-03-24. Checked 2026-10-17. [^unc5537]: [UNC5537 Targets Snowflake Customer Instances for Data Theft and Extortion](https://cloud.google.com/blog/topics/threat-intelligence/unc5537-snowflake-data-theft-extortion) — Google Cloud blog (Mandiant), 2024-06-10, updated 2024-06-17. Checked 2026-10-17. [^ocsf-site]: [Open Cybersecurity Schema Framework](https://ocsf.io/) — OCSF (Linux Foundation project since 2024-11-19), undated. Checked 2026-10-17. [^ocsf-releases]: [Releases, ocsf/ocsf-schema](https://github.com/ocsf/ocsf-schema/releases) — OCSF on GitHub, latest 1.9.0 of 2026-08-03. Checked 2026-10-17. [^ocsf-understanding]: [Understanding the Open Cybersecurity Schema Framework](https://github.com/ocsf/ocsf-docs/blob/main/overview/understanding-ocsf.md) — OCSF (P. Agbabian), document v1.17, April 2026. Checked 2026-10-17. [^ocsf-auth]: [Authentication (3002) event class](https://schema.ocsf.io/classes/authentication) — OCSF schema browser, schema 1.9.0. Checked 2026-10-17. [^ocsf-network]: [Network Activity (4001) event class, schema 1.3.0](https://schema.ocsf.io/1.3.0/classes/network_activity) — OCSF schema browser. Checked 2026-10-17. [^ocsf-enrichment]: [Enrichment object, schema 1.3.0](https://schema.ocsf.io/1.3.0/objects/enrichment) — OCSF schema browser. Checked 2026-10-17. [^ecs]: [Elastic Common Schema (ECS) reference](https://www.elastic.co/guide/en/ecs/current/ecs-reference.html) — Elastic, version 9.5.0. Checked 2026-10-17. [^ecs-otel]: [Announcing the Elastic Common Schema (ECS) and OpenTelemetry Semantic Convention Convergence](https://opentelemetry.io/blog/2023/ecs-otel-semconv-convergence/) — OpenTelemetry blog (Reiley Yang), 2023-04-17. Checked 2026-10-17. [^otel-logs]: [Logs Data Model](https://opentelemetry.io/docs/specs/otel/logs/data-model/) — OpenTelemetry specification, status Stable. Checked 2026-10-17. [^asim]: [Normalization and the Advanced Security Information Model (ASIM)](https://learn.microsoft.com/en-us/azure/sentinel/normalization) — Microsoft Learn, updated 2026-09-10. Checked 2026-10-17. [^splunk-cim]: [Overview of the Splunk Common Information Model](https://help.splunk.com/en/data-management/common-information-model/5.2/introduction) — Splunk Help, CIM 5.2. Checked 2026-10-17. [^udm]: [UDM overview](https://docs.cloud.google.com/chronicle/docs/event-processing/udm-overview) — Google Security Operations documentation, undated. Checked 2026-10-17. [^sigma-logsource]: [Log Sources](https://sigmahq.io/docs/basics/log-sources.html) — SigmaHQ documentation, undated. Checked 2026-10-17. [^attack-dc]: [Data Components](https://attack.mitre.org/datacomponents/) — MITRE ATT&CK, content v20.0. Checked 2026-10-17. [^attack-v18]: [Updates – October 2025 (ATT&CK v18)](https://attack.mitre.org/resources/updates/updates-october-2025/) — MITRE ATT&CK, 2025-10-28. Checked 2026-10-17. [^dettect]: [DeTT&CT: Detect Tactics, Techniques & Combat Threats](https://github.com/rabobank-cdc/DeTTECT) — project README, `rabobank-cdc` on GitHub, v2.2.0, GPL-3.0. Checked 2026-10-17. [^dettect-dq]: [Data quality scoring](https://github.com/rabobank-cdc/DeTTECT/wiki/Data-quality-scoring) — DeTT&CT wiki, undated. Checked 2026-10-17. [^seclake]: [What is Amazon Security Lake?](https://docs.aws.amazon.com/security-lake/latest/userguide/what-is-security-lake.html) — AWS documentation, undated. Checked 2026-10-17. [^seclake-custom]: [Collecting data from custom sources in Security Lake](https://docs.aws.amazon.com/security-lake/latest/userguide/custom-sources.html) — AWS documentation, undated (OCSF 1.3 limit as of the check). Checked 2026-10-17. [^seclake-lifecycle]: [Lifecycle management in Security Lake](https://docs.aws.amazon.com/security-lake/latest/userguide/lifecycle-management.html) — AWS documentation, undated. Checked 2026-10-17. [^s3-classes]: [Understanding and managing Amazon S3 storage classes](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html) — Amazon S3 User Guide, undated. Checked 2026-10-17. [^s3-object-lock]: [Locking objects with Object Lock](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock.html) — Amazon S3 User Guide, undated. Checked 2026-10-17. [^azmon-plans]: [Azure Monitor Logs overview (table plans)](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/data-platform-logs) — Microsoft Learn, updated 2026-09-02. Checked 2026-10-17. [^azmon-retention]: [Manage data retention in a Log Analytics workspace](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/data-retention-configure) — Microsoft Learn, updated 2026-09-02. Checked 2026-10-17. [^sentinel-lake]: [Microsoft Sentinel data lake overview](https://learn.microsoft.com/en-us/azure/sentinel/datalake/sentinel-lake-overview) — Microsoft Learn, updated 2025-08-11. Checked 2026-10-17. [^cloudtrail-data]: [Logging data events](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/logging-data-events-with-cloudtrail.html) — AWS CloudTrail User Guide, undated. Checked 2026-10-17. [^sysmon]: [Sysmon](https://learn.microsoft.com/en-us/sysinternals/downloads/sysmon) — Microsoft Learn, Sysinternals (M. Russinovich, T. Garnier), 2026-09-10. Checked 2026-10-17. [^wef]: [Use Windows Event Forwarding to help with intrusion detection](https://learn.microsoft.com/en-us/windows/security/threat-protection/use-windows-event-forwarding-to-assist-in-intrusion-detection) — Microsoft Learn, updated 2025-08-18. Checked 2026-10-17. [^snowflake-ah]: [ACCESS_HISTORY view](https://docs.snowflake.com/en/sql-reference/account-usage/access_history) — Snowflake documentation, undated. Checked 2026-10-17. [^snowflake-qh]: [QUERY_HISTORY view](https://docs.snowflake.com/en/sql-reference/account-usage/query_history) — Snowflake documentation, undated. Checked 2026-10-17. [^lme]: [Logging Made Easy (LME)](https://github.com/cisagov/LME) — CISA on GitHub; support retired and repository archived 2026-05-22. Checked 2026-10-17. [^vector-remap]: [Remap with VRL](https://vector.dev/docs/reference/configuration/transforms/remap/) — Vector documentation, undated. Checked 2026-10-17. [^vector-route]: [Route transform](https://vector.dev/docs/reference/configuration/transforms/route/) — Vector documentation, undated. Checked 2026-10-17. [^vector-releases]: [Vector releases](https://vector.dev/releases/) — Vector, latest 0.59.0 of 2026-10-06. Checked 2026-10-17. [^vector-syslog]: [Syslog source](https://vector.dev/docs/reference/configuration/sources/syslog/) — Vector documentation, undated. Checked 2026-10-17. [^vector-secrets]: [Secrets management](https://vector.dev/docs/reference/configuration/secrets/) — Vector documentation, undated. Checked 2026-10-17. [^vector-buffers]: [Buffering model](https://vector.dev/docs/architecture/buffering-model/) — Vector documentation, undated. Checked 2026-10-17. [^vector-e2e]: [End-to-end acknowledgements](https://vector.dev/docs/architecture/end-to-end-acknowledgements/) — Vector documentation, undated. Checked 2026-10-17. [^vrl-functions]: [VRL function reference](https://vector.dev/docs/reference/vrl/functions/) — Vector documentation, undated. Checked 2026-10-17. [^fluentbit]: [Data pipeline](https://docs.fluentbit.io/manual/concepts/data-pipeline) — Fluent Bit documentation, undated. Checked 2026-10-17. [^otel-redaction]: [Redaction processor README](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/redactionprocessor/README.md) — OpenTelemetry Collector Contrib on GitHub, main branch. Checked 2026-10-17. [^cribl-cost]: [Reduce Costs (use case)](https://docs.cribl.io/use-cases/usecase-reduce-cost/) — Cribl documentation, undated. Checked 2026-10-17. Further reading: [Query's Phase 1 discovery guide](https://www.query.ai/resources/blogs/security-data-operations-discovery/) (Neal Bridges, 2025-07-01), an interview-led assessment method. **Changelog.** - 2026-10-17: first edition. Term status confirmed as emerging after opening the vendor and author sources; Lab 1 scripts executed locally; Vector, Sysmon, AWS CLI and Athena examples syntax-checked or documented but not executed. - 2026-10-17 (r1 revision): after an independent review, the reference pipeline parses strictly, keeps event time, listens on authenticated TLS and emits OCSF 1.3.0 for the lake; M-21-31 retention scope, the CSRB case and Microsoft's 2023 logging change corrected; enrichment and delivery guarantees added; acronyms defined; Lab 1 fixture made reproducible; three diagrams fixed; restatements cut to keep the body within the length band. - 2026-10-17, post-score polish (`reviews/secdataops-polish.md`): AU-11 and M-21-31 quotations made verbatim; zstd wording, Entra lag, Lab 3 query and lifecycle storage class corrected; integrity options, "as of" dates and acronyms added; FAQ repeats trimmed; figures 2 and 3 uncrossed, 3 and 4 shortened; Lab 1 re-executed with unchanged output. - 2026-10-17, polish check (`reviews/secdataops-polish-check.md`): DeTT&CT credited to its authors and sponsor.