How to Integrate Continuous Improvement in Daily SOC Operations - Focus Point: Integrity and Configuration Compliance Monitoring

Désirée Sacher-Boldewin (Security Architect · Finance Informatik)

Swiss Cyber Storm 2021 · Day 1 · Scenario_2

Overview

In this insightful talk from Swiss Cyber Storm, Désirée Sacher-Boldewin, a seasoned Security Architect at Finance Informatik, tackles a pervasive challenge plaguing Security Operations Centers (SOCs): the overwhelming volume of false alarms and the resulting analyst burnout. Her presentation, "How to Integrate Continuous Improvement in Daily SOC Operations - Focus Point: Integrity and Configuration Compliance Monitoring," argues passionately for a paradigm shift in how security alerts are analyzed and categorized. Sacher-Boldewin contends that the traditional binary classification of "true positive" or "false positive" is fundamentally flawed and detrimental to effective security operations.

Watch on YouTube · Slides

Visual summary for How to Integrate Continuous Improvement in Daily SOC Operations - Focus Point: Integrity and Configuration Compliance Monitoring by Désirée Sacher-Boldewin
Visual summary for How to Integrate Continuous Improvement in Daily SOC Operations - Focus Point: Integrity and Configuration Compliance Monitoring by Désirée Sacher-Boldewin

Key moments

  1. 0:30 Speaker's motivation for sustainable security and efficient SOC
  2. 3:15 SOC often focused on IT operations, not cybersecurity threats
  3. 4:25 First steps in analyzing and categorizing security alerts
  4. 12:05 Identifying undocumented ("black") changes in the infrastructure
  5. 15:20 The goal: Unauthorized changes requiring incident management
  6. 16:20 Standardized IT service management and hardening documents are key
  7. 21:10 Q&A: Building metrics that prevent teams from gaming the system

How to Integrate Continuous Improvement in Daily SOC Operations - Focus Point: Integrity and Configuration Compliance Monitoring

Speakers: Désirée Sacher-Boldewin, Security Architect, Finance Informatik

Conference: Swiss Cyber Storm

YouTube: https://www.youtube.com/watch?v=ra4LZouxIyk

Overview

In this insightful talk from Swiss Cyber Storm, Désirée Sacher-Boldewin, a seasoned Security Architect at Finance Informatik, tackles a pervasive challenge plaguing Security Operations Centers (SOCs): the overwhelming volume of false alarms and the resulting analyst burnout. Her presentation, "How to Integrate Continuous Improvement in Daily SOC Operations - Focus Point: Integrity and Configuration Compliance Monitoring," argues passionately for a paradigm shift in how security alerts are analyzed and categorized. Sacher-Boldewin contends that the traditional binary classification of "true positive" or "false positive" is fundamentally flawed and detrimental to effective security operations.

The core of her message is the necessity of moving beyond simplistic metrics to implement intelligent processes and efficient workflows that foster sustainable security. By focusing on the root causes of alerts, rather than merely their immediate security impact, organizations can empower their SOC teams, improve their detection capabilities, and build trust with management. The talk specifically highlights the critical role of integrity monitoring and configuration compliance monitoring as foundational elements for uncovering systemic issues that often masquerade as "false positives," ultimately enabling a more mature and responsive security posture.

Background

▶ Watch: Speaker's motivation for sustainable security and efficient SOC (0:30)

Désirée Sacher-Boldewin brings over 15 years of experience in IT security, with a decade spent in highly regulated industries, to her analysis of SOC operational challenges. Her motivation stems from firsthand observations of how security operations often falter, leading to inefficiencies and employee dissatisfaction. She identifies several key pain points: the struggle of junior analysts to make informed decisions without proper guidance, the high turnover rate due to analyst burnout caused by an inundation of meaningless alerts, and the underutilization of expensive security tools because teams haven't learned to leverage them effectively.

A significant part of the problem, Sacher-Boldewin explains, lies in management's preoccupation with "true positive" versus "false positive" ratios. This focus, she argues, distracts from the more crucial objective: improving the quality and relevance of the alerts themselves. In her experience, SOCs often devolve into "operational data verification and technical security quality assurance centers," spending disproportionate time chasing issues that should ideally be addressed by robust IT service management processes, rather than focusing on genuine cyber threats. This operational drift prevents SOC analysts from applying their specialized forensic skills and fulfilling their core mission.

To counteract this, Sacher-Boldewin advocates for a structured approach to analyzing security events post-triage. She proposes that by meticulously categorizing the underlying reasons for alerts, organizations can identify systemic weaknesses in processes, tooling, and communication. Her framework, which has evolved over several presentations and is documented in her published papers, aims to provide a granular understanding of alert origins. For this particular talk, she chose to illustrate her methodology using compliance configuration monitoring and integrity monitoring as specific examples, demonstrating how these often-overlooked areas can reveal critical insights into an organization's security maturity and operational health.

Key Findings

▶ Watch: First steps in analyzing and categorizing security alerts (4:25)

The central thesis of Sacher-Boldewin's presentation revolves around several critical findings that challenge conventional wisdom in SOC operations:

  1. Security Event Analysis is Not Binary: The most profound finding is the assertion that analyzing security events is "never ever a binary thing." The simplistic true positive/false positive categorization is inadequate. Sacher-Boldewin identifies at least eight distinct categories for what might initially appear to be a "false positive" or even a "true positive" in a superficial assessment. These categories reveal underlying organizational issues rather than just an alert's immediate threat level. The specific classification of an event, she stresses, must be resolved within each organization based on its unique risk appetite and operational context.
  1. Standardized IT Service Management (ITSM) is Foundational: A mature SOC cannot function effectively without robust and standardized ITSM processes. This includes a stable change management system that encompasses all relevant teams, a well-defined incident management framework, an accurate and trusted asset management system, and effective problem management. Without these basics, SOC analysts are forced to spend their time "running after information" and filling operational gaps, leading to burnout and an inability to focus on genuine security threats.
  1. Regular Review of Hardening and Scoping Documents is Crucial: For effective integrity and configuration monitoring, a clearly defined "normal" state is indispensable. The scoping of hardening documents and files — which dictate the expected, secure configuration of systems — must be regularly reviewed and integrated into the system lifecycle. Without a reliable baseline, the SOC lacks the necessary reference points to accurately identify anomalies, rendering monitoring efforts significantly less effective.
  1. Meaningful KPIs Drive Improvement, Not Gaming: The proposed alert categorization system allows for the generation of Key Performance Indicators (KPIs) that provide actionable insights rather than easily gamed metrics. Instead of simply measuring the number of alerts handled, Sacher-Boldewin suggests tracking metrics such as "number of changes without formal documentation" (identifying shadow IT or black changes), "number of activities with no change documentation required" (revealing unreviewed security policies), or the "number of unauthorized changes without legitimate cause." These KPIs foster a discussion about the quality of services and processes, enabling management to understand and address foundational problems.
  1. Metrics Should Be Designed for Trust and Impact: To prevent metrics from being "hacked" (e.g., assigning tickets quickly without actual work), Sacher-Boldewin emphasizes building metrics that teams want to report correctly. If reporting accurate information leads to positive impacts for the team and the organization (e.g., addressing root causes, improving tools, streamlining processes), then teams will be incentivized to provide honest data. This requires managers to actively use the reported information to drive improvements and engage in meaningful discussions.
  1. Integrity and Compliance Monitoring are Increasingly Critical: With the industry's shift towards cloud services and Software-as-a-Service (SaaS) models, direct agent-based monitoring becomes less feasible. Integrity and configuration compliance monitoring emerge as "really critical" methods for verifying that external configurations remain in their desired state. This makes the detailed categorization of alerts originating from these monitoring types even more important for understanding external dependencies and ensuring baseline adherence.

Technical Deep Dive

▶ Watch: Identifying undocumented ("black") changes in the infrastructure (12:05)

The technical core of Sacher-Boldewin's talk is a sophisticated decision tree or flowchart designed to guide SOC analysts through a granular categorization of security alerts, specifically focusing on those generated by integrity and configuration compliance monitoring. This process moves far beyond a simple "true positive" or "false positive" assessment, aiming to uncover the organizational, process, or tooling root cause behind each alert.

The process begins after an alert has been investigated and a root cause identified by an analyst or technical engineer. The first crucial question is: "Was this now a potential threat to the infrastructure?"

  • If YES, the event is a genuine security incident and is handed over to the incident management process for further analysis and response. This is the ultimate "true positive" path in the traditional sense.
  • If NO, the alert is not an immediate threat, but the investigation continues to understand why the alert was generated. This is where the nuanced categorization truly begins, breaking down the traditional "false positive" into several actionable sub-categories:
  1. Was this event caused by an authorized action?
  • If YES, and it was previously communicated to the SOC: This is often a temporary configuration operation (e.g., a planned change for maintenance).
  • Analyst Action: Decide whether to suppress the alert temporarily (requiring a future removal of the suppression) or, if it's a longer-term workaround, to document the blind spot with a risk entry. This highlights a process problem – a communication gap between technical teams and risk management, where security operations' suppressions are not translated into broader risk awareness.
  • If NO, but it was a legitimate violation authorized by change documentation: This means the infrastructure was changed according to official procedures, but the SOC was not informed or included in the process chain.
  • Analyst Action: Immediately adjust the baseline configuration in the monitoring system. Crucially, the SOC must become proactive to get included in future reporting process chains before such changes are rolled out, preventing future false alerts and maintaining trust. This is another process problem.
  • If NO, and it was a test alert by the SOC: This is an internal test to validate the monitoring setup.
  • Analyst Action: Do not include this in standard management reports. However, it's vital for demonstrating a mature SOC to auditors and for establishing trust in the tuning process. The speaker emphasizes that these tests are "not appreciated often enough" but are critical for validating detection capabilities.
  1. Was the reporting device configured correctly? (This path is taken if the event was not an authorized action or a SOC test).
  • If NO: This indicates a problem with the baseline configuration itself or incorrect information provided by the device owners. The SOC is basing its alerts on faulty data.
  • Analyst Action: Distinguish this as a specific alert type. This allows the SOC to report how often it's triggered by "wrong information from the outside." This is not a measure of SOC performance but rather an indicator of issues in upstream operational IT security, enabling managers to correct problems "at the source."
  1. Was there a limitation in the verification product? (This path follows if the device was configured correctly).
  • If YES: The security tooling itself is not capable of writing a specific enough rule to detect the required change. The SOC is "working with the wrong tool set."
  • Analyst Action: Categorize this specifically. Accumulating data on such instances can be used to justify reviewing the product choice and potentially acquiring a "different vendor." The speaker metaphorically calls this the "frog" product – one hoped would work, but ultimately disappointed.
  1. Did the change owners follow official policies for the change process? (This path follows if there was no product limitation).
  • If NO: This points to "black changes" – IT organizations making changes without official documentation in the change management system. This is a critical indicator of shadow IT or policy non-compliance.
  • Analyst Action: Adjust the baseline configuration. More importantly, this type of event should be reported from a broader perspective, as managers are "very interested in knowing what kind of departments are not following the regulations."
  1. Was this an activity with no change documentation required? (This path follows if official policies were followed).
  • If YES: This category covers situations where an activity occurred, but security management had previously decided it was not in scope for active monitoring or deemed relevant (e.g., ignoring AV alerts because the antivirus is expected to handle it). These are risk decisions taken by security management.
  • Analyst Action: Communicate these alerts in a way that prompts security management to regularly evaluate and review their past risk decisions. The speaker notes a common problem where security management places too much trust in the "first line of defense" without sufficient oversight, leading to unreviewed accepted risks.

Only after navigating these detailed questions, if all paths lead to an unauthorized change without legitimate cause, does the event finally get classified as a genuine security incident for incident management. This comprehensive process ensures that every alert contributes to a deeper understanding of the organization's security posture and operational health, pinpointing issues ranging from communication breakdowns and tool deficiencies to policy gaps and unmanaged risks.

Demo / Proof of Concept

▶ Watch: Standardized IT service management and hardening documents are key (16:20)

The talk primarily focuses on a conceptual framework and a decision-making process rather than demonstrating a live tool or specific proof-of-concept implementation. Désirée Sacher-Boldewin presents a decision flow diagram and discusses the categories as a structured analytical approach that analysts would apply after an initial alert investigation. While she mentions having created "mock-up reports" in previous presentations (available on YouTube and GitHub), the presentation itself does not feature a live demo of a system implementing this categorization or a proof-of-concept showing how it detects a specific "black change" or configuration drift. Instead, the emphasis is on the methodology for improving the analysis of existing alerts, irrespective of the underlying monitoring tools.

Defensive Implications

▶ Watch: Q&A: Building metrics that prevent teams from gaming the system (21:10)

The framework proposed by Désirée Sacher-Boldewin offers profound implications for how organizations can bolster their defensive capabilities and foster a more resilient security posture:

  1. Adopt Nuanced Alert Categorization: Defenders must move beyond the simplistic binary of "true/false positive." Implementing a detailed categorization framework, such as the eight-plus categories outlined, allows SOCs to identify the root cause of non-threat alerts. This shift enables the SOC to become a source of intelligence for process improvement, rather than just an alert-triage center. This means equipping analysts with the training and tools to perform this deeper analysis.
  1. Prioritize IT Service Management Maturity: The talk underscores that a mature SOC is built on the foundation of robust IT Service Management (ITSM) processes. Organizations must invest in and enforce standardized change management, incident management, asset management, and problem management. Without these, the SOC will continue to be burdened with operational tasks that drain resources and distract from genuine security threats. Defenders should advocate for these foundational improvements across the IT organization.
  1. Regularly Review and Maintain Baselines: For effective integrity and configuration compliance monitoring, it is critical to have accurate and up-to-date hardening documents and defined baselines for critical systems. These "defined states of normal" must be regularly reviewed and integrated into the system lifecycle. Defenders should establish processes to ensure these reference documents remain current, as they are the bedrock for detecting meaningful anomalies.
  1. Bridge Communication Gaps: The framework highlights pervasive communication failures between the SOC, IT operations, risk management, and security management. Defenders need to establish formal channels and processes to ensure the SOC is included early in change processes, that suppressions are communicated to risk management, and that security management regularly reviews its accepted risk decisions. This fosters organizational transparency and shared responsibility for security.
  1. Redefine KPIs for Quality, Not Quantity: Security leaders should overhaul their metrics to focus on the quality and impact of security operations, rather than just the quantity of alerts handled or incident response times. KPIs should measure systemic issues, such as the "number of changes without formal documentation" (identifying shadow IT) or the "number of activities with no change documentation required" (highlighting unreviewed risk policies). These quality-focused metrics provide actionable intelligence for driving continuous improvement and gaining management buy-in.
  1. Empower Analysts and Address Tooling Limitations: The categorization process empowers analysts to identify when tools are inadequate ("limitation and verification product" category). Defenders should use this data to make informed decisions about tool investments, demonstrating to management why new or improved tooling is necessary by showing the operational burden caused by current limitations.
  1. Embrace Internal Testing: SOCs should proactively implement test alerts to regularly validate their detection capabilities and ensure their configurations are working as expected. While these shouldn't inflate external reports, they are crucial for building internal trust, proving system maturity, and continuously tuning the monitoring environment.

By adopting these defensive implications, organizations can transform their SOC from a reactive, firefighting unit into a proactive, intelligence-gathering engine that drives continuous improvement across the entire security landscape, effectively distributing the "load of the security operation to everyone."

Key Takeaways

  • Move Beyond Binary Alert Classification: Security event analysis is complex; the traditional "true positive" vs. "false positive" dichotomy is insufficient. A nuanced categorization system is essential for understanding the root causes of alerts.
  • Foundational Role of IT Service Management: Robust and standardized IT Service Management processes (change, incident, asset, problem management) are non-negotiable prerequisites for a mature and efficient SOC.
  • Maintain Accurate Baselines: Regularly review and update hardening documents and configuration baselines. A clearly defined "normal" state is critical for effective integrity and configuration compliance monitoring to detect genuine anomalies.
  • Leverage Metrics for Quality and Improvement: Develop Key Performance Indicators (KPIs) that measure the quality of security operations and identify systemic organizational issues (e.g., undocumented changes, communication gaps), rather than focusing solely on easily gamed quantitative metrics.
  • Integrity Monitoring is Increasingly Critical: With the shift to cloud and SaaS, integrity and configuration compliance monitoring become vital tools for detecting unauthorized changes and configuration drift, especially where traditional agent-based monitoring is not possible.
  • Empowerment Through Categorization: The detailed alert categorization empowers SOC analysts to identify and report on process flaws, tooling limitations, and policy gaps, fostering continuous improvement and distributing security responsibility across the organization.

About the Speaker(s)

Désirée Sacher-Boldewin (formerly Désirée Sacher) is a highly experienced Security Architect at Finance Informatik, a prominent IT service provider for German savings banks (Sparkassen). With over 15 years in IT security, including a decade in heavily regulated industries, Désirée has gained extensive insight into the operational challenges faced by security teams. Her professional journey has exposed her to various positions, including security engineering roles, which motivated her to develop and publish frameworks aimed at improving security operations. She is driven by a passion to create sustainable security by designing intelligent processes, efficient workflows, and robust detection capabilities. Désirée is a recognized contributor to the security community, having presented at various security events and published papers on her methodologies. She is also actively involved in the FIRST metrics SIG, working to define standard frameworks for SOC metrics that emphasize quality over mere quantity.

Reviews

Dr. Zero (Offensive Security Researcher) — STRONG ACCEPT

Sacher-Boldewin takes the binary true/false positive metric that management loves, throws it in the trash, and replaces it with a categorization system that forces IT to own their undocumented changes. It is a highly practical, no-bullshit methodology for anyone trying to run a SOC without burning out their analysts in six months.

Heather Calloway (CISO) — STRONG ACCEPT

A highly practical framework that stops treating SOC false positives as unavoidable weather and starts treating them as measurable IT failures. I'd have my SOC director adopt this taxonomy to stop using security headcount to subsidize broken change management.

→ Top-rated talks at Swiss Cyber Storm 2021

All talks from Swiss Cyber Storm 2021