In 2007, Beyoncé topped the charts with her hit song “Irreplaceable,” and BlackBerry commanded 40% of the U.S. smartphone market. Within five years, Queen Bey still reigned, but BlackBerry was virtually wiped out with less than 1% of the market. The Innovator’s Dilemma had made BlackBerry the opposite of irreplaceable. The future of the SIEM market hinges on whether Splunk is more like Beyoncé or BlackBerry.

Subscribe now

A Winning Formula

IT, and later security teams, fell in love with Splunk’s fast and flexible search engine. In the early 2000s, the company pioneered an approach to centrally analyzing and troubleshooting vast amounts of event logs. Splunk made working with the icky structure of most machine data a more palatable experience.

At the heart of the platform’s success was its innovative indexing technology. Unlike traditional databases that required structured data input, Splunk was designed to handle a wide range of unstructured data. This flexibility allowed organizations to ingest logs, metrics, and other machine data without needing upfront schema definitions. What does this look like behind the scenes?

During ingestion, Splunk breaks down the data into individual events and stores them in an indexed format. This index is not a simple database table. It's a highly optimized structure that allows for lookups at a latency that supports real-time troubleshooting of IT issues like website outages and application crashes. Rapid searching is facilitated by storing raw data alongside metadata, such as timestamps, source types, and host information.

The appeal to IT teams organically extended to security operations, where the previous generation of Security Information and Event Management (SIEM) solutions like ArcSight were proving slow and cumbersome. Splunk was adopted across thousands of cybersecurity organizations, and revenues rose accordingly.

More Log Data, More Problems

Splunk became the dominant SIEM platform at a time when most enterprise infrastructure was in data centers. When its customers shifted to the cloud, Splunk struggled to adapt its architecture and pricing model. Customers became increasingly concerned when it became evident that cloud environments produce more log data—a lot more.

While I can’t cite a source for this specific figure, my experience has been that the shift to the cloud resulted in 3x more log data for a typical environment. This can be explained by the highly dynamic nature of Infrastructure-as-a-Service (IaaS). If you’re doing it right, cloud infra is constantly spinning up and down with new virtual machines, clusters, and entire networks, popping in and out of existence based on fluctuating user demands. Every change in the cloud is logged, hence terabytes of additional machine data for the security team to collect and analyze.

Splunk’s approach to indexing log data to support fast search became a source of constant cost concerns. This isn’t just anecdotal. In the customer spending data reported by SiliconAngle in 2023, a year-over-year downward trend can be seen in customer spending momentum. While 47% of customers reported plans to grow Splunk spending a few years prior, by the time the report was compiled, that number had dropped to 37%. Splunk was still prevalent in the polling data (yellow line below), but spending velocity was way down (blue line). The number of customers planning to decrease or eliminate Splunk spend had increased by over 50%.

This should not be interpreted as just a pricing problem. As explained in “Understanding Splunk’s New License Model: It’s Not the Pricing Model, It’s the Price Tag that Matters” by Cribl CEO Clint Sharp,

Offering additional models should not be confused with lowering prices. Changing models, for most customers, will offer at best marginal benefits… The use cases most likely to see lower query workloads are where Splunk is used as an online archive or primarily as an investigation tool without much scheduled workload. Additionally, if your users don’t particularly care about performance, you can likely ingest a lot more data but have a much poorer performing system with this license model.

In other words, the historical vestiges of Splunk’s internal architecture are such that they struggle to deliver significant cost savings without making painful tradeoffs. This goes back to the indexing strategy that closely ties compute and storage. Without a fundamental redesign of how data is collected, stored, and searched, cost concerns have naturally increased with the inexorable shift to the cloud and the expansion of log volume. This is supported by a recent customer survey where the top response to the question of Splunk’s current pricing was: “We don’t like Splunk’s pricing, but we feel locked in.”

Dipping its Toe in the Data Lake

The challenge posed by the exploding volume of log data has not gone unnoticed. Dropping junk logs in transit is a fast way to slim down, but there’s a limit to how much data can be dropped before the security team flies blind. Using cheap cloud storage is another approach that offers a significant upside.

Consider that Splunk Cloud storage reportedly costs between $100-$150 per uncompressed TB a month (official figures are not available). Cloud-native options like Snowflake or S3 Parquet tables cost $23 per compressed TB a month. With log data compressed at a conservative 8x, the actual storage cost in the data lake is under $5/month per collected terabyte—a fraction of the Splunk cost.

Splunk didn't lose sight of the potential benefits of cloud-native data lakes for log data. Five years ago, the company launched Data Fabric Search (DFS) with support for external data lakes and Splunk indexes, aiming to provide a unified search experience across storage options.

Then, in 2021, Splunk launched a separate data lake product called Federated Search for Amazon S3. Done right, this could have enabled a rebalancing for customers looking to optimize between storage cost and search performance. With Federated Search, Splunk users could search against data in cheap cloud storage without paying for indexing upfront. Why has this feature not addressed the price concerns that are clearly still pervasive in the latest customer polls?

The first issue is performance. Splunk’s official documentation warns, “Customers attempting to use this feature for real-time searches will perceive slower performance and reduced search functionality when compared with indexed Splunk searches on ingested data.” How much slower depends on a number of factors but, unlike alternatives like Snowflake, scaling up when necessary is cumbersome and manual. An approach that can’t deliver a consistently adequate search experience will struggle to achieve widespread adoption.

The second issue is around predictability. Splunk added an additional cost component to Federated Search that meters by how much data is queried. The documentation states, “Customers need to acquire additional licensing to enable Federated Search for Amazon S3. This is based on "Data Scan Units", and is described as the total amount of data that all searches using Federated Search for Amazon S3 have scanned in the customer's Amazon S3 buckets. This licensing SKU is independent from Splunk Virtual Compute (SVC) and any Ingest based licenses that customers may have acquired.” It would be challenging to predict how much data a SOC will need to scan, and detection engineers may be discouraged from paying by the byte for scanning their log data throughout the day. An organization might figure out how much 100 saved search detection rules would cost. Now for every new rule they develop, what would be the cost in terms of Data Scan Units?

Finally, this approach might reduce storage costs but become more expensive overall. As Splunk’s documentation warns, “Customers attempting to use this feature for high frequency searches will likely incur higher costs than natively ingesting and searching in the Splunk platform.” The DSU model drives cost anxiety even higher as stated in the documentation, “that over the course of a year, the DSUs operate on a 'use it or lose it' model.” It’s no wonder that most Splunk customers have been unable to address their cost concerns with Federated Search.

Signs to Watch For

The shortcomings of Federated Search in Splunk are not necessarily here to stay. Security teams that have invested years into their Splunk deployment may be wondering if their puppy can learn new tricks. If this is you, what should you watch for in coming product and pricing announcements?

Telltale investment areas include the junk data cleanup that drove many big customers to adopt Cribl upstream from Splunk. Improving the usability and closing functionality gaps in Ingest Actions would indicate that Splunk is serious about addressing its spiraling ingest costs.

More importantly, watch for a Federated Search that does not actively discourage its users. The elimination of Data Scan Units as a tax on top of the already significant Virtual Compute cost would be a sign that Splunk is serious about enabling data lake adoption. If DSUs are not completely eliminated, they can still be changed. Successful cloud data platforms like Snowflake, BigQuery, and Databricks all offer time-based compute pricing due to its advantages over volume-based pricing. These include predictability (new detections don’t increase costs as long as they run in parallel) and affordability (speed improvements reduce run time, driving down costs). Charging by bytes-scanned only makes sense for limited, infrequent search use cases.

Cisco paid many billions for Splunk’s products and customer base. Will the new owners be able to overcome the innovator’s dilemma that threatens Splunk’s hold on the SIEM market? New competitors are leaning heavily on architectures that effectively expose the cost and scale efficiencies of the cloud. Splunk’s future depends on its ability to embrace flexible data storage options and customer data ownership.

While security organizations can build up data expertise over time, the best way to adopt data analytics best practices is through collaboration with the data team. Here’s proven advice on how to get started.

Subscribe now

Are Data Dynamos Security Savvy? 

Unfortunately, traditional SIEM solutions, with their proprietary backends and languages, have driven a wedge between cyber and data teams. The two organizations have lacked common ground for years.

The sophistication of recent ransomware, credential stuffing, and cryptomining attacks makes the divide even wider. For example, hacker tools like OpenBullet come packed with features and plugins designed to bypass defensive measures. Understanding present-day attacks to the point of effectively detecting and mitigating them takes years of domain expertise. We can’t expect all data practitioners to also be security experts.

This makes it critical to take the right approach to collaboration between cyber and data organizations. Your company’s Director of Data Analytics might not yet be familiar with the MITRE ATT&CK framework or the persistence methods of Linux cryptomining malware, but they can still start playing an important role in defending the business. This lines up with how most centralized data teams support other departments.

When reaching out to the central data team, you can give examples of threat detection challenges that can be seen as typical data problems. Framing use cases in familiar terms early in the conversation will establish confidence that the right people are in the room and no one is wasting their time.

• Communication with known bad actors: Identifying communication requires point lookups in network traffic data against known malicious IPs and domains.

• A series of actions associated with attack techniques: Piecing together event sequences using event correlation methods to detect known attack patterns.

• Large amounts of data copied out of the organization: Calculating data volumes and checking for high or unusual results.

• Failed login attempts recurring across accounts: Analyzing login logs for repetitive failed attempts using pattern recognition techniques.

• Unexpected changes to critical systems and environments: Applying contextual datasets to log events from operating systems and cloud providers.

You can also frame the new collaboration using the “Three Vs” that are familiar to any experienced data practitioner. The organization’s threat detection challenge involves volume, with terabytes of security data generated across the environments. High SIEM ingest and storage costs often limit data collection and retention.

It also involves velocity in that threats often take too long to identify, scope, and mitigate. Since many event log sources are never collected to the SIEM, activity records often must be reactively loaded and analyzed days or weeks after the initial breach event. Proactively analyzing such sources can help the security team keep up with attacks as they unfold.

Finally, the data/cyber partnership must deal with the variety of security data. Activity logs and contextual datasets come in diverse formats and schemas, which presents a challenge when integrating, analyzing, and correlating the feeds.

Enabling the new collaboration

You need to put a few things in place to bring together the cybersecurity and data teams. The first is to establish a common data platform. This is the Achilles heel of locked-in SIEM platforms like Palo Alto Network’s XSIAM—they keep security and data orgs apart. In contrast, general-purpose analytics platforms like Snowflake support the enterprise’s typical data analytics use cases and cybersecurity. This provides the necessary foundation for effective collaboration.

Once you and your new friends from the data team agree on a joint data platform, you’ll need to agree on a data ingest strategy. Depending on the use cases you prioritize, this might involve native connectors within the data platform, managed connectors within the security platform, or a dedicated solution for collecting security data. Analytics teams will be familiar with the term ETL, which stands for Extract, Transform, Load, with leaders like Fivetran and Matillion in the data space. Security-specific equivalents have emerged in the form of observability pipelines and security data fabrics. As described in a previous post, it’s an interesting space that’s seeing rapid development and likely has a solution to meet your needs with different approaches.

Another prerequisite that the data/cyber partnership must address is the common language for collaboration. While many security products have their own proprietary search syntax, data analysts have converged on the Standard Query Language (SQL). Security teams may initially be daunted by the need to express detection logic in SQL, but recent improvements have lowered the adoption curve.

For example, Anvilogic provides a detection content library with thousands of SQL-based rules that can be deployed off the shelf or used as a starting point for custom detections. Correlated detection scenarios can be created in a low-code builder that converts logic from the UI to SQL code behind the scenes. Recent advances in Gen AI are directly relevant to the SQL skills gap, with security practitioners able to express their threat detection logic in English, which the AI copilot converts to code.

Taken together, a common data platform, collection strategy, and query language enable a fruitful collaboration between security operations and data analytics teams. Good options for each of these areas are proven and readily available. Once you agree on these, you’ll be ready to improve security outcomes together.

A Roadmap for Getting Started

A data-driven security operation is not a solution you buy. While various products can accelerate your initiative and increase its likelihood of success, the following areas should be built out in collaboration with the data team. Taken together, these form a roadmap for taking ownership of your data and aligning with how the rest of the enterprise turns data into insights.

1. Data Preparation

Collecting logs from your systems and environments involves bringing streaming them to your data platform, but making them useful takes preparation. In the separate, siloed world of cybersecurity, this was called “parsing”. Data analytics best practices establish that more work upfront is a worthwhile investment.

According to an article by data management provider Talend on the topic, “Data preparation is the process of cleaning and transforming raw data prior to processing and analysis. It is an important step prior to processing and often involves reformatting data, making corrections to data, and combining datasets to enrich data.”

Work with your data team to identify the necessary data preparation steps that will support reliable, high-fidelity threat detection. This includes having standard time zones, extracting commonly used fields from nested semi-structured data, and creating “views” that join together related datasets for easy querying. For example, you might have one table for Windows logs and another for Linux logs—a view that joins the two could be used for detections that are operating system agnostic.

2. Health Monitoring

Security operations teams often struggle with health monitoring. I’ve seen third-party audit findings ding a SOC for missing sensors, dead feeds, and out-of-date field names that cause detection misses. Even worse is when these issues are uncovered in a security incident retrospective.

While some SOCs have responded by assigning dedicated headcount to ensure that expected data is available and valid, the partnership between cyber and analytics can unlock powerful automation for this area. Data teams have mature tools and techniques for monitoring data health.

For example, the open source tool Elementary is used by thousands of data engineers to bake health checks directly into their data platform. This gives them an early warning on issues affecting freshness, volume, or schema changes. A data team using Elementary for their enterprise data health monitoring could easily extend their tests to cover the security data lake.

3. Reports and Dashboards

Building reports and dashboards is one of the most common ways in which data teams support the business. In collaboration with the security operations team, business intelligence (BI) professionals can deliver powerful and actionable insights. For example, calculating the SOC’s workload in terms of daily alert volume and how well the team is keeping up can influence hiring and training plans. Breaking down top contributors to alert volume can focus tuning efforts on noisy detection rules. And making detection coverage metrics available to leadership can raise confidence and give executive stakeholders a sense of the progress that the SOC is making in staying ahead of the threats facing the organization.

With the security team using the same data platform as the rest of the enterprise, there will surely be a well-maintained and broadly accessible BI tool where the new SOC reports can be hosted. I recommend regularly meeting with your data analysts to review how reports are being used and refine the dashboards to reflect the KPIs that drive real action across detection engineering, hiring, and architecture.

4. Built-in Functions

Another advantage of giving the cyber team access to the enterprise’s central data platform is the availability of powerful built-in functions. These functions deliver a range of capabilities relevant to threat detection, threat hunting, health monitoring, and incident response. What’s really cool is that they are delivered “as a service” without the need for developer cycles.

For example, Snowflake’s built-in Haversine function takes the geolocation of two points and returns the distance between them. Advanced functions for time window operations, data generation, and semantic classification can play a role in the detection of beaconing malware, synthetically generated domains, and other TTPs.

5. Data Science

We’ll never know how much damage resulted from security operations being late to the AI/ML revolution. It’s undeniable that the lack of data science access faced by most cybersecurity organizations has hampered their ability to succeed. Security data lakes and the partnership between cyber and data teams present an opportunity to catch up.

After lining up with the data org on an analytics platform, the security operation can start exploring which of its initiatives could benefit from data science. From insider threats, to credential stuffing and “low and slow” detection evasion, data science can help with use cases that are top of mind for many security organizations. See my previous post on behavior analytics for ways to get started.

As you can see, partnering with the data team can tremendously benefit the security operation. To get started, you don’t need to hire world-class data experts for the security team. Your company already has great data resources on staff. They’re just helping other teams.

The foundational requirements and roadmap outlined above can help you and your peers on the data side of the house know what to expect from a partnership. You’ll be surprised how much support cyber/data collaboration generates among senior leadership. Data practitioners are usually happy to help defend the enterprise against threat actors. This is a great way to do more with what your organization already has—a top CISO priority in 2024.

Insider Threats Remain a Top Challenge

H. L. Mencken was a famously pessimistic American journalist and critic. A hundred years ago, he wrote classic gems like "No one ever went broke underestimating the intelligence of the American public" and "Bachelors know more about women than married men; if they didn’t they’d be married too." His relationship pessimism might be misguided, but it’s right on the money for insider threats.

Subscribe now

How serious is the problem? In 2023, nearly 3 out of 4 organizations reported feeling moderately to extremely vulnerable to insider threats. Rightfully so, says the report, with over 30% of data breaches attributed to insider attacks. Another study found that these attacks have increased by around 50% over the past two years. And the financial impact of an insider-related breach averages $17 million in North America.

A well-documented insider incident hit Tesla in late 2018 when an employee, Guangzhi Cao, stole thousands of sensitive files before joining a Chinese competitor. The case was settled in April 2021, with Cao agreeing to pay an undisclosed amount to Tesla and cooperate in the investigation. Notably, the security investigation began only after Cao resigned from his role and was not triggered by any activity related to the data theft.

Insider threats pose a unique challenge to security teams because they skip many of the initial steps in the cyber kill chain. They know the environment, they’re provided some level of authorized access, and they don’t need to get directions from an external Command & Control (C2). They can go straight to taking Actions on Objectives.

Threat detection depends on breaking the cyber kill chain before the threat actor can complete their objectives. Detection engineers consider ways to spot recon activity, malware execution, and other signs of an attack. A threat that can jump to the last stage in the chain is at a distinct advantage.

As a result, many security teams become receptive to alternative approaches. User Behavior Analytics (UBA), involving machine learning and anomaly detection, is closely associated with combatting insider threats. The idea is that the algorithms will learn what “good” looks like, and anything else will be flagged as likely malicious. For large enterprises, this becomes the basis of a data science project for the insider risk program. Smaller SOCs buy and deploy off-the-shelf solutions for the same goal.

Mencken’s Law of Threat Detection

From large organizations with teams of data scientists to smaller ones buying commercial UBA products, everyone seems to be struggling to pull off their insider threat initiatives. I’ve spoken with security leaders over the years at enterprises ranging from large banks to travel tech. Their insider risk initiatives always seem to be in science project mode, with production launch just a few quarters beyond the horizon. Commercial UBA products have disappointed all but the blissfully unaware. The result is that most CISOs have not achieved an effective insider threat detection program.

Could a pessimistic approach hold the key to more successful outcomes? I propose we turn to that old downer H. L. Mencken. Considering all the failed data science projects and UBA deployments for insider risk, he might inspire a new guiding principle for addressing this challenge.

At first glance, this is a depressing rule. Why should machine learning success depend on the SOC’s threat modeling? I previously wrote about the challenge of implementing cybersecurity anomaly detection and achieving a consistent relationship between unusual and anomalous event categorization. Lacework, for example, attempted to eliminate detection rules with machine learning but was itself eliminated as a private entity when Fortinet eventually acquired it at a fraction of the $8.3 billion valuation it once held.

The same challenges apply to insider threat detection, where there are no training datasets for insider threat activity, and “unusual” things happen every day across an even wider range of systems and exfiltration paths.

While machine learning may be able to detect some cases of insider TTPs, it is wildly optimistic to assume that it would detect something we haven’t explicitly set out to catch. There are too many potential points of failure, from access to the relevant log data, to successful baselining, to tuning to a degree of accuracy where a SOC analyst would review the resulting alert. How would a pessimist like Mencken go about detecting insider threats?

Pessimistic or Realistic?

Mencken’s Law points us to a more limited but potentially successful path to insider threat detection. Something very interesting happens when we start with threat modeling early in the insider risk initiative. For a primer on threat modeling and how it supports detection engineering in general, check out The Detection Responsibility Handshake:

Insider threat modeling is critical because it often reveals a surprising path to detection. It turns out that many insider threat techniques can be effectively detected without machine learning. And where detection implementation avoids the need to train and maintain an ML baseline for each employee, the required effort is dramatically reduced. The odds of overall project success are improved accordingly.

For example, take the previously mentioned scenario from Tesla, in which an insider copied 300,000 files to his personal iCloud account. In evaluating this TTP, we can analyze historical file transfer records and chart the occurrence of high-volume transfers. If a transfer of over 1,000 files is rare in the environment, then that can serve as a fixed threshold in a static detection. There would be little additional value in benchmarking individual employees and alerting when users transfer an “unusually high” number of files. Both approaches would have spotted the insider threat that evaded detection at Tesla.

In other cases, threat modeling would reveal detection strategies where a single event achieves a similar outcome without building and maintaining an ML black box. For example, a team might model an insider performing data exfiltration via email forwarding. UBA-style profiling of anomalous forwarding activity could be one way to improve detection coverage for this TTP. However, alternative techniques, such as alerting when a user creates a mailbox-level email forwarding rule or comparing destination email addresses to the sender’s name, could catch many instances of this threat.

A guiding principle for mature detection engineering teams is that coverage is never absolute, and tradeoffs always exist. The time it takes to build and maintain a detection can be spent working on another detection- so tradeoffs must be considered. Threat modeling allows SOC teams to evaluate options for insider threat detection and consider the full range of effective alternatives, including highly-quality detection libraries, threat-informed static detections, and multi-dimensional correlations.

Many insider threat scenarios can be addressed without the effort and uncertainty of ML-based user behavior analytics. Mencken himself once said, "All men are frauds. The only difference between them is that some admit it. I myself deny it." Insider threat detection poses an enduring challenge for security operations teams after years of disappointing products and scuttled data science projects. Security leaders should instead consider making incremental progress toward insider risk reduction through investments in threat modeling and detection engineering.

Don’t get me wrong—Security Orchestration, Automation, and Response (SOAR) is an increasingly valuable part of security operations. Also, this isn’t a “SOAR is dead” post, although hyperautomation is certainly a cooler name. This is about using the right tool for the job. Read on to learn how SOC teams unintentionally take on long-term risk and complexity by using (or abusing) SOAR for detection engineering use cases.

Subscribe now

SOAR Picks Up The SIEM Slack

Security automation is a best practice for reducing the burden of manual SOC processes. For example, many SOCs have an automated playbook that sorts through employee-reported phishing emails, checks elements against threat intelligence, and yanks any related emails from company mailboxes. Should the same automation approach be applied to threat detection?

The problem starts when detection requirements involve uncollected activity logs, a common situation given SOCs tend to have less than half their data in the SIEM. Visibility gaps become detection gaps. The workaround goes like this: the SOAR reaches out to a data source API, asks a question, and notifies the team based on the result. It sounds workable, and I’ve spoken with several security leaders considering SOAR to solve their SIEM detection gaps.

Another way SOAR compensates for SIEM shortcomings is by attempting to reduce alert noise. Many SIEM deployments generate far more alerts than the SOC can process, resulting in missed threats and a generally bummer experience. Some teams have taken to automation playbooks as a correlation layer for noisy atomic alerts. The hope is for SOAR to close out a significant portion of alerts or tickets based on automated steps before they reach the analyst’s queue.

Rubber Bands, Paperclips, and Playbooks

We just reviewed some compelling-sounding applications for security automation. The effectiveness of these use cases in an enterprise production environment is another story. Since SOAR solutions were designed to help with manual tasks downstream from the SIEM, they have critical gaps affecting teams trying to MacGyver a threat detection solution.

One challenge with using SOAR as a rule engine stems from how logs are turned into alerts. Logs that match detection logic are often clumped together, with many matching events happening around the same time. For example, a password-spraying attack involving thousands of attempts in a few minutes. That’s why SIEM rule engines include sophisticated deduplication and debouncing mechanisms to merge related detections and prevent issuing a succession of identical alerts.

Also, detection engineering has matured recently, including through alignment with the MITRE ATT&CK framework. Security organizations speak in ATT&CK terms and measure coverage against ATT&CK-mapped tactics and techniques (TTPs). A team that makes up for SIEM shortcomings by breaking out some of its detections to the SOAR cannot effectively track and prioritize its detection coverage. The result is a detection engineering mess.

SOC metrics suffer because the SOAR does not report key SIEM stats such as top offenders, mean time to detect, and false positive rates. As with MITRE ATT&CK coverage, these metrics are not supported by solutions not designed for threat detection. This impacts the SOC's overall maturity, especially that of the detection engineering program, at a time when improving maturity is a priority for many CISOs.

Finally, security analysts are directly impacted by SOAR abuse when they find themselves on the hook to build extensive detection content from scratch. Unlike SIEM products with frequently updated content packs, SOAR providers do not consider threat detection coverage in the scope of their solution. The customer team must then bear the burden of researching emerging attacks, developing detection logic, and testing it for false positives and negatives. All this is especially time-consuming when performed outside of a purpose-built detection engineering workbench.

The Right Tool for the Job

Using the intended tool for each phase in the SOC lifecycle can avoid workarounds that cause a detection engineering mess. This is especially clear as the traditional SIEM continues to unbundle (or Splunkbundle). From data collection to analytics to automation, reliance on best-of-breed tooling removes the motivation to hack unintended use cases.

For data collection, owning the data pipeline and the data lake removes the aforementioned limitations on visibility. With 80% lower costs and always-hot retention, previously siloed source data becomes directly available for downstream analytics- including purpose-built threat detection tooling.

A purpose-built detection engine should include differentiated capabilities for cutting development time, reducing noise, and improving fidelity. Since the fastest detections to create are the ones you don’t need to build yourself, a threat detection content library is the ultimate timesaver and is out of scope for SOAR vendors.

Also significant are the advanced detection engineering capabilities of purpose-built platforms. For example, multi-stage detection scenarios that identify events of interest in log data and correlate those into threat scenarios. Alerting on threat scenarios, rather than individual atomic detections, is an emerging best practice that results in significantly higher fidelity.

Implementing multi-stage scenario-based detections in a SOAR solution would be cumbersome, if not impossible, at scale. From time-window correlation to grouping indicators by ATT&CK technique, workflow automation does not support detection engineering best practices. With good visibility, SIEM can apply enrichment data and context in an iterative process to avoid false positives and negatives. Detection logic, rather than workflow automation, is the right place to address alert fidelity.

There’s also a data angle to consider. Events of interest, as shown in the second tier below, are fertile ground for threat hunters and data scientists to connect the dots in patterns that don’t emerge from raw events. Advanced use cases require quality detection output stored back in the data platform. When a team relies on downstream automation to sort through the noise, they take on new visibility gaps around activity patterns in the environment. This will be an increasingly important SOC design principle as a wave of AI-powered triage automation solutions hits the shelves.

The relationship between SIEM and SOAR is too often a case of “garbage in, garbage out.” Fixing that requires detection engineering maturity, where high-fidelity alerts set up the automation solution for success. Reliable detection output can be enriched by the SOAR and routed across the organization to reduce the pressure on the SOC and enable self-service remediation. By avoiding SOAR abuse, investments in security automation can go further, and the potential of hyperautomation can be unleashed.

For some of its largest vendors, cybersecurity has become a race towards maximum lock-in. Palo Alto Networks, CrowdStrike, SentinelOne, and other major providers now compete across endpoint, cloud, identity, and SIEM- with success dependent on customers buying it all and staying put.

For security leaders, this trend threatens to curtail access to innovative new solutions and erode the bargaining power needed to keep budget expansion aligned with the rest of the business. Can organizations continue to leverage products from platform providers without losing their independence? Like the Eagles sang in Hotel California: “You can check out any time you like... But you can never leave."

Subscribe now

Buying Your Business

One sign that platform players are counting on long-term lock-in is their willingness to buy new business. They’re willing to take losses upfront because they believe any customer entering their walled garden will face difficulty switching later. This approach can be seen in the recent string of increasingly aggressive “loss leader” moves.

For example, CrowdStrike announced in May that Microsoft customers could get a heavily subsidized endpoint agent to run alongside the competing Defender product. In an interview, CrowdStrike Head of Products Raj Rajamani made it clear that Microsoft customers would find this offering “very attractively priced.”

Palo Alto Networks has been especially aggressive at luring customers onto its platform. Earlier this year, the company announced a new offer where they give away their product for free for the whole duration of new customer’s existing contracts. According to the press release:

The offer enables qualified customers to accelerate platformization and seamlessly transition to Cortex XDR by providing  a "no-cost" period of the solution until existing legacy contracts expire. Additionally, the program includes a baseline package of "no-cost" professional services to assist with the agent migration.

Platformization, with its high switching costs, is what justifies extended free-use offers. Buyers should take this into account when considering heavily subsidized product offers.

This approach was taken to the extreme a few months later when Palo Alto struck a major deal with IBM for its SIEM business. Interestingly, IBM’s cloud SIEM product is being taken to the woodchipper. This deal isn’t about technology or capabilities. It’s about capturing customers.

Unconfirmed rumors place the deal size at around $500 million, on top of which PANW will pay IBM consultants to migrate customers to their platform. It would take years of subscription payments to offset these substantial expenditures, and the new class of “all-in-one” cybersecurity platforms is designed to make that happen with the full Hotel California experience.

Platform Exclusivity

Why is switching from a platform that includes both SIEM and endpoint agents nearly impossible? Consider the challenge of an ordinary SIEM migration, where endpoint telemetry is one of the top sources for threat detection. The migrating team reviews migrated detection logic and verifies that coverage is maintained in the new system. But for a SIEM+endpoint migration, all endpoint-related SIEM detections must be rebuilt—a huge additional effort.

The secret to this double lock-in is how next-gen SIEM platforms mandate their own endpoint agents. For example, Palo Alto Networks XSIAM licensing documentation specifies that the SIEM cannot be purchased separately from the Cortex agent.

Platform exclusivity also extends to content and capabilities. Feeding agent telemetry to a SIEM is one thing, but detection rules and models designed and tuned for a particular agent provide questionable effectiveness for third-party agents. Also, agent health monitoring is being increasingly folded into the all-in-one SIEM platform, potentially breaking essential SOC processes that depend on analyzing agent logs to identify deployment gaps and agent health issues.

I’ve heard of rare situations where an exception was made for a customer to keep their former agent while ramping up the next-gen SIEM. A team in that situation should take extra caution in validating that the competitor’s platform properly analyzes the signal from the “other guy’s” agents.

The tight coupling between SIEM and endpoint makes migrating either one incredibly difficult. Many SOCs could find migrating both at the same time to be nearly impossible. The platform vendors are incentivized to maximize this lock-in at the expense of security organizations.

The Modular Alternative

The prospect of double lock-in has driven interest in open, modular SOC architectures. This week, Jon Oltsik, the well-known analyst and founder of the cybersecurity practice at Enterprise Security Group, wrote about how “no vendor will deliver the whole enchilada.” In his CSO Online article “Cybersecurity at a crossroads: Time to shift to an architectural approach,” Oltsik rallied against the idea of an all-in-one platform.

First, the notion of moving all the data to one repository is completely outdated due to data volume and constant change. Future security operations must adhere to a federated data model…

Note that I do see large organizations standardizing with data lake technologies like Databricks and Snowflake, and I also see a role here for things like the Amazon security lake. While this makes sense today, we’ll see new data management platforms in the future with compelling security use cases. Enterprise security operations architectures must have the flexibility to migrate or integrate data in the future.

Turning to an architecture with enough flexibility to run security operations across a mix of data platforms presents an alternative to platformization. For mega vendors spending huge sums to construct walled gardens, this is the ultimate nightmare. But how do enterprise security leaders feel about it?

CISOs Umesh Yerram and Arvin Bansal, each responsible for protecting large and complex enterprises, recently published an excellent whitepaper titled Modular Stack: The Future of Cybersecurity Design. Yerram and Bansal acknowledge that big changes are happening in the SIEM market but warn of the dangers that consolidation presents to security operations:

Enterprises are made up of heterogeneous, hybrid environments that CISO organizations must secure using a range of security controls. The consolidation in the SIEM marketplace will not benefit CISO organizations significantly due to few key reasons:

1.      Consolidation will result in more vendor lock-in

2.      Innovation is put on the back burner to prioritize integration efforts.

3.      Integration of these consolidations will take time, and most will fail based on the history of past consolidations.

Speaking from experience, Yerram and Bansal urge their fellow security leaders to watch out for short-term discounts from platform consolidators. They argue that “the significant changes in the SIEM market necessitate a strategic approach to security detection & response architecture that emphasizes the principles of standardization, flexibility, data ownership, cost-efficiency, and advanced technology adoption.”

Data ownership is a key enabler for a modular design strategy. Last week, I wrote about four elements of ownership and why XDR vendors that built their SIEM around agents see independent ownership as a threat. They want to hold your data on their platform, so swapping out any part of their portfolio becomes a monumental task.

The Modular Stack whitepaper also includes several great diagrams showing what forward-thinking CISOs envision as the opposite of the locked-in platform. In the image below, the security organization is aligned with the rest of the enterprise on a data platform that supports myriad use cases and cross-collaboration. Security data lives alongside business data and is used to power threat detection, vulnerability management, compliance automation, etc. These use cases can be supported by best-of-breed applications with SIEM capabilities and detection content plugged into the team’s data lake.

With logs normalized in the security data lake and a best-of-breed SOAR solution downstream, a SIEM purchase decision becomes reversible. A team might choose to buy Anvilogic, for example, as the SIEM layer for the data lake, and a year or a decade later, replace it with an alternative that meets new requirements or offers better pricing. The modularity in this approach de-risks the whole operation, leading to lower costs and full access to innovation.

The competition between two movements, one towards platformization and one towards modularity, may be decided by an awkward question. Like signing a prenup before the wedding, security leaders should ask, “Is it reversible?” Not because they think they’ll soon part ways but out of recognition that lock-in inevitably leads to higher costs and fewer options. As CISOs Yerram and Bansal concluded in their whitepaper, access to “innovative best-of-breed threat detection capabilities on top of standardized data” is the best position for security organizations. Just don’t tell that to the platform companies banking on lock-in.

This isn’t another post about the morning of May 15, when IBM Qradar customers learned that the company was exiting the SIEM business. They could start from scratch with Palo Alto Networks’ XSIAM product or something else entirely. You don’t have to go home, but you can’t stay here.

Industry pundits have been discussing the news ad nauseam, so I won’t rehash why these products hit a wall and whether those were savvy business deals. Instead, I present how an ownership mindset can prepare your organization for what comes next.

Subscribe now

One: Own Your Pipelines

So far, the biggest progress towards breaking lock-in and taking ownership has been around data pipelines. In the past, data collection tooling was seen as an integral part of SIEM. Splunk has Universal Forwarders, Securonix has Remote Ingestion Nodes, etc. Typical enterprise deployments include thousands of agents installed on servers and networks throughout the organization.

The many-to-one approach became less popular as demand grew for multiple pipeline destinations, including cloud storage and data science platforms. Cribl (founded by former Splunk product leaders) built a huge following by pointing out that SIEM vendors charging by volume might lack the motivation to add robust volume reduction features to their pipeline products. The business case was made to separate the pipeline from the SIEM, and many security organizations successfully took ownership of their pipeline.

Running thousands of vendor-specific agents across servers, clusters, and networks is a huge source of lock-in. We should all feel for Qradar customers, who have significant on-prem environments and depend on agents like “IBM® WinCollect 10” for their pipeline. As I described in my previous post on security data fabric, there are plenty of open-source or independent alternatives available now.

The important thing is that your team can send data from any source to any destination they choose- including multiple parallel destinations. This opens up effective evaluation and migration options as the SIEM landscape twists and turns.

Two: Own Your Data

One thing is having the flexibility to ship logs to your destinations of choice. Owning the data is something else, and it’s often overlooked, especially when it’s all in the cloud. What’s the difference between storing your data in a cloud-based security platform like Exabeam or a cloud-based data platform like Snowflake?

Your level of data ownership depends on the freedom to:

• Query the data directly from third-party tools

• Use the data for data science and ML model training

• Easily export data to an open format such as Parquet or JSON

• Set your own retention policies

• Create views on the data for easy access by different users and applications

• Govern who can access data based on granular role-based access controls

• Audit how the data has been accessed and modified

Since most security platforms don’t give you open access to their data backend, your level of data ownership is significantly limited. This is especially true for the XDR vendors that built their SIEM offering around EDR, where data ownership was never on the table. As a result, a replatform project would involve wholesale export and import of many terabytes- an effort that they’re incentivized to discourage.

Cloud data platforms like Snowflake were designed for interoperability, with dozens of tools for ETL, BI, and data science expected to be plugged in. Many Snowflake customers use multiple data visualization products, for example, and can easily try new ones. The flexibility and level of ownership demonstrated below are becoming available to security organizations, promising to enable more “plug-and-play” security operations.

Three: Own Your Detections

Security operations teams spend an incredible amount of energy on detection engineering. Haider Dost wrote a great article about the seven stages his team goes through in their detection development lifecycle. The thought and effort required to work through these stages are necessary to achieve high-fidelity detections in an enterprise environment.

SIEM vendors tend to underplay the investment required to preserve detection investments during migration. The Google Chronicle team, for example, released a post titled Migrate Off That Old SIEM Already! where they paid lip service to this challenge, writing:

Don't migrate all content. Migrating all of your existing detection content, rules, alerts, dashboards, and visualizations to a new SIEM is a lot of work and it's not always necessary. Take the time to evaluate your current detection coverage and only migrate the rules that you need for your new environment.

That misses the point! While some detections might no longer be required, any SIEM migration strategy must account for the hundreds or thousands of threat detections that must be ported to the new platform.

This is an even bigger challenge when moving between platforms with different capabilities, where migration must account for both syntax and functionality. To pick on the SIEM formerly known as Chronicle (they’ve rebranded to SecOps), GCP posted this month on newly added support for statistical search using “Count Distinct.” Detection engineers building rules in Splunk have used aggregate statistics for years. This begs the question: Do the XDR next-gen SIEMs support this function yet? What other functionality is not supported by the newcomers?

The safest approach is investing in detection rules not tied to a particular platform. Standard languages like SQL and Python will always have broad platform support and include analytics functions that cover everything that the detection engineering team needs. Already, several leading SIEM solutions support SQL, and the trend from proprietary, vendor-specific rule languages to general-purpose analytics languages enables detection ownership. This may be the biggest factor in minimizing the impact of an unexpected SIEM migration situation.

Four: Own Your Requirements

When platform vendors spend millions to buy out their competition, they count on customers having fuzzy, loosely defined requirements. The fallout is experienced only later when security operations teams scramble to work around limitations in their new tooling. I’ve seen cases where this happens mid-migration, forcing the whole project to revert after months of effort. The fallout includes breach risk, team burnout, and compliance issues.

With all the shakeups in the SIEM market, it’s prudent to proactively capture detailed SIEM requirements based on what’s working in the current deployment. In large organizations, these requirements span multiple teams and use cases. Documenting the capabilities and performance expectations around custom ingest, activity analytics, incident investigation, health monitoring, and maturity metrics is an extensive project. But an up-front investment in owning your requirements enables fast reactions and confident decision-making if your SIEM of choice changes hands or goes under.

The hits will keep on coming. Unlike some segments of cybersecurity, the SIEM space in 2024 is still far from stabilizing on a handful of safe choices. Your organization can develop an ownership mindset by seeing its pipelines, data, content, and requirements as independent and modular. Owning your future can help you survive and thrive through these crazy times.

Bruce Schneier is a genuine cybersecurity legend. The author of Applied Cryptography, inventor of the term “security theater,” and long-time blogger wrote a short piece about USB sticks in 2007. I was reminded of this post while walking down the aisles at this year’s RSA conference.

Schneier’s “A Security Market for Lemons” proves, with references to Noble Prize-winning research, that in cybersecurity, it’s not the best product that beats the competition. The nature of our industry is such that weak solutions routinely get purchased over stronger alternatives. This unfortunate dynamic dominates the SIEM space, where buyers rarely measure and compare threat detection performance—even for one of the largest items in their budget. That needs to change for the sake of the industry and everyone’s security posture.

Subscribe now

A Security Market for Lemons, Explained

In his post on the “market for lemons” phenomenon, Schneier asks, “Why are there so many bad security products out there?” That question is as relevant today as ever. The disruption caused by recent breaches at UnitedHealth and First American Financial Corp must have involved several layers of failed protection. Why are industry-leading solutions failing our defenders?

Schneier explains that the issue is not just that cybersecurity is a challenging problem. The cybersecurity market suffers from an acute case of “information asymmetry.” In cryptography, threat detection, and much of the cyber industry, the vendor knows much about its product’s effectiveness that the buyer doesn’t. The analogy is to a used car salesman who knows he’s selling a “lemon” model where the transmission tends to fall out after a few hundred miles on the highway. He knows the defect, but the poor soul who just stepped onto the lot does not.

This imbalance is more dangerous than it seems. An American economist won a Nobel prize in the 1970s for showing what happens when buyers can’t spot the lemons. Since good products are inherently more costly than lousy ones, they’re priced out by the seemingly identical lemons. A vicious cycle of “adverse selection” emerges; eventually, the whole market is lemons.

Among the most cited and downloaded economic journal papers of all time, the market for lemons theory applies to various areas of our lives. But cybersecurity effectiveness is uniquely hard to judge. No performance benchmarks are trusted for comparing competing products, and the test scores in advertisements seem quite fishy when compared as a group. If a high school class averaged 99% on an exam, that test was probably left in the copy machine the day before.

It speaks volumes that the SIEM space hasn’t seen an equivalent attempt at comparative effectiveness analysis. In his article, Bruce Schneier describes the effect that the market for lemons dynamic has on the competitive landscape.

In the late 1980s and early 1990s, there were more than a hundred competing firewall products. The few that “won” weren’t the most secure firewalls; they were the ones that were easy to set up, easy to use and didn’t annoy users too much.

When buyers can’t accurately assess the value or effectiveness of a product in advance, the market opens up for lemons. Other conditions that drive lemon markets include an incentive for sellers to pass off a low-quality product as a higher-quality one, limitations on data sharing between market participants, buyer pessimism, and lack of regulation and warranties. Each of these conditions is present in cybersecurity, especially in the SIEM market.

Spotting the Lemons

I often encounter “adverse selection” when speaking with security leaders evaluating SIEM solutions. Their selection criteria revolves around feature availability (“Does the product do UBA? Is there a management API?”) and content quantity (“How many integrations? How many rules?”). Both appeal to decision-makers because they’re easy to check before purchasing. Unfortunately, the link between these selection criteria and the product’s effectiveness at threat detection is weak at best.

The perils of buying detections by the pound were laid out in my post on The Detection Responsibility Handshake. Regarding feature availability, missing features can be a dealbreaker if, for example, your plan is to detect threats across multiple log repositories in two or more clouds (it’s pretty cool how Anvilogic does that). However, security leaders should distinguish between feature availability and product effectiveness. An outcome-based approach means performance testing across the people, processes, and technology involved in the SIEM deployment. Specifically, threat detection performance.

Take, for example, this recent exchange between an irate financial services SOC director and one of my colleagues. The SOC had engaged a third-party red team to drill an attack on the cloud environment. As usually happens, the red team achieved their objectives and captured the metaphorical flag. The SOC director was mad because his team had received no alerts during the exercise. No warnings that may have given them a chance to fight back. What the hexadecimal happened?!

Detection failure can result from a range of causes. These causes can be identified with a breach postmortem or testing during the SIEM selection process. Obviously, it’s better to know about these issues proactively. They include:

• Visibility failure: Was something (e.g., license or capacity limitation) preventing the relevant log data from being collected?

• Rule failure: Was the logic needed to surface an event of interest missing or disabled?

• Bandwidth failure: Was the alert buried in a queue where the SOC could not keep up?

• Triage failure: Was the alert too vague or lacking the context needed for proper escalation?

• Retention failure: Was the data needed for detection and response no longer accessible when relevant intelligence was received or the investigation began?

When the SOC director and my colleague reviewed what happened during the red team exercise, the root cause surprised them. The log events capturing the attacker’s tracks had been received. These events triggered rules designed to flag the TTPs used in the exercise, and alerts were issued. So why didn’t the SOC know about the attack?

It turned out that the alert notifications were dropped at the SOC’s outsourced service provider. The MSSP didn’t pass the alerts along because they had applied overly broad allowlists that automatically suppressed them. In this case, it was a triage failure, but any of the causes described above could have been responsible.

What had started as a frustrating experience for the SOC director turned into an opportunity to examine the performance of their SIEM and its dependencies. This level of testing is crucial for measuring performance and identifying shortcomings.

Let’s Talk About SIEM Performance

The outcome of a drill like the one described above should include a performance score that quantifies the SOC's success at spotting attacks. Performance metrics will depend on the solution, its coverage for the environment, its fit for the team, and many other factors that depend on the vendor and the organization doing the testing. That’s why we may never see standard scores or reliable third-party comparisons in the SIEM space. But that shouldn’t discourage you from testing SIEM performance upfront and repeatedly after deployment.

Armed with an outcome-based evaluation, you shrink the vendor’s information advantage. You can start spotting the lemons through red team exercises, adversary emulation tooling, or tabletop simulations. You might find that you are relying on a lemon today or that the next-next-gen solution with the great conference swag is actually a lemon. A committed movement towards SIEM testing will level the playing field and encourage the development of products and features that make a difference in the fight against threat actors.

Could anyone have predicted the spectacular downfall of cloud security heavyweight Lacework? Leading VCs had poured over a billion dollars of funding into the company, attracting high-profile tech executives, top-tier engineering talent, and over 1,000 employees at its peak. What they missed about the company’s AI strategy is instructive for a cybersecurity industry counting more than ever on artificial intelligence.

In my analysis below, I’m relying only on what’s publicy available on Lacework’s website and documentation. Everything that follows is my personal interpretation of how a flawed AI strategy doomed the startup once called “one of this generation’s most important cybersecurity companies.”

Subscribe now

Lost in Translation

Lacework’s crown jewel was its Polygraph technology. Applying artificial intelligence across extensive cloud activity logs, Polygraph was described as “the revolutionary way to use your data to automatically find what matters most across your cloud environment.” Out with threat detection rules that define what attacks look like. In with anomaly-based detections that learn what’s expected in the environment and flag the bad guys when they do something out of the ordinary.

Shifting threat detection from signatures to algorithms was not without precedent. Before Lacework created Polygraph, endpoint vendors like Cylance and CrowdStrike used machine learning to disrupt the incumbents in their industry. Symantec, McAfee, and other traditional antivirus providers relied on signatures that encoded telltale bits of malware for detection. As researchers identified new viruses, signature rules were pushed out to millions of endpoint protection agents worldwide.

The new anti-malware solutions took a different approach, relying on machine learning instead of signatures. CrowdStrike explained the difference in a 2016 post titled “CrowdStrike Machine Learning and VirusTotal.”

Traditional AV engines look for signatures or heuristics, i.e. sequences of specific bytes in the file. A malware author can easily change those detected sequences or add obfuscation layers. In contrast, using machine learning, we look at the broader picture and extract so-called “features” from the files analyzed.

CrowdStrike trained its ML models on millions of known malware samples before it could detect threats as effectively as signature-based antivirus. Once its algorithms were sufficiently trained, updates could be delivered less frequently, and new threats could be identified more reliably. Could Lacework achieve a similar breakthrough against threats in the cloud?

Unfortunately for the Polygraph team, there was no equivalent database with millions of cloud attacks that could be fed into an ML model. Researchers have been collecting malware file samples for years. In the cloud, attacks are mainly API calls recorded in log data, spread over time, and are rarely packaged for analysis. Lacework’s “no rules” and “high fidelity” algorithms would need to look elsewhere for their training.

Unusual but Not Malicious

The difficulty of cloud threat detection would never stand in the way of enterprises shifting their data centers to AWS, Azure, and GCP. With the great cloud migration underway, Lacework told anxious security leaders that they could “uncover unknown threats like abnormal logins and escalation of privileges with patented Polygraph anomaly-based approach.”

From this description, we can learn where Lacework’s algorithms would find the needed training data: in the customer’s environment.

The polygraph technology dynamically develops a behavioral model of your services and infrastructure. The model understands natural hierarchies including processes, containers, pods, and machines. It then develops behavioral models that the polygraph monitors in search of activities that fall outside the model’s parameters.

In a post from 2017, one of the Lacework cofounders explained how they apply machine learning to turn the customer’s own activity patterns into highly effective threat detections that attackers would struggle to circumvent.

We use unsupervised machine learning to build a baseline for each cloud deployment. We develop exhaustive insights, with information about all entities and their behaviors. Every baseline is as unique as the deployment it protects, making it easy to accurately spot the changes (using supervised machine learning) that always accompany an attack. A successful hacker would 1) have to have an omniscient understanding of your specific cloud deployment and 2) design an attack that perfectly mimicked normal behavior in that deployment. A tough challenge indeed.

This approach may have worked well in Lacework’s early days when Polygraph ran in a test environment and at early customers— many of whom were startups. Unfortunately, production cloud environments running at full scale are notoriously busy places. The DevOps movement encourages multiple code releases a day. AWS has over 200 services, where cloud users can spin up complex clusters in seconds and tear them down just as easily. Lacework’s Polygraph technology had to monitor environments where “unusual” things happened frequently and rarely involved an attack.

While Lacework promised “high fidelity alert reduction” for the cloud, they didn’t experience the same liftoff as CrowdStrike saw on the endpoint. Instead, the company laid off 20% of its workforce just months after raising $1.3 billion at a whopping $8.3 billion valuation. This wasn’t just a matter of rising interest rates. AI for anomaly detection on cloud activity wasn’t the same magic sauce as AI trained on millions of samples for malware detection.

The “No Rules” Dillema

Lacework heavily promoted its technology for threat detection with “no rules.” Its official documentation states that “polygraph is the first and only zero touch cloud workload protection platform, which requires no rules, no policies, and no logs for breach detection.” The company released an ebook titled “Cloud Security Automation for Dummies” that equated automation with having “No rules to write or maintain.” But even if the AI worked as intended, was the “no rules” philosophy doomed from the start?

While security teams at early-stage startups might appreciate a cloud security solution with few knobs to turn, established security organizations have detection engineering functions dedicated to striking a balance between noisy alerts and missed attacks. This balance requires customization and fine-tuning. Detection-as-code has seen widespread adoption as a best practice for spotting threats. And detection-as-code, at its core, is all about rules.

Likely in response to pressure from up-market customers demanding greater flexibility, Lacework eventually launched the Lacework Query Language (LQL). LQL enabled custom rule creation and detection signatures maintained by the Lacework research team. For example, a blog on Kubernetes threat detection describes how “Lacework has released LQL policies that detect deleted and deprecated API calls for all Kubernetes API versions in this GitHub repository.”

The new slogan for the “No Rules” cloud security platform would be “Rules Optional.”

Adding custom rules with the Lacework Query Language doesn't appear to have achieved the necessary balance between automation and flexibility. The company’s Customers page still contains mainly smaller organizations, and the GitHub project to help customers use LQL for threat hunting hasn’t been updated in over two years. The “no rules” approach to cloud threat detection wasn’t suited for the medium and large enterprises that cybersecurity companies depend on for success in the long term.

AI Lessons Learned

Cybersecurity entrepreneurs, investors, and practitioners can take away several important lessons from Lacework’s story. As artificial intelligence technology hits peak hype, how can you evaluate the limits of an AI strategy for threat detection use cases?

• Training data matters: What training datasets were used and were they big enough to support the inference the models will be expected to make?

• The black-box/flexibility tradeoff: Does the solution achieve the necessary balance between automation and customization?

• Independent validation: What tests could we run to expose model issues like overfitting or knowledge gaps?

You can use these considerations as a framework for emerging AI use cases like security copilot. Ask how the copilot’s models were trained and how much actual analyst activity was used during training. Find out what feedback mechanism exists to tune the copilot and whether customization “as code” options exist for integration with the organization’s triage workflows. Finally, validate independently by asking it to do some basic and not-so-basic analyst work to map out the edges of the copilot’s abilities.

Shifting hard, tedious work from humans to machines has been the story of humanity from the wheel to sliced bread. Artificial intelligence is an abstract and opaque technology, and cybersecurity presents it with a fog of war where effectiveness can be especially hard to tell. Some of Silicon Valley’s sharpest minds missed the flaws in Lacework’s application of machine learning to cloud security. When a lesson costs over a billion dollars, we should pay attention.

Usage-based pricing for security operations is about as popular as trepanning for curing headaches. But does the “pay for what you use” model deserve a closer look? This post will highlight four concerns I’ve heard from security leaders and why SOCs should embrace the cost model that’s helped unleash data-driven transformations across the enterprise.

Subscribe now

What is Usage-Based Pricing?

When data platforms ran on-premises, physical servers were deployed based on an expectation of how much compute power would be needed to keep users happy. In off-hours, the utilization of those resources was low, and when things got busy, the available resources were maxed out. At that point, searches might slow down or wait in a queue. This fixed-capacity model is still the case for virtually all cloud-based SIEM deployments.

One of the breakthroughs that Snowflake brought to data platforms was how it took advantage of the elasticity of the cloud. Users could tap into a virtually limitless pool of compute resources when needed, and “give back” resources when idle. The technology to make fully elastic scaling easy and instant was groundbreaking. The pricing model would need to be innovative as well.

Snowflake created a pricing model where users query their data with a virtual warehouse that has a T-shirt size. These sizes range from X-Small to 6X-Large (that’s XXXXXX-Large), with each size twice as big and twice the cost. The total cost is calculated by the number of seconds for which the warehouse is active. So a Medium warehouse that runs for 84 seconds costs the organization 0.0011 x 84 = 0.0924 credits. These warehouses can automatically suspend and resume based on query activity, hence the term usage-based pricing. But is this a good fit for security operations?

Myth: My Team Would Cut Back on Investigations

When learning about usage-based pricing, some security leaders worry about analysts feeling pressured to cut short investigations. No one wants to adopt a new pricing model only to miss threats in the environment. In my work with dozens of security teams that have adopted data lakes, I have yet to see this happen.

The chart below shows why analysts don’t hold back under usage-based models. Compared to a traditional data warehouse or SIEM with fixed resources, a cloud data platform that elastically scales to meet demand is radically more cost-effective. During idle hours, the cost difference is substantial. In times of peak activity, the cost may be comparable. So while analyst activity may affect the degree to which savings are realized, cost savings are still achieved over time. Security analysts don’t need to worry if the organization is saving 80% over alternative approaches or “only” 50%. Properly informed during onboarding, analysts do not work differently on account of usage-based pricing.

Myth: We’d Get Surprise Bills

Many SIEMs get a bad wrap for ingest-based pricing and its effect on detection coverage and IR. But at least they let you control which sources ship data, and tune down sensors that are “too good” at generating visibility. There’s a fear that in periods of heightened SOC activity, usage-based bills could become a nasty surprise at the end of the month.

The truth is that some forms of usage-based pricing do run the risk of spiky, even unexpectedly high, costs. Data platforms that charge by “bytes scanned” have been responsible for unpleasant CFO conversations for at least one SOC leader I worked with. In their case, it was the AWS Athena service where the IR team ran a sweep that touched months-worth of log data. As a result, a single investigation cost them over $5,000. good thing it was a false alarm right?

It’s unfair and counterproductive to expect SOC analysts to predetermine how many terabytes each search will scan. At scale, this is a recipe for disaster. In fact, AWS recently introduced a “provisioned capacity” pricing model for Athena where customers get access to always-on resources without per-bytes pricing. While not exactly usage-based, AWS explained that the change was prompted by customers reporting that “it is difficult to forecast [the] Athena costs. Athena charges by the volume of data scanned, which is often difficult to predict as it depends on the size of your data set, the construction of the user queries, and the storage format for the data.”

The time-based usage model does not carry the same risk. There are only 24 hours in a day, and query timeout enforcement can automatically prevent runaway queries. I’ve seen cases where incident responders temporarily request beefier warehouses to meet their SLAs— but resizing can be subject to management approval where cost/latency tradeoffs can be considered. Sizing can later be readjusted as the situation permits.

Across all of these options, Snowflake has released a slew of features to monitor costs and prevent overruns. Budgets, generally available as of last week, enable teams to set soft and hard limits to track, notify and adjust resources as needed to stay within preset budget guardrails. SOC teams can receive early warning when they are on track to blow through their weekly or monthly budgets and adjust whatever changes have taken their spending off course.

Myth: Low-Latency Threat Detection and Usage-Based Pricing Don’t Mix

Another concern I’ve heard from SOC leaders considering a move to usage-based pricing for their data platform is around detection latency. We need to run threat detection rules around the clock, they explain, and that would mean non-stop usage. Wouldn’t that break the usage-based pricing model?

Not necessarily. This is where “separation of compute from compute” comes into play. We can take advantage of Snowflake’s support for querying a database with multiple warehouses in a single account. There’s no limit to how many warehouses can be created, and each only incurs cost when it is active.

Security operations on the data lake can therefore divide its compute requirements across several warehouses. A small, low-cost warehouse can be running 24x7 against recent data for timely threat detection. In parallel, a much more powerful one can be standing by for ad hoc investigations. When a human analyst needs to search log data going back weeks or months, the large warehouse can return answers quickly and suspend itself within minutes of inuse.

Having multiple compute clusters available, each with its own balance of cost and performance, enables security operations with low-latency use cases to benefit from usage-based pricing. Snowflake’s product direction towards services that aren’t tied to running warehouses, like Snowpipe for ingest and serverless tasks for transformations, means ever-increasing granularity aligning each use case’s cost to its latency requirements.

Myth: The Benefits Aren’t Worth the Added Complexity

While leading cloud data platforms enable security leaders to save millions on detection and response, adoption has been gradual. Fears around the complexity of usage-based cost models may have played a role in that. The reality is that managing usage-based costs is easier than ever, but you don’t need to take my word for it.

One benefit of the “pay for what you use” model is how it supports experimentation. A team can try a usage-based platform without any up-front commitment and get first-hand experience with sizing resources, per-second metering, and cost guardrails. With Snowflake, for example, an account can be provisioned with a maximum budget of a few hundred dollars. The team can then freely experiment with querying data using warehouses of different sizes and know in advance exactly how much the experiment will cost. (Side note: you can get a good amount of free credits for testing out a security data lake but you didn’t hear it from me. I don’t even work there anymore.)

Every data platform has a fixed relationship between cost and performance. For most SIEMs, that relationship is hidden from users. The vendor charges you for ingest or licenses based on how much search activity they expect you to perform, and they keep infrastructure costs in check by throttling your speed. Even with SIEMs that offer some level of control over your deployment’s compute power, it’s usually so cumbersome to scale up and down that the SOC is compelled to operate at a fixed level of compute power.

Usage-based pricing blasts this model apart by removing the SIEM vendor’s “safe” assumptions for what the SOC will need to do its job. Customers get to align their spend to actual requirements, and in cybersecurity this translates to substantial savings when unused resources don’t incur costs. It also fosters better customer support, as Snowflake CRO Chris Degnan explained in Consumption-based Pricing: Ensuring Every Customer’s Value and Success, “Your account team becomes your day-to-day advocate and an integral part of your project team, working hard to earn your business every day.”

Perhaps most importantly, it removes the hard caps on performance that otherwise create search-related coffee breaks. When the SOC goes to Defcon 1 (that’s nuclear war, not the event in Vegas), the option to temporarily 10x search performance is something that even hard-nosed CFOs can get behind.

For all of these reasons, the move from fixed or ingest-based SIEM pricing to a usage-based model should be considered a sign of SOC maturity. While most SIEMs don’t yet support usage-based data platform options, some do. Security leaders should bear in mind that some preparation and best practices are needed for such a shift. But the flexibility to pay for the time and performance your team needs can do wonders for your budget while helping to detect and respond to threats when seconds count.

Data Collection Wants to Be Free

Big changes are happening to how security teams get their data. For years, data collection was a function of the SIEM. Splunk’s app store, for example, includes hundreds of supported connectors that integrate with everything from firewalls to vulnerability scanners. In parallel, large SOCs formed teams to manage open-source pipelines for shaping and routing data using technologies like Apache Kafka and NiFi.

Security organizations have now reached a tipping point for their data. On one side are the three V’s of data explosion: volume, velocity, and variety. Security teams are dealing with an avalanche of logs from endpoint agents, multi-cloud hybrid infrastructure, distributed workforces, and SaaS applications. Information must be collected from within the environment and from outside via APIs. It’s also more valuable than ever, with advances in AI enabling new insights on risks and threats. Add “value” as a fourth V driving the need for data pipeline investment.

Subscribe now

On the other side, alternative destinations have opened up. Cheap cloud storage, subsidized cloud provider SIEMs, and powerful cloud data platforms could be better options than the monolithic SIEM for some datasets. The resulting dynamic has been described as The Great Splunkbundling, where previously integrated components are freed from the SIEM— starting with the data pipeline.

Cribl has dominated this space with its Cribl Stream product. More powerful and cost-effective than Splunk’s own solution, Cribl Stream gained widespread adoption by maintaining compatibility while opening up a range of sources and destinations. Investors value Cribl at over $2 billion because they know that security teams (as well as IT, DevOps, and other log consumers) value optionality and specialization.

As security operations embrace commercial off-the-shelf data collection solutions, more providers are entering the scene. Some, such as Databahn, tout their cybersecurity expertise and AI use cases. Others, such as Dassana, Monad, and DataBee emphasize data sources beyond logs- including vulnerability, identity, and asset datasets that can support compliance analytics and security metrics. These newcomers identify as security data fabric solutions.

Investors and industry heavyweights have poured hundreds of millions of dollars into this emerging category. Zscaler’s Avalor acquisition and SentinelOne’s investment in Auguria are examples from last month. Cole Grolmus captured the activity in a post that drew plenty of comments- not least from security data fabric companies that didn’t get included in his cool diagram:

But how should security leaders evaluate newcomers like Tarsal (“One click to build your security data lake”)? And what is the identity crisis that threatens to crash the security data fabric party?

Demystifying Security Data Fabric

Gartner defines data fabric (generally, not just for security) as “an emerging data management design for attaining flexible, reusable and augmented data integration pipelines, services and semantics.” The top job for data fabric is making data available in the right places, in the right formats, and reliably. It sounds highly relevant for security operations! But where does it fit into the existing SOC stack?

As Pramod Gosavi points out, security data fabric is not a security data lake. Gosavi writes that it “ingests data from multiple feeds, then aggregates & and compresses, standardizes, enriches, correlates, and normalizes that data before transferring a full-time-series dataset to a security data lake.” In other words, a pipeline that connects the multitude of logs, findings, and contextual datasets to several destinations. With many enterprises juggling multiple SIEMs and data lakes, that “many to many” fabric must address substantial complexity.

For large enterprises, many sources are located within the organization— in the data center or private cloud. These require the kind of forwarder infrastructure that Cribl has perfected and most security data fabrics do not offer. Enterprise log sources such as network flows and operating system events are very high volume, so the data reduction capabilities of Cribl are another big advantage.

Where the new security data fabric solutions may have a role to play is in collecting data from behind APIs. SaaS security tools like Okta, Wiz, and Axonius have important pieces of the overall SOC puzzle. When moving away from bundled SIEM connectors, security leaders can turn to security data fabric solutions for collecting, cleaning, and enriching cloud-based sources. As in the Avalor example below, these tend to describe posture, assets, and identities.

The sources supported by different security data fabric solutions form one of the considerations for selection and implementation. The bigger challenge comes from what happens after the data is available.

Diagnosing the Identity Crisis

The original consumers of data fabrics and Extract-Transform-Load (ETL) products are data engineers and analysts. The trouble with applying these approaches to cybersecurity is that it’s an insular field with lots of specialized knowledge and not enough data skills. Fivetran can build a big business connecting Salesforce to Snowflake for customer relationship management. Its users have been using previous iterations of the same stack for decades. The cybersecurity equivalents of Fivetran face a customer base that’s often new to SQL, BI, and data science.

The result is an identity crisis where data collection products feel compelled to provide “last mile” analytics use cases. Dassana, for example, talks about “Revolutionizing security data ETL” on one page, while its solution includes an app for vulnerability management and another for security KPIs. These are great use cases to address, but they beg the question: can one startup solve the thorny issues of “many to many” data connectivity while also solving risk-based finding prioritization and scoring the SOC?

The answer may depend on the customer. Smaller security teams may prefer a security data fabric that includes connectors and prebuilt analytics. Larger security organizations, on the other hand, should strive to get as many integrations as they can off the shelf while developing their people’s data analytics capabilities.

Building and maintaining API connectors is a slog that is best to outsource— and no longer requires SIEM lock-in. But “one-click to build your security data lake” still requires you to consider how that data will be put to work for threat detection, risk management, and executive reporting.

Putting Security Data Fabric to Work

The value security data fabric delivers to an organization depends on the consumer side. Security leaders deploying a fabric or security ETL solution should thoroughly plan in advance for the people, processes, and tooling that will use the data.

One of the big differences between a security data fabric approach and traditional data tiering is that all the destinations should be usable. Data in the fabric is routed to where it can best be used rather than just stored for future rehydration. This means that the people involved in those use cases need broadly applicable analytics skills. A security data fabric architecture turns proprietary languages into a liability, as they would only be useful for one of the destinations in the fabric. SQL (the S stands for Standard) and Python are increasingly valuable for SOCs because they are supported across a broad range of security products and adjacent tooling, such as Jupyter Notebooks and PowerBI.

Consumer-side processes should also be considered early in the security data fabric initiative. With data from many sources being analyzed at many destinations, onboarding processes should ensure a consistent mapping to data models with standard schemas. Quality assurance, aging out old data, and iteratively reviewing which analytics platforms are used for each use case are all valuable processes to consider.

Finally, on the tooling side, security leaders should opt for analytics solutions that plug into the stack without breaking it. Since most security products were not designed to work with a data fabric, this is easier said than done. From threat detection platforms to compliance automation, analytics products have traditionally been built to connect directly to individual sources or the data pipeline. This is convenient for smaller organizations that haven’t yet rolled out extensive connectors. For larger SOCs, however, the “inline” approach conflicts with their existing data flows. Having to ship data out to a third-party analytics solution only to bring it back into various storage and analytics platforms impacts cost, risk, and flexibility.

The “downstream” approach is preferable for enterprise security teams that have invested in their pipelines and want to avoid sensitive data leaving their perimeter. With downstream analytics, a vendor’s solution can plug into the customer’s data platforms for analytics. Not all products support this approach due to dependencies on streaming analytics and other design considerations. But products like CyberSaint for risk management and Anvilogic for SIEM that can work “downstream” from the customer’s data platforms are better aligned with security data fabric success.

Take these considerations into account when planning your security data fabric initiative. No single tool delivers successful security analytics across many sources connected to many destinations. But a combination of proven platforms and an exciting batch of well-funded innovators is set to unlock dramatic gains for data-driven security organizations.

A common misconception of the MITRE ATT&CK framework is that it’s a cybersecurity version of Bingo. A game where you score points by covering columns and win by covering the whole board. This approach causes frustration for analysts who find themselves working on noisy alerts triggered by pointless rules. It also leads to wasted budgets and, worst of all, a false sense of security. But is there any other way to measure detection progress? And does ATT&CK still offer value to detection engineers and SOC leaders?

Subscribe now

Mapping Tactics and Techniques

By now, most SOC practitioners are familiar with the table shown below. Each column represents a “Tactic” within the MITRE ATT&CK framework. These roughly correspond to Cyber Kill Chain stages, each being the threat actor’s steps towards their objective. The chain metaphor is helpful for defenders in the SOC, whose job is to break the chain and disrupt the attack before it can impact the business.

Naturally, security leadership wants to ensure that detections are deployed to spot attacks as early as possible. Each square in the matrix represents a technique attackers use at that stage. For example, an attacker may compromise a VPN service through stolen credentials or exploit a software vulnerability in the service itself.

The SOC leader may ask to review a heatmap like the above to confirm that detection coverage is in place. They may hear back from their team: “Good news, we’re covered for T1078: Valid Accounts.” In doing so, however, they risk turning a valuable tool into a high-risk Bingo game.

Mounting Concerns on ATT&CK Misuse

Leading infosec voice Mick Douglas recently suggested calling out cybersecurity products that advertise “100% MITRE ATT&CK coverage”:

This is the latest in a growing chorus of concerns about how security teams use the framework. Douglas pointed to alert fatigue as a major issue, while others like Forrester’s Josh Zelonis have pointed out the potential false sense of security. Zelonis described how “techniques such as Process Injection (T1055) have multiple methods of performing them that you would have to exhaustively research and test against.” In other words, your coverage heatmap might show that you covered a technique, but you would still miss an attacker executing the technique in a different way or on a system different from the “covering” detection expected.

The experts are right to raise the alarm. Marketing departments at leading cybersecurity vendors continue to create the impression that the ATT&CK framework is something you should try to cover. Google for “mitre attack coverage” and you’ll get ads like this one:

To avoid falling into this trap, switch from using ATT&CK as a goalpost to using it as a foundation for your detection engineering lifecycle.

ATT&CK for Threat Modeling

Instead of playing ATT&CK matrix bingo, use ATT&CK as a knowledgebase and common language for developing and prioritizing detection requirements. Keep in mind that ATT&CK was intended to represent the ways in which the bad guys operate. As such, it can help in every step along the Detection Responsibility Handshake.

First, ATT&CK gives us a way to talk about the threats facing the organization. If we’re worried about becoming the next UnitedHealth Group ($22 million in ransom, government investigation incoming), we might refer to the CISA advisory for the threat actor behind the breach. Five techniques associated with ALPHV Blackcat are laid out in the report:

These are critical components of the initial threat analysis performed in threat modeling.

Across these techniques, we refer to MITRE ATT&CK for associated data sources and components. Our objective at this stage is to understand the relevant systems and datasets. For example, knowing that the Blackcat group uses Kerberos token generation for domain access, we refer to technique ID T1558 and read up on the platforms affected by this technique: Linux, Windows, and macOS. We then apply knowledge of our environment and its vulnerabilities: maybe we’re a Windows shop, and some of our servers run an older version of the OS, which makes it easy to steal Kerberos tickets.

Looking up the threat actor’s technique on MITRE.org also yields guidance on what logs we need to collect. Notice that in the descriptions below, the idea of “coverage” gets very fuzzy. We can log “Service Ticket Operations” to the SIEM but only incrementally build detections to spot “irregular patterns of activity.” This one aspect of detecting the technique can take numerous detection engineering efforts.

Reviewing the rich details provided by MITRE, CISA, and independent researchers can show us what threat actors like Blackcat would likely try in our environment. Weighing the risk factors for our environment, we can design the detection requirements for the SOC to address. The team can then demonstrate coverage for its prioritized threat scenarios rather than the techniques of the ATT&CK framework.

ATT&CK for Building Detections

Knowing that ATT&CK coverage is not a goal to pursue doesn’t mean your detection engineering velocity doesn’t matter. On the contrary, the numerous ways attackers can operate in your environment should translate to greater urgency. As the SOC implements detections for the prioritized threat scenarios, the ATT&CK framework provides several opportunities to speed up detection engineering.

The first way to knock out detections faster is to get them from a detection content provider. Attackers reuse techniques across their victims and common targets, from Windows to AWS and Okta. Understanding your threat detection requirements regarding the MITRE technique links you to the relevant content in threat detection libraries like Anvilogic’s Armory. The fastest rule to build is one that someone else has already built.

Using ATT&CK as part of detection engineering is also important for efficiency at scale. As your team reuses detection components across tens or hundreds of multi-stage scenario detections in production, they need a systematic way to keep up with new components being created. For example, a detection scenario might look for a malware alert followed by indications of Windows autostart manipulation. This is a great way to spot an attacker that evaded endpoint protection. As your team adds additional rules for spotting this persistence technique, ATT&CK can link the new components to the detection scenarios that use them.

The example above defines a threat scenario where autostart manipulation follows an endpoint attack. There is no end to the ways in which an attacker can abuse autostart features in Windows. That’s why “coverage” for this persistence technique is counter-productive while using it to automatically pull in relevant new rules is helpful.

Making demonstrable progress toward covering your prioritized threat scenarios is a worthy outcome. MITRE ATT&CK can help with achieving that progress. And if you can’t cover the board and win the game, maybe that’s a sign that you’re doing real SOC work.

I’m no data scientist, but I know machine learning can be a SOC’s best friend. In a previous role, my security engineering team had to convince our management (and ourselves) that we would get alerted if the Capital One hacker tried the same attack with us. Machine learning for behavior analytics gave us a way to detect a compromised AWS account trying to steal documents from our cloud. And now, prebuilt machine learning functions in Snowflake cut the effort involved from weeks to hours. And you don’t need to be a data scientist to use them.

Subscribe now

Could Capital One Have Detected the 2019 Breach?

The hacker “erratic” behind one of the largest breaches of all time was not stealthy. She bragged about copying over 30 GB of documents out of 700 cloud storage buckets. Despite all that activity, the breach stayed undetected from March to July. When we read the breach reports, my team knew we needed detection time that would not be measured in months.

Could behavior analytics help detect this kind of cloud breach? We considered how a compromised account would act differently from how it typically behaved. This is where ML can play a role by learning what is normal for users, like how many documents they download in an hour. Then, a detection rule could compare recent activity to this baseline and flag suspicious behavior. Since “erratic” copied out tens of thousands of documents, she would likely have tripped this alarm early in the exfiltration phase.

Baselining Your Cloud with ML Functions

My team spent a long time building the machine learning code for our UEBA baselining project. We needed a data scientist with skills in R and Jupyter to plug into the data lake and move data back and forth. Luckily for security operations teams everywhere, Snowflake now provides fully automated built-in machine learning functions as a shortcut to UEBA.

Snowflake is a cloud data platform with cheap storage and a compute engine that scales up and down to run fast detections and investigations. That’s made it a popular option for security teams looking to augment their SIEM. Like everything in Snowflake, the new machine learning functions are delivered as a fully managed service. Snowflake’s Anomaly Detection function is described as follows:

Anomaly detection allows you to detect outliers in your time series data by using a machine learning algorithm. You use CREATE SNOWFLAKE.ML.ANOMALY_DETECTION to create and train the detection model, and then use the <name>!DETECT_ANOMALIES method to detect anomalies.

Helpful examples are given in the official docs here and in blog posts like No Data Science Team? No Problem which aren’t just a good starting point- they’re evidence that data science is getting democratized. This is all great news for the SOC.

You'll need relevant log data to train the anomaly detection service in your Snowflake account. AWS can generate CloudTrail logs for S3 file downloads, but these “GetObject” events get so chatty in production that many security teams haven’t been able to collect them. This is where the cheap and limitless storage of the data lake comes in. As I showed in a previous post, Snowflake is so cost-effective for security data that you can afford to bring in datasets that would otherwise never get collected- and now you can use that data for behavior analytics.

The result of baselining file downloads for your AWS users is a trained model representing what is normal for each one. You can also extend the ML function to baseline users together with additional fields, such as the user’s department. In this way, your model learns not just from activities but also from tags that represent the user’s context.

No installation, maintenance, or tuning is required to run the model training function. The same goes for the function that compares new data to the existing model. It’s just a simple command of DETECT_ANOMALIES to get back a result like this for each user:

This example shows user123 downloaded 150 files from the organization’s S3 buckets in the last hour. The ML model had forecast 50 downloads for this user and set 70 as an upper bound for what should be considered an anomaly. The bounds of the “normal” range are determined automatically by the model during training and may be recalculated when you retrain the model. Since user123 is way out of the normal range, the function returns TRUE. This is Snowflake saying that the user is acting very fishy.

From Anomaly to Threat Detection

To complete our UEBA project, my team needed a way to pull our homegrown anomaly detection model into our detection engineering process. At the time, this involved our wonky but lovable scripts and container clusters galore. This is still the way many enterprise SOCs use data lakes today. But Snowflake and its cybersecurity ecosystem have made progress that changes the game.

One option mentioned in the documentation linked above is the new Snowflake Alerts service. You can use it as a fully managed way to run the DETECT_ANOMALIES command on a schedule and receive an email containing the results. This has the advantage of not requiring an external solution but doesn’t provide a framework for developing and managing detections.

Another option is to use a SIEM solution that can serve as a security wrapper for Snowflake. I joined Anvilogic because security teams at big companies were choosing it as their way to turn Snowflake into a security data lake. Now, with its anomaly detection functions, Snowflake gives security teams more than just cost savings and performance at scale. Any threat scenario you create in Anvilogic can call the DETECT_ANOMALIES function as part of its detection logic. The function’s output can trigger an alert or serve as one indicator in a sequence of events for the SOC. Anomaly detection built into the data lake gives security teams without data science expertise an easy shortcut to ML-powered threat detection.

It took decades for the security community to get the upper hand against attacks on endpoints, and cloud environments pose an even bigger detection challenge. Where endpoint attacks often involve malware to package attacker capabilities, cloud attacks succeed through seemingly legitimate requests to the cloud provider. In the hustle and bustle of an enterprise cloud environment, atomic (one-dimensional) detections face a harsh tradeoff between low accuracy and overwhelming noise. Correlating between cloud and non-cloud activity can provide fidelity for effective detection coverage.

Subscribe now

Most Cloud Detections Don’t Work

A recent study of cloud threat detection by Invictus IR should be a wake-up call for any SOC with a cloud footprint to secure. The researchers ran 32 well-known attack techniques in AWS, each leaving traces in the CloudTrail logs. Despite having access to evidence of the attack, the researchers found that “GuardDuty only triggered for just three of the simulated attacks, which is very limited. Especially because these attacks are all well-known.”

The built-in AWS threat detections were a no-show. Sigma, the leading open-source project for threat detection rules, also failed the test, with barely a third of attacks being detected. You can see how few of the well-known attack techniques were identified in the table below.

But wait! The results reflect only the disturbing proportion of false negative results (93% and 66% respectively). In a real deployment where detections feed a SOC’s alert backlog, detection rules must also demonstrate a low level of false positive results. Would the Sigma rules triggered in the study fire excessively in the real world?

On review, I believe many of the study’s successful detections would not have been enabled in a production environment. For example, IAM Backdoor Users Keys accounted for three of the eleven hits in the lab. This detection rule is designed to identify when an AWS API key is created for one user by another, which could indicate a backdoor being set up for persistent access to the AWS environment. The logic filters out cases where users create keys for their own account, aiming to reduce false positives. However, there are many legitimate cases in a real environment that meet this criteria- including developers and admins going about their daily duties.

Two successful detections were attributed to the rule AWS STS Assumerole Misuse, but the name is misleading. The rule triggers on the use of AssumeRole by an entity with the AssumedRole identity type, where the session issuer is also a role. The documentation warns that this activity is not necessarily misuse:

falsepositives:

AssumeRole may be done by a system or network administrator. Verify whether the user identity, user agent, and/or hostname should be making changes in your environment.

AssumeRole from unfamiliar users or hosts should be investigated. If known behavior is causing false positives, it can be exempted from the rule.

That sounds like a lot of work for the triage team. So much extra work that they’d probably shut the rule off entirely. How can a detection engineer reduce the noisiness so that rules stay enabled and ready to detect cloud attacks? And what about spotting attack techniques that seem to blend in with the normal activity of the cloud environment?

Bringing in Another Dimension

Like a detective collecting evidence, cloud threat detection should include more than one log source. An attack that is easy to miss when looked at only from the cloud perspective can become obvious when looking across cloud, endpoint, identity, and other types of activity. This turns the cloud’s unique properties (API-driven, all activity associated with a user) to the defender’s advantage.

Detection engineers can still start with basic detections like those from the study above. To improve their fidelity so that they become useful in a real-world environment, the rules can be expanded to account for the involved users, assets, and networks. Weak signals that would otherwise be ignored can amplify the detection such that when it alerts- the SOC does take notice.

Design decisions and tooling selection determine how easy it will be to create these high-fidelity detections. Because logs from different sources may arrive at different times, a log repository that depends on combining or enriching data at load time may struggle to support multi-dimensional detections. Data platforms that are good at join operations, on the other hand, make it easy to combine signals from different sources at query time. This is an area where the Splunk Effect pushes analysts to search for standalone indicators, while SQL-based analytics encourage analysts to combine dimensions like Lego blocks.

Getting Started

Building multi-dimensional detections is not rocket science. SOC analysts can start with threat modeling as described in The Detection Responsibility Handshake and consider the different areas where an attack would likely leave traces. Combining these areas into one detection can achieve the fidelity required for use in prod.

Once you have in mind the two or more activity types you want to combine, you’ll need to determine the field that they have in common. This is known as the join or pivot field. For example, to combine cloud and endpoint activity, you may join on the email address associated with the cloud user and the endpoint agent. Sometimes, this field is present in each of the logs, and sometimes, a lookup table is required. That table would be kept up to date with a mapping of computer names to email addresses, for example, based on data pulled from a CMDB.

The following three screenshots demonstrate a multi-dimensional detection implemented in SQL. This detection is designed to catch an attacker in the recon stage, early in the kill chain. A detection engineer may start by looking at AWS CloudTrail, where a bunch of enumeration events have taken place:

However, this logic on its own would be too noisy in production. Many legitimate use cases and tools run “describe” and “list” commands across AWS resources. Rather than giving up on detecting reconnaissance in the cloud, the detection engineer can add another dimension based on their threat modeling. They might reason that an attacker who compromises a cloud admin’s laptop would do a few things once they’ve established a foothold.

One of the attacker’s moves would be to enumerate the cloud resources that the compromised admin can access. Another would be to scan the local network to identify vulnerable hosts that would be more persistent and reliable than a laptop, such as local file servers or network devices. These scans could be identified by looking at network connections recorded by the EDR agent.

A user performing both scans around the same time is much less likely to happen during normal operations. To combine the two signals into one high-fidelity alert, the detection engineer would join the two logic blocks above into one query that can run against the aggregated datasets. In this example, a helper function resolves the user’s identity based on their hostname or cloud resource identifier. The function referenced below would reference the lookup table described above, where CMDB data is regularly pulled in to map hosts to users or cloud user accounts to email addresses.

The query above combines the two previous queries into multi-dimensional detection logic. A platform like Anvilogic that supports SQL JOIN operations could be used to operationalize these rules by automatically adding the necessary time windows, scheduled execution, and version control.

More Examples

Many cloud detections look for actions that are easy for attackers to bypass, such as root user activity in production, or logging being disabled. Multi-dimensional detections can help with more realistic attack techniques, such as stolen credentials used to tamper with a cloud environment. What might that look like?

An attacker that steals a user’s access token from a public code repository could limit their actions to blend in with the cloud environment, but they would need to use it from their own systems. A multi-dimensional detection could check for activity in the cloud control plane that doesn’t align with the corresponding activity on the user’s endpoint. The lack of network activity for a user supposedly interacting with the AWS console, for example, could be a dead giveaway.

A similar approach could amplify weak signals from a security agent on the endpoint. If a laptop or server triggers some low-severity endpoint detections, the SOC would likely not be notified. A report from Microsoft’s IR team describes how threat actors often deploy malware as a stepping stone toward more sensitive environments:

Tokens can also be stolen from endpoint devices themselves via credential-stealing malware. Microsoft IR has been engaged with organizations where credential-stealing malware was installed on an administrator’s endpoint device via an initial phishing email. While the admin used separate accounts for their day-to-day and administrative work, the Global Administrator had signed into both accounts from the same device. The malware had the capability to extract all the credentials and tokens on the device, eventually leading to tenant-level compromise.

Tokens on endpoints are typically stored as cookies, and theft can occur in several ways. Commodity malware such as Emotet, Redline, IcedID, and others have the capability to steal both credentials and tokens. Pirated or cracked software often has token and cookie stealing malware embedded within it as well.

Weak signals around malware activity or pirated software could serve as one detection dimension. When they happen in conjunction with cloud account role assumptions or privilege escalations for the host or associated users, a high-severity alert could be triggered for the SOC to investigate.

By all accounts, threat actors have become much more comfortable targeting cloud infrastructure. It’s now on security operations teams to keep up on the detection side. But as we saw in the Invictus report, relying on alerts from the cloud service provider is risky, and community-based content for cloud detection remains basic and noisy. Effective cloud threat detection takes a more advanced approach—with multi-dimensional threat detection providing an opportunity for defenders to gain the upper hand against attackers in the cloud.

The data revolution has transformed every facet of modern enterprises, unleashing productivity and insights across each department with one notable exception. In cybersecurity, a generation of analysts finds itself years behind its peers, tethered to an approach that is powerless to tap into the biggest thing in tech since the World Wide Web. Much of the blame for this predicament falls on Splunk—the pervasive platform that inadvertently shaped security operations into a silo of search-driven rather than analysis-driven methodologies. Understanding how we got here can help us rejoin the rest of the enterprise and hopefully not miss out on the promise of data and AI.

Subscribe now

Like Google for Your Logs

In 2006, ComputerWorld published an article titled “Splunk Inc.'s Splunk Data Center Search Party” that described the hit new product’s key to success.

Splunk's sweet spot is knowledgeable IT experts who have a good idea of what they are looking for but are having difficulty finding it in the haystack of error logs and application dumps from a myriad of different servers.

Like Google, it automatically indexes everything, but its true power is unleashed when an experienced searcher is looking for something specific.

With Splunk, a network admin troubleshooting a connectivity issue could search the index for firewall error codes associated with dropped packets. Searches would often be successful because the relevant error codes were well-known. IT professionals fell in love with a product that could quickly tell them where to find the needle in the haystack and fix the outage.

The Effect on Security Operations

While Splunk started as an observability solution, security teams also began adopting it. Then-dominant SIEM products like ArcSight had been designed to run on powerful but rigid databases like Oracle. The downside of this approach was that the solutions were cumbersome and required lots of upfront work to prepare the data. Splunk’s flexibility and the fact that it already contained relevant log data at many enterprises led security teams to pull Splunk onto their toolbench.

Splunk responded to this demand by launching Enterprise Security with rule sets and dashboards designed for SOC use cases. Cybersecurity eventually came to account for over half of Splunk’s sales, while the product itself came to define the SIEM category. But the platform remained a “Google for Logs” at heart.

As a result, security operations methodologies became shaped around Splunk’s product capabilities. Many SOC practitioners today view these as equivalent:

• Search == Analytics: Splunk's design as a log search engine has led analysts to prioritize search queries over deeper data analytics. This focus on looking up data has constrained the development of analytical competencies among security professionals.

• Logs == Data: The emphasis on logs as the primary source of security insights has narrowed analysts' perspectives. Detection logic often fails to see data as a rich, multifaceted resource that extends beyond event logs to user context, asset properties, trends, and baselines.

• Enrichment == Contextualization: Index-based search engines deal with enrichment on load much better than they do joining datasets at query time. This has fostered a preference for adding superficial and often outdated properties (enrichment) rather than integrating and understanding security events' broader and up-to-date context (contextualization).

• Leaderboards == Metrics: The search platform's orientation towards displaying data in leaderboards (“Top 10 Attacker IP Addresses”) overshadowed meaningful metrics aligned with security operations' strategic goals. In other fields, metrics are often calculated periodically and recorded in dedicated tables- a kind of batch processing for which data warehouses are well suited.

• UEBA == Data Science: Dedicated User and Entity Behavior Analytics (UEBA) features introduced basic behavioral analytics to many security operations. However, this has often come at the expense of adopting more comprehensive data science methodologies that could offer deeper insights and predictive capabilities tailored to the organization.

The root of the problem is that cybersecurity is fundamentally not a search problem. Some approaches to threat detection do involve looking for known string values, such as for a user agent string in the crypto miner alert shown below.

This approach, however, is brittle in that an attacker can easily modify their user agent string and bypass the rule. Rules that search for event names associated with an attack, such as bucket enumeration, are prohibitively noisy in an enterprise environment. Cybersecurity is adversarial; anything the defender can easily search for is the first thing the attacker would change. It's very different from troubleshooting a network connection issue!

The ineffectiveness of applied search methodologies was described in David Bianco’s The Pyramid of Pain in 2013. He observed that while the good guys were winning some battles against cybercriminals, “seeing how these indicators were being applied…almost no one is using them effectively.” Security analysts took hashes and IP addresses from threat reports and googled them in their logs. Avoiding these detections was a piece of cake for the bad guys.

The pyramid of pain teaches us that adversaries will dodge the only indicators you can search for and the only indicators that matter you can’t search for.

The problem for our field is that defenders have been trained to go into battle with search engine technology designed to look up log events containing strings. Those strings range from trivial to simple for an attacker to switch and avoid detection- but security analysts have been conditioned to work within these constraints. As such, many of the “next-gen” SIEM solutions, such as Google Chronicle, address some of Splunk’s scale challenges while perpetuating the SOC’s dependence on search. The rule below (from the public Chronicle GitHub account) demonstrates the brittle nature of search-based detection rules still found in the latest generation of SIEM technology.

rule ryuk_ransomware_detector_sysmon_behavior {
 meta:
    author = "Lee Archinal"
    description = "This detects characteristics of the Ryuk Ransomware strain of malware  License: https://github.com/Neo23x0/sigma/blob/master/LICENSE.Detection.Rules.md."
    reference = "https://tdm.socprime.com/tdm/info/vZQdVgPbH0b7"
    version = "0.01"
    created = "2019/07/15"
    product = "windows"
    service = "sysmon"
    mitre = "impact, t1486, execution, t1204"

  events:
($selection1.metadata.product_event_type = "11" and (re.regex($selection1.target.file.full_path, `.*\.ryk`) or $selection1.target.file.full_path = "RyukReadMe.html"))

  condition:
    $selection1
}

The SOC Can Catch Up

Contrast the search rule example above with how the rest of the enterprise derives insights from data. Departments that “1337 h4x0rs” might consider less technically proficient, such as sales, finance, and marketing, are years ahead and significantly more successful at achieving their objectives. Without detouring into the questionable incentives of the security operation (quis custodiet ipsos custodes?), we can just look at how an enterprise sales organization answers its data questions, for example, how it predicts which of its customers is most likely to jump ship. Such “churn analysis” can trigger a friendly phone call, special discounts, or other attempts to avoid loss.

In this Snowflake Quickstart example, the customer has ten dimensions that can be used to evaluate churn risk. Some may be associated with activity, such as calling the customer service center, while others wouldn’t appear in the log data.

These inputs are crunched in a data science model that leverages a Random Forest classifier trained on historical data within an analytics platform. The sales team would collaborate with data scientists to develop a data app that can be used in daily sales operations, even by less data-savvy salespeople.

No one would suggest that a sales team try to “search” for churn-risk customers or that finance would “search” for budget inefficiencies. But in security operations, facing the double challenge of finding someone actively trying to avoid detection, search is the status quo. Splunk’s success as a SIEM bears much of the blame.

Meanwhile, a new class of cloud data platforms blends data warehouse, data lake, and search workloads to address the shortcomings of past platforms like Oracle and Hadoop. Snowflake, BigQuery, and others have added native JSON support with schema-on-read, fast search, and streaming analytics. All are delivered as a service. The limitations and overhead that drove the SOC away from Oracle and Hadoop to NoSQL alternatives like Splunk and Elasticsearch have been eliminated.

To ditch its over-dependence on search, an enterprise SOC can start by leveraging the cloud data platform that other departments at the company are already using. In this way, security analysts can access production tooling (e.g., data pipeline and data warehouse), functioning processes (e.g., role management and cost monitors), and people who are experts at the data stuff. Working with the central data science team is a great strategy for security leaders navigating their organization toward analytics best practices.

Also helpful is the growing ecosystem of security products that plug into cloud data platforms with support for the full spectrum of analytics- including search, statistical baselining, and data science. Anvilogic, for example, recently demonstrated how general-purpose data science methodologies can make a big difference in effective threat detection at scale.

A narrow fixation on search has held back security operations. The historical reasons for why this happened are no longer binding, but many SOCs haven’t yet adjusted. To move forward, security leaders can recognize the implications of Splunk's historical influence on security operations and embrace the transformative potential of security data lakes as the path to joining the rest of the enterprise in analytics success.

Endpoint Vendors Stock Up on SIEM

Let’s unpack the trends pushing SOCs to rely on multiple SIEMs. We’ll see that these powerful currents will likely shape security operations for years to come.

First, what do all the Gartner EDR MQ leaders have in common? Under the banner of XDR, they’ve all started selling SIEM.

• CrowdStrike says we’ve got Next-Gen SIEM and Log Management

• Microsoft will sell you An easy and powerful SIEM solution

• Palo Alto didn’t like that others copied “XDR” so they called their SIEM by a different name and have written that SIEM solutions will adapt and evolve

And the same goes for the rest. None of these vendors are content with selling endpoint agents that feed into Splunk or other dedicated SIEM platforms.

Splunk’s challenges have likely played a role in this industry trend. The recently Ciscoed company controls more of the security event management space than all the endpoint vendors combined. But its estimated $2 billion in annual security revenue is seen by many as up for grabs, waiting to become a line item in a bill from the vendor that already runs on thousands of your endpoints.

Subscribe now

Technological advances have also played a role, as the groundbreaking search performance that previously drew SOC teams to Splunk and Elastic has aged. CrowdStrike paid $400M for Humio, while SentinelOne bought Scalyr for less than $200M. Palo Alto rents big data technology from Google, while Trellix went with Snowflake. As a result, these crossover challengers are able to claim comparable capabilities as pure-play incumbents.

I say “claim” because a healthy dose of skepticism is warranted. Humio had not seen significant enterprise adoption before CrowdStrike stuck its logo on the tin. Splunk is infinitely more proven at large-scale analytics for hunting and threat detection. I believe endpoint vendors hope their search platform is “good enough” to replace SIEM beyond the SME market. Given that the EDR vendors have thousands of agents deployed across every enterprise, they’ll likely get a shot to prove it.

Your Cloud Comes With SIEM

Since every large business and government agency now has a public cloud footprint, another significant force towards multiple SIEMs is that each major CSP has its own SIEM offerings. AWS is still in the early days with its Security Lake offering, but Azure and Google are serious about theirs.

Both have poured billions into related acquisitions and engineering. Microsoft reports that thousands of customers use Sentinel and offer significant subsidies for keeping Azure and Office365 event logs on the platform. As a result, many organizations have chosen to use the CSP’s built-in SIEM, at least as a stopgap measure for analyzing that platform’s logs.

When a SIEM is Just a Side-SIEM

Several factors will keep these new SIEMs from replacing King Splunk at large enterprises. If several of these are present, the XDR or cloud SIEM would take an additive position as a side-SIEM.

• Data also used by other teams: Splunk is often used by teams outside of cybersecurity, including IT and DevOps. Much of the data collected to Splunk may also be used by these teams, which have their own requirements to support observability use cases.

• Other active security use cases: Splunk may be used for security use cases beyond security operations, such as vulnerability management, fraud detection, and regulatory compliance.

• Analytics requirements: Enterprise detection teams perform significant analytics, including correlation, anomaly detection, and behavior modeling. Underlying search technology with limited joining abilities and no data science support will not provide the parity needed for a complete replacement.

• Reporting requirements: Splunk dashboards can be pretty epic.. and many security leaders rely on them for metrics and trend charts.

• Cloud egress: A SIEM that’s only offered in one cloud may impose prohibitive egress costs and compliance issues for data shipped out from other clouds used by the enterprise.

• Integration requirements: Hybrid environments and enterprises with infrastructure deployed over decades have a “long tail” of integration requirements. Switching from Splunk would require significant log collection and normalization development.

• Content requirements: Solutions provided as part of a highly opinionated suite (such as Palo Alto Networks) tend to be optimized for their products. Third-party EDR or CNAPP may receive little or no pre-built content. The effort involved with building and maintaining effective rules and correlations would fall on the individual security team.

Beyond these reasons that Splunk will likely stick around, there’s also the soft side of cyber. Many security analysts are deeply invested in SPL expertise and certifications that would become obsolete if their Splunk were decommissioned.

The Impact on Security Operations

The problems caused by working across multiple SIEM solutions have not received much attention. Network security provider Corelight, which generates oodles of data and knows this problem well, has observed that “defenders have been deploying a secondary SIEM” in a post titled One SIEM is not enough? Anton Chuvakin warned in Living With Multiple SIEMs about the “Complexity and hence fragility of the multi-system setup (due to both data flow integration needs and detection content organization). Complexity kills security.” But the trends described above warrant a deeper analysis of the dangers.

First, it’s harder to detect threats across multiple SIEMs. Attackers don’t limit themselves to one part of the environment; They strike through email to land on the endpoint and pivot to the cloud control plane from where they extract data through the network. A sensor limited to just one of these areas will not reliably identify the attack as it unfolds.

The defenders must also contend with hundreds of detections managed in separate repositories. Detection engineering at scale requires a lifecycle with version control, testing, and repeatable processes to ensure that detections work as expected. This is hard enough within a single platform and is unlikely to succeed across multiple sets of detection rules.

Consider what happens when a new zero-day vulnerability hits the headlines. Detection engineers scramble to ensure that exploitation attempts are identified, contained, and mitigated immediately. Comparing coverage from the various SIEM providers, for example, one in Azure and one in AWS, would require the team to manually confirm that both have covered the relevant services and TTPs.

Consider that each detection and response solution may use a different language with its proprietary rule syntax and format. Teams may be tempted to divide responsibility between the analysts, with some specializing in SIEM A and some in SIEM B. Rule development would progress in separate tracks, with coverage progressing piecemeal. At scale, writing good queries is vital to optimize performance, and this expertise is hard to develop across multiple solutions.

There’s also the challenge of ensuring data quality. Without one central SIEM, the security operation takes on substantial complexity to ensure the expected datasets consistently arrive as expected. This is also true for maintaining data models, as changes to upstream data must be applied in different locations. A unified pipeline, not exclusive to one of the SIEMs, helps here, but data quality can still be affected by varying retention policies and modifications in the different systems.

Incident response is slower and less effective in a spinning swivel-chair, with investigations performed across separate search consoles. Each set of results may represent a portion of the attack and must be manually reconciled into a unified timeline. This becomes exceedingly difficult when an attack spans significant time and assets.

Finally, the all-important goal of automation takes a drastic hit. Each SIEM requires separate integration through a distinct set of APIs running against a unique data model. Playbooks developed for one set of alerts would need to be built from scratch for a parallel environment monitored in a separate SIEM. Attempts to leverage data science and machine learning for automating the routine tasks that grind down the SOC might be supported in one SIEM or another but would not be portable across systems.

Conclusion

Security operations leaders should be mindful of the downsides of using multiple SIEMs. While eager vendors may downplay the difficulties introduced with their side-SIEM, buyers should take time during planning to connect with similar organizations that have deployed the same set of multiple SIEMs. Discussing key concerns should include detection fidelity, rule management complexity, ensuring data quality, investigation effort, and automation success. While industry trends have made adopting multiple detection and response platforms increasingly tempting, potential cost savings must be weighed against the risk and complexity inherent in the “two-headed” approach.

Cost Factor: Data Collection

Snowflake supports loading new data using its Snowpipe service. There are several ingest modes available, each with its own mix of latency, cost, and overhead tradeoffs. Snowpipe auto ingest is the most popular option for consistent intake of machine data. It typically makes new data available within one minute, though latency and cost are affected by the shape of the loaded files. Low-volume “trickle” data or lots of small files can be more expensive to load. The typical cost to load an uncompressed terabyte of log data is around 20 credits or $60 at $3 per credit. That’s a conservative credit cost, many customers have negotiated discounts to less than $3 per credit.

It is worth mentioning that Snowpipe Streaming, the newest ingest mode, provides lower latency and cost for many scenarios—especially for environments with security logs already flowing through Kafka (currently a requirement). Not having to stage files before loading also makes the ingest cost more predictable- data lakes getting better at streaming data is an exciting trend to watch.

There’s also an opportunity to further reduce costs by batch-loading certain datasets that aren’t needed for real-time detections. Loading a high-volume, low-use data source three times a day, for example, instead of streaming it continuously can save big bucks.

Let’s ignore those optimizations for our purposes here. We can estimate costs for an organization using Snowpipe auto ingest to collect 7 TB/d uncompressed log data, with each credit costing $3, as follows:

• Daily Snowflake Credit Count: 7 TB * 20 credits/TB = 140 credits/day.

• Daily Cost in Dollars: 140 credits * $3/credit = $420.

• Annual Cost: $420/day * 365 days = $153,300.

Subscribe now

Cost Factor: Hot Data Retention

In Snowflake, all data is treated as 'hot', meaning that it's always accessible for analysis. This approach simplifies the data management strategy and cost planning. The cost for hot data retention in Snowflake is usually $23 per compressed terabyte per month. That’s the same cost as plain old S3, with the difference being that the data is automatically compressed, encrypted, and ready to query.

It’s easy to trip up your calculations on this, so remember whether you’re budgeting for data in its original state (uncompressed) or storage (compressed). Snowflake has improved its compression over the years, and I typically see higher than 8x compression these days, but to be conservative we can assume a 5x compression factor.

This cost factor needs to take into account the retention period dictated by your security policy. Snowflake doesn’t impose a retention limit, and I’ve seen security data lakes with over five years of hot data. This works because the storage in the data lake is fully separate from the compute clusters. Data that isn’t in scope for a given query is pruned away by the engine and does not affect performance.

There’s also the option to create tasks that drop records based on logic such as source type, date, environment, etc. This enables a retention policy that is tuned to operational requirements and not just a fixed time period. But to keep things simple here, let’s assume that all collected data will be kept for one year.

Another area where it’s easy to make a calculation error is how the data accumulates. In the first year, you start with no data stored and each month more data adds up. Since we consider retention to be one year, from the second year onwards the new data will replace old data and storage will level out. You can be conservative in your calculations by ignoring the lower storage costs of the first year and estimating based on the steady-state amount. In our example scenario, 7 TB/d retained for one year works out to approximately 500 TB of compressed data in storage.

For our 7 TB/d scenario with 365 days retention, we can estimate annual hot data retention costs as:

1. Monthly Hot Storage Cost:

   • Cost per TB: $23

   • Total Storage: 500 TB

   • Monthly Cost = 500 TB * $23/TB = $11,500

2. Annual Hot Storage Cost:

   • Annual Cost = Monthly Cost * 12 months

   • Annual Cost = $11,500/month * 12 = $138,000

Cost Factor: Cold Data Retention

Ironically, Snowflake doesn’t have a cold storage tier. That’s a significant departure from traditional SIEM solutions that often require managing multiple tiers. This eliminates the added cost of cold data storage and, more importantly, prevents the complexity and unpredictability of additional charges for data retrieval or rehydration.

Annual Cold Storage Cost: $0

Cost Factor: Detection Processing

SIEM use cases require Snowflake to run automated queries for threat detection. These detection queries vary widely by the security team’s environment, maturity, and risk profile. Many SOCs only have a handful of rules running against their data lake, and some (like IOC matches) can safely be run once or twice a day. But we can be conservative (meaning surprises bring savings instead of overages) by assuming the security team has deployed several hundred rules and runs them continuously.

Snowflake provides compute power through virtual warehouses that have T-shirt sizes from XS to 6XL. Even the XS is pretty beefy, and a virtual warehouse can be resized in a few seconds as needed. Going up a size brings twice the power at twice the cost, measured by the time the warehouse is powered on.

I polled the forward-deployed engineers that support Anvilogic customers with multi-data platform threat detection. The recommendation they gave me for an organization ingesting 7 TB daily to Snowflake was to budget at that level for a Medium-sized warehouse running 24/7. That’s based on experience with Anvilogic customers, and anyone getting started with detection processing in a data lake should run their own tests to confirm the compute power needed for detection rules in their environment.

To estimate the annual cost for detection processing in Snowflake using an 'M' size warehouse, which costs 4 credits per hour, we need to consider the warehouse's operation over the entire year. Here's how the calculation would look:

1. Hourly Cost of 'M' Size Warehouse:

   • 4 credits per hour.

2. Cost per Credit:

   • Assuming a credit cost of $3.

3. Daily Cost of Running the Warehouse 24/7 for Detection:

   • Daily Cost = Hourly Cost * 24 hours/day

   • Daily Cost = 4 credits/hour * $3/credit * 24 hours/day

   • Daily Cost = $288/day

4. Annual Cost of Detection Processing:

   • Annual Cost = Daily Cost * 365 days

   • Annual Cost = $288/day * 365 days

   • Annual Cost = $105,120.

Cost Factor: Investigation Processing

One of Snowflake’s key benefits to SIEM use cases is how compute resources can be instantly turned on and off. This eliminates wasteful spending that would otherwise be incurred during inactive periods.

The ability to quickly resize compute power, in the form of virtual warehouse T-shirt sizing, is also a source of savings. I’ve seen security operations teams running at relatively low power levels in their day-to-day, pump up their warehouse by 10x when the proverbial stuff hits the fan. That let those teams meet remediation SLAs in crunch time while avoiding high compute spend during normal operations.

Going by guidance from Anvilogic’s experienced security engineers, we can estimate investigation processing spend for investigation with a Medium sized warehouse. A conservative estimate might be 8 hours a day of actively searching the data lake. We’ll include weekends and holidays to build a buffer and account for busy times and breach investigations (which somehow always start on a Friday afternoon of a holiday weekend).

1. Daily Cost of Running the Warehouse for 8 Hours for Investigation:

   • Daily Cost = Hourly Cost * 8 hours/day.

   • Daily Cost = 4 credits/hour * $3/credit * 8 hours/day.

   • Daily Cost = $96/day.

2. Annual Cost of Investigation Processing:

   • Annual Cost = Daily Cost * 365 days.

   • Annual Cost = $96/day * 365 days.

   • Annual Cost = $35,040.

Cost Factor: Archive Processing

Archived data needs to be processed in order to be useful again. As many organizations find out at the most stressful time possible, rehydration from archive to SIEM can be complex and costly. Snowflake eliminates the need for an archive tier as hot data is stored directly in cheap and limitless cloud blob storage. I covered this concept in my post on security data lakes here.

Annual Archive Processing Cost: $0

Cost Factor: Cloud Egress Costs

The big cloud service providers tax data moving out of their network. According to the recent study quoted below, moving 50 TB out of AWS costs around $4,300. That would be the weekly egress cost for our 7 TB/d example organization if they were based in AWS but using SIEM hosted in Azure or GCP.

A significant benefit to using Snowflake for security data is that it is available in all three of the major clouds. So a SOC could potentially use a Snowflake account in AWS for CloudTrail, CrowdStrike, and other sources originating in Amazon services- while using a Snowflake account in Azure for o365, Defender, and other Microsoft services. There would still be a need for correlating threat signals across those environments, and this could be achieved by extracting events of interest without moving the raw data between clouds. Multi-cloud support provides an opportunity for significant savings.

Annual Cloud Egress Cost: $0

Cost Factor: SIEM Solution

Snowflake needs a security layer on top to make it useful for threat detection and response. Some SOCs have built their own rules engine for Snowflake and create threat detection rules in SQL from scratch. There are also well-established SIEMs that can operationalize Snowflake for threat detection and response, including off-the-shelf rules and analytics.

Some of these solutions are charged by data volume, in addition to the Snowflake ingest cost factor. Others have done away with traditional SIEM pricing in favor of feature or asset-based pricing. Ensure that the security layer for your data lake does not become a limiting factor for visibility and effectiveness.

Annual SIEM Solution Cost: Varies by solution.

Conclusion

We’ve now gone through each of the Snowflake cost factors associated with threat detection & response use cases. The Snowflake data platform cost estimate for our example 7 TB/d SIEM with one year of hot retention is approximately $450,000.

Cost factor analysis helps us to understand the ways in which budget constraints may impact our security initiatives with a given platform. We can plot ingest costs at different volume levels to verify that siloed datasets can be brought in to support better detections. We can see whether the team can afford to adopt a full year of hot storage. And we can budget for new use cases like threat hunting, which may require their own cost factor.

Be sure to run your own numbers and validate them in a test environment before committing to any new approach. Hopefully, this walkthrough gave you a better sense of Snowflake’s transformational cost-effectiveness for security use cases and an approach to measuring the potential in your own environment.

Overview

The cost model defines how your SIEM draws from the cybersecurity budget across the security operations lifecycle. The move to the cloud added significant complexity for larger SOCs, making it especially important to enumerate the ways in which you incur hard costs. Your SIEM cost model should include all the factors listed below. You should even track factors that don’t apply to you today, recording them as zero cost and using the complete list for comparing alternative approaches.

1. Data collection

2. Hot retention

3. Cold retention

4. Detection processing

5. Investigation processing

6. Archive processing

7. Cloud egress

8. SIEM solution

Each cost factor will have a way in which the provider meters it and a cost level. One solution can be more cost-effective for a given factor by metering differently (e.g. charging by feature instead of by data volume) or by having a lower cost level (e.g. $20 per TB instead of $50 per TB). Let’s review what goes into each cost factor and consider illustrative examples.

Subscribe now

Cost Factor: Data Collection

What you spend on extracting source data and loading it to your data platform.

Event logs and other security records must be copied from their origin across your network, cloud accounts, or point solution APIs. The initial leg of the data journey is sometimes performed on SIEM infrastructure, such as Splunk forwarders, and in other cases tooling with additional metering, such as Cribl Stream.

Data collection costs are typically measured in terms of annual spend by daily volume (e.g. $900k annual cost for 1.5 TB/d) but some solutions price by ingest amount (e.g. $0.10 per TB). Some simple multiplication can get you to an annualized “TB/d” figure either way. Don’t forget to include costs associated with related services such as Logstash virtual machines, AWS-managed Kafka, or Snowflake’s Snowpipe service.

Cost Factor: Hot Retention

What you spend on storing your security data for active use.

Hot retention refers to the storage of data that is frequently accessed and used for immediate analysis in a SIEM environment. This type of data storage is essential for real-time threat detection and rapid response in security operations. The expectation is that hot data can be used without rehydration processes which introduce delays, management overhead and analytics constraints.

Hot retention costs should be captured in a way that lines up with data collection and accounts for the retention period. For example, $200k annual cost for 7 TB/d retained for 90 days. So while individual records may be retained for only 90 days in hot availability, the cost is represented in annual terms. As the retention period can have a dramatic impact on the effectiveness of threat detection and response, you don’t want to lose this aspect when comparing solutions.

Some organizations have a security policy that mandates high availability (HA) for the SIEM. Consider the level of HA that you require, and if multiple copies of the data will be needed as a result.

Bear in mind that storage costs may depend on how the data is compressed. While data collection figures track how much uncompressed data is generated by the environment, storage costs for some solutions are calculated by compressed data volume. When that’s the case, your annual storage cost estimate should factor in the daily volume of data collected, the compression ratio, and the retention period.

Cost Factor: Cold Retention

What you spend on storing security data that is not fully available for active use.

Many security operations teams rely on cold storage archives to meet compliance requirements within budget constraints. This introduces challenges when archived data is need for investigations, IOC sweeps, threat hunting, data science or metrics. But archiving is often necessary with technologies with expensive hot storage or for organizations with multi-year regulatory retention requirements.

Storage costs for cold retention are calculated the same way as for hot retention. The additional costs involved with moving data to and from the archive are factored separately in “Archive Processing” below.

Cost Factor: Detection Processing

What you spend on automated analytics for threat detection.

The security operations team relies on detection rules to automatically identify events of interest in the environment. Running these detection rules requires computational power (“compute” for short) to analyze collected data against logic or algorithms designed to spot attacks.

In the cloud, compute can take many forms. Some services charge by “bytes scanned”, some use abstract CPU cycles, others use query time. The cost model for compute affect the level and predictability of detection processing costs. While compute costs may have been a secondary concern in the on-prem days of fixed hardware and tightly coupled storage/compute, the game is very different in the cloud.

Costs should be estimated based on the expected quantity and frequency of detection rules in production. Developing heuristics that account for ingest volume, data complexity, rule quantity and analytics complexity is a significant challenge. Machine learning models that continually train on new data also play a role in this cost factor. Security operations teams should work with their SIEM vendor to review the heuristics provided for estimating compute requirements. These should reflect extensive experience with similar organizations and help to avoid resource exhaustion or surprise overages.

Cost Factor: Investigation Processing

What you spend to search during triage, incident response and threat hunting.

Like for running detections, investigations also require compute power to crunch the collected data. Some solutions use different engines (e.g. stream vs batch processing) for detections and investigations, while others will use the same for both. Getting to at least a basic understanding of how the query engine works will help the security organization to estimate how much its investigations will cost.

Consumption-based pricing is common in cloud environments, and security operations teams increasingly need to predict costs based on future consumption. But cybersecurity is intrinsically unpredictable, and any SOC can go through quiet periods, fire drills, and full-blown breach response. As with detection processing, planners should work with their vendors to estimate costs based on heuristics developed over time at organizations with similar profiles. Ample buffers and the natural tendency of things to “even out” will help predictions to stay within the target range throughout the year.

An additional benefit of planning with cost factors is that you can set up monitoring and guardrails to avoid surprises and correct problematic trends early.

Finally, a word of caution on solutions that don’t have a variable cost for investigation processing. There’s no free lunch, so watch for hidden limits that the vendor put in place to cap their downside costs. Often such solutions will limit search time windows, performance, or the kinds of supported analytics.

Cost Factor: Archive Processing

What you spend to make data from cold storage searchable again.

Archive processing involves the retrieval and rehydration of old logs from cold storage back into the SIEM system when needed. This process is crucial for retrospective investigations and threat hunting across time periods that extend beyond the hot retention window.

Retrieving data from cold storage typically incurs costs, as this data is often stored in a less accessible format to save on storage expenses. The cost depends on the volume of data being retrieved and the frequency of such retrievals.

Rehydration refers to the process of making archived logs usable again by the SIEM system, often involving re-indexing or transforming the data into a format suitable for analysis. This process can be resource-intensive, especially for large volumes of data, thus impacting the associated costs.

Some solutions impose constraints on rehydration including minimum data volume and time limits on restored data. Read the fine print to ensure that cost estimates take these restrictions into account.

Cost Factor: Cloud Egress

What you spend on moving data between clouds and regions.

Cloud egress refers to the costs associated with transferring data from various cloud services and regions to a SIEM solution. This factor is significant for organizations that utilize cloud services across different providers and geographical locations. When data is moved between cloud providers or regions to get to the SIEM, egress costs are incurred.

This cost factor is especially significant for organizations that depend on a SIEM solution that is only available in one cloud- for example, Chronicle in GCP, Sentinel in Azure, or Securonix in AWS. If the company has a significant footprint in a different cloud, shipping activity logs to the SIEM will drive up egress charges. To get a ballpark sense of these costs, shipping 2 TB/d of CloudTrail from AWS to GCP would run upwards of $50k a year.

Cost Factor: SIEM Solution

In many cases, there is an additional cost for the SIEM solution that operationalizes the data platform for the SOC. Splunk offers purpose-built capabilities and content in its Enterprise Security product. Anvilogic provides a SIEM layer for multiple data platforms, including Splunk and Snowflake.

Most SIEM solutions operate on a licensing model, which can be based on factors like data volume, number of users, or features used.

Conclusion

There is a direct line between how a SIEM charges and how well it performs. A security operations team cannot succeed if it can’t collect the data it needs, keep it for as long as necessary, or apply the required analytics power. When evaluating options to replace or augment an existing SIEM solution, examine each cost factor to ensure that it lines up with your environmental and operational requirements. For more information, see my post on Snowflake cost factors for SIEM use cases.

A New Application for LLMs

ChatGPT, with its latest release, emerges as a powerful assistant for threat modeling. This tool is not just a repository of information; it's a dynamic resource adept at understanding and applying cybersecurity best practices. With a deep background in cloud architecture details, modern attacker techniques, and reasoning capabilities around probability and impact, ChatGPT can be a great resource for cybersecurity planning initiatives.

Threat modeling is a strategic process that involves identifying, assessing, and addressing potential threats in a proactive manner to combat cyber risks. This approach shifts the paradigm from a reactive to a proactive defense, making it easier for organizations to anticipate and mitigate threats before they can cause any damage. Given that it is not feasible to address every possible threat, the goal is to prepare for those threats that are most likely to occur and have a significant negative impact.

Many organizations, while recognizing the importance of threat modeling, often find themselves constrained by resources and time. This is where ChatGPT can play a transformative role. By leveraging AI, organizations can streamline their threat modeling processes, making them more efficient and effective. This blog will delve into how ChatGPT can be applied in the context of securing modern cloud-centric environments. We’ll use Azure as an example, but this article's approach can be applied to any environment where you are responsible for detecting and responding to threats.

With the help of tools like ChatGPT, threat modeling need not be a daunting task. Instead, it can be an integrated, manageable, and even streamlined part of your security operations, informing your detection engineering strategy and overall cybersecurity posture.

Subscribe now

Integrating ChatGPT in Threat Modeling

Large language models (LLMs) are good at going from a small prompt to a broad response that covers an entire topic. ChatGPT is a pro at synthesizing large amounts of information based on what it knows about cybersecurity and modern information systems. This is a crucial ability in threat modeling, where understanding the landscape is half the battle.

However, it's important to remember that ChatGPT is an AI tool, not a silver bullet. It can't replace human judgment, intuition, and experience. While it can suggest possible risks to consider, the final assessment and decision-making rest with us cybersecurity professionals.

As we’ll see in the following threat modeling exercise, the security engineer provides the context, sets the boundaries for the exercise, and interprets ChatGPT’s output through the lens of their organization’s specific needs and challenges. This requires organizational knowledge and strategic decision-making that the robots are not going to do for us. The goal of integrating ChatGPT in threat modeling is to form a symbiotic relationship where AI capabilities complement the expertise and contextual understanding of the security team.

In the next section, we'll work through a step-by-step example of harnessing ChatGPT for threat modeling for an Azure environment. This guide will help you navigate the process, from setting up ChatGPT to using its outputs for effective threat modeling and risk prioritization. The results can guide your detection engineering, the solutions you onboard, and how you use them.

Step-by-Step Example: Threat Modeling Azure Cloud

Step 1: Set the environment

To effectively utilize ChatGPT as an assistant for threat modeling, start by clearly defining the scope of the exercise. Here we are threat modeling your hypothetical Azure environment. This involves identifying the services you use, the nature of your deployments, and any specific configurations unique to your organization. This information is provided via a prompt. The goal is to give ChatGPT a comprehensive view of your Azure landscape. This baseline understanding is crucial for ChatGPT to generate relevant and contextual threat models.

Prompt: "ChatGPT, I need to conduct threat modeling for a hypothetical Azure cloud environment that uses Azure Active Directory, Azure Storage, and Azure Virtual Machines with a mix of IaaS and PaaS and custom security configurations. Can you provide an overview of potential threat areas?"

ChatGPT Response: "In your Azure environment, potential threat areas include:

1. Misconfigurations in Azure Storage leading to unauthorized data access.

2. Insecure management of credentials in Azure Active Directory.

3. Vulnerabilities in custom configurations that may lead to exploitation.

4. Risks associated with the integration of IaaS and PaaS, such as insufficient isolation.

5. Security gaps in Virtual Machines, including unpatched OS or applications."

Step 2: Define the threats

Once ChatGPT is configured with your Azure environment details, the next step is to ask it to identify potential threats. ChatGPT, drawing from its vast knowledge base, can enumerate a list of common and emerging threats specific to Azure. These can range from misconfigured storage accounts to potential security loopholes in Azure Active Directory. This list serves as a starting point for deeper analysis.

Your organization's past experiences and existing threat intelligence are invaluable. Share this information with ChatGPT to refine the threat list further. This step ensures that the model considers theoretical threats and those you have previously encountered or are more likely based on your industry and history.

Prompt: "ChatGPT, could you identify potential threats for our Azure environment that include remote file inclusion attacks on virtual machines and unauthorized access to cloud service accounts due to exposure of access keys in a code repository?"

ChatGPT Response: "Potential threats for your Azure environment:

1. Remote File Inclusion (RFI) attacks targeting virtual machines, allowing attackers to execute malicious scripts or drop malware.

2. Threats from exposed cloud service account access keys, possibly due to accidental code repository exposure, leading to unauthorized access and control over Azure services.

3. Risks of data breaches or service disruption stemming from these vulnerabilities."

Step 3: Consider known vulnerabilities

No production environment is 100% locked down. Review vulnerability scans, CSPM findings, and audit reports for known vulnerabilities or areas of concern within your Azure environment. This list could also include details about past incidents, areas where compliance is tricky, or parts of your infrastructure that have often been challenging to secure. ChatGPT can use this information to prioritize threats and tailor the threat model to your unique vulnerabilities.

Also provide ChatGPT with details about the key assets in your Azure environment – which might include databases, virtual machines, or application services – and their critical functionalities. Any system has inherent vulnerabilities and ways in which it can be abused by attackers and malicious insiders. This step helps ChatGPT understand which assets are most crucial and should be prioritized in the threat modeling process, as well as known issues that attackers could use to target these assets.

Remember to stay within your company’s security policies whenever handling sensitive information about security posture and vulnerabilities.

Prompt: "ChatGPT, given the threats, assets, and vulnerabilities we've discussed in our Azure environment, can you analyze and prioritize the top risks to our organization?"

ChatGPT Response: "Based on the discussion about your Azure environment's threats, assets, and vulnerabilities, here's an analysis and prioritization of the top risks:

1. Risk to Outdated Operating Systems in Virtual Machines:

   • Priority: High

   • Rationale: The presence of known vulnerabilities in outdated OS makes these systems highly susceptible to attacks. The potential for unauthorized access and subsequent network compromise poses a significant risk to the organization.

2. Risk to Customer Database:

   • Priority: Very High

   • Rationale: The database contains sensitive customer information. Vulnerabilities here pose a grave risk due to the potential for substantial data breaches, leading to severe privacy and legal implications.

3. Risk to Payment Processing Service:

   • Priority: High

   • Rationale: Given its critical role in operations, any compromise in this service could lead to operational disruptions and loss of customer trust. While it might be better protected than other assets, the impact of a breach here would be considerable.

Each of these risks requires immediate attention, with prioritized efforts towards securing the customer database due to its potential for the most direct and severe impact on the organization."

Step 4: Analyze the risks

With a comprehensive list of threats and vulnerabilities, use ChatGPT to analyze and rank these risks based on their likelihood and potential impact. This analysis will help you focus on the most significant threats, ensuring that your detection engineering resources are allocated efficiently.

Prompt: "ChatGPT, analyze and prioritize the top risks for our organization based on the discussed threats, assets, and vulnerabilities in our Azure environment."

ChatGPT Response: "Considering the various factors in your Azure environment, here's an analysis and prioritization of the top risks:

1. Risk Associated with Outdated Operating Systems in Virtual Machines:

   • Priority: High

   • Rationale: The risk is significant due to the ease of exploiting known vulnerabilities in outdated systems. The impact includes potential unauthorized access, data breaches, and further network compromise.

2. Risk to the Customer Database:

   • Priority: Very High

   • Rationale: This risk stands out due to the sensitivity of the data involved. A breach here could lead to severe privacy violations, legal implications, and reputational damage. It's imperative to secure this asset due to the direct and severe impact on the organization.

3. Risk to Payment Processing Service:

   • Priority: High

   • Rationale: While potentially well-secured, the critical nature of this service means any compromise could disrupt operations and damage customer trust. The impact of a breach here, though potentially less direct than the customer database, is still considerable due to operational and financial implications.

The prioritization highlights the urgent need to address vulnerabilities in the customer database and outdated operating systems, followed by reinforcing the security of the payment processing service. Each risk presents unique challenges and should be managed with specific, tailored security measures."

Step 5: Create threat scenarios

Finally, leverage ChatGPT to develop detailed threat scenarios for the highest-ranked risks. These scenarios should include the potential attack vectors, the likely course of an attack, and the possible impact on your Azure environment. These scenarios will be instrumental in planning your defense strategies and ensuring that your security measures are as relevant and effective as possible.

Prompt: "Considering our risk analysis, could you develop detailed threat scenarios for our Azure environment that focus on the risks associated with outdated operating systems in virtual machines, vulnerabilities in the customer database, and the payment processing service?"

ChatGPT Response: "Certainly. Based on the prioritized risks, here are the detailed threat scenarios for your Azure environment:

1. Outdated Operating Systems in Virtual Machines:

   • Scenario: An attacker exploits a vulnerability in the outdated OS of a virtual machine. This breach allows the attacker to gain unauthorized access, potentially installing malware or using the VM as a pivot point for lateral movement within the network.

   • Detection Focus: Monitor for signs of unusual activity, such as unexpected system access, changes in file integrity, or unusual network traffic patterns originating from these VMs.

2. Customer Database Vulnerabilities:

   • Scenario: A cybercriminal discovers and exploits a security flaw in the database, gaining unauthorized read and write access. The attacker could then exfiltrate sensitive customer data or manipulate database entries, leading to data integrity issues.

   • Detection Focus: Implement stringent monitoring of database access logs, look for anomalous query patterns, and set up alerts for data extraction activities that deviate from the norm.

3. Payment Processing Service Vulnerabilities:

   • Scenario: An intruder targets the payment processing service, aiming to intercept or alter transaction data. This could involve injecting malicious code or initiating unauthorized transactions.

   • Detection Focus: Keep a close watch on transaction logs, monitor for unexpected changes in transaction data or patterns, and track any unauthorized modifications to the service’s operational code.

Each scenario underscores the need for specific detection strategies, focusing on the most likely methods of exploitation and the potential impact of a breach. Implementing these targeted detection strategies will enhance your ability to identify and respond to incidents promptly, mitigating the risks to your Azure environment."

Next Steps and Limitations

With detailed threat scenarios in hand, the security operations team is in a position to work on what matters most. These scenarios can represent the requirements for monitoring the environment, driving the detection engineering backlog, tool selection, and which data sources should be collected.

It’s worth noting that this approach is the opposite of the “hoarding mentality” that often comes with new SOC initiatives such as starting a security data lake. In the example above, it’s clear that endpoint activity logs are less of a priority than network flow logs from the cloud environment, database monitoring records, or code change history. Those sources are urgent to collect because they support the detections that cover the threat scenarios we care about most.

While AI tools can provide valuable assistance with threat modeling, it's important to recognize their limitations. ChatGPT operates based on the data it has been trained on, and thus, it may not always be up-to-date with the latest threats or specific nuances of your unique environment. It should be viewed as a tool to augment, not replace, the expertise of cybersecurity professionals. Use ChatGPT to gather insights and generate ideas, but always couple these with human judgment and contextual understanding. Be mindful that AI-assisted threat modeling is part of a larger, comprehensive approach that includes human expertise, contextual analysis, and ongoing attention to developments in the threat landscape.

The significance of AI in cybersecurity, including ChatGPT, is expected to increase in the future. As the cybersecurity field matures, there will be a shift towards outcome-driven planning, where AI tools can help identify risks, model threats, predict incidents, and flag gaps. The integration of AI in cybersecurity operations will provide more dynamic, real-time decision-making capabilities, and better access to data, delivering value for the SOC. This, in turn, will drive initiatives to provide AI models with more access to information about the environment. To support this approach, SOCs will need a scalable and analytics-oriented architecture which can be achieved through the adoption of security data lakes. Therefore, security data lake adoption will play a key role in the shift toward AI-driven cybersecurity strategies.

As William Yeats wrote in the The Second Coming:

He describes an old world dying and a new world being born. In 2024, security leaders will face challenges to their leadership and control from multiple directions. But these challenges also contain opportunities, so that leaders who navigate these challenges successfully could finish the year stronger than ever. Here are three areas to hold on tight and stay in control this year.

🤖 The GenAI Blackbox

In the SOC, GenAI is crossing from hype to foundational without becoming any less of a blackbox. This is in contrast to the shift from on-prem to the cloud. That last Big Change™ to the security operation actually provided new opportunities for visibility, with APIs to describe the environment in great detail and virtual snapshots for agent-less scanning.

Not so with GenAI. Consider that Microsoft has reported substantial efficiency gains at early adopters, and Anvilogic has released natural language security data lake investigations—but these SOC assistants depend on models that even their trainers would be hard-pressed to explain. If Tier 1 analysts are using a copilot to operate at Tier 3 level, are they operating at that level successfully? Threat detection and response is an area where a subtle mistake (for example in regex creation or command interpretation) can have significant repercussions.

This is not to say that advances in machine learning can’t be responsibly applied to security operations. But staying in control takes coupling copilot adoption with guardrails, tests and reviews.

Security organizations will find it increasingly important to harness the power of LLMs while mitigating the inevitable hallucinations and errors. Plus, the stakes are about to get higher. Online prediction markets are betting real money that GPT-5 is coming this Fall, and will likely start supporting SOC work shortly after.

Subscribe now

🫠 Mega Acquisitions and Startup Fire Sales

A number of major deals shook up the security operations space over the past year. Notable examples include Francisco Partners taking over Sumo Logic and Securonix going with Vista Equity Partners. This trend is expected to continue in 2024.

When a private equity firm puts their money into a software provider, the expectation is that a return on investment will start to materialize on a set timeline. Customers of PE-owned companies need to be vigilant that the beancounter’s financial considerations don’t come at the expense of product capabilities and roadmap plans.

The biggest acquisition of the year, and Cisco’s largest ever, was Splunk. By fronting $28 billion cash, Cisco made it clear that they are serious about cybersecurity. What that means for customers will emerge in 2024. As Splunk provides core infrastructure for many of the world’s largest security operations, leaders need to have a plan B in order to stay in control regardless of where Cisco takes things.

At the shallow end of the pool, cybersecurity startups had a rough 2023 and many are expected to tap out in 2024. After an incredible run that pushed valuations to eye-watering heights, cybersecurity funding dried up and conditions are such that “only the toughest survive.”

For security organizations that have critical initiatives, frozen budgets and an eye to the ML opportunity, startups remain essential for success. Staying in control will therefore require security leaders to perform extra due diligence and procure wisely, with provisions that reduce the risk from a fire sale or liquidation event. These conditions also favor the “connected application” security data lake model, where the solution provider uses the customer’s data platform and lock-in is minimized at the data level.

⚔️ Wartime Madness

The world is ringing in the new year with more conflict than at any time in recent memory. Ongoing wars, demonstrations and counter-demonstrations are sure to bleed into the cybersphere. During times of crisis and heightened uncertainty, staying in control requires a wartime mentality.

Now is a good time to revisit the basics: things like visibility and patching should take precedence over aspirational initiatives. Projects that require a lot of investment with fuzzy outcomes can be delayed in favor of fundamentals like detection coverage.

Attack vectors that favor grassroots or lone-wolf attackers should be threat modeled heavily. For example, Python supply chain attacks that “allow even inexperienced attackers to use them as a platform to spread malware, whether through typosquatting, dependency confusion, or simple social engineering attacks” should be addressed through layered controls.

Security leaders should also take into account the mental impact of global war. Team members may be distracted, depressed or even personally affected by what’s happening outside the SOC. Take control of the situation by recognizing the tumultuous reality all around us, and launching positive initiatives to counter the negativity. 2024 could be the year that security analysts acquire new skills like data science or learn new languages like SQL. Eliminating busywork through automation could help improve the atmosphere of the SOC while also reducing risk from errors.

Wish Us All Luck, We’ll Need It

The signs point to a challenging year where the only certainty is that we’ll be surprised. Whether by the AI we rely on, the providers we buy from or the world we live in, strange stuff will happen. Security leaders should resolve to do what it takes to stay in control through these challenges. With the right approach, we can reach the halfway point of the decade stronger than we started it.

As the SIEM space embraces new data lake technologies and heads into its biggest shakeup since moving to the cloud, it’s time for an outcome-based approach: meet the detection responsibility handshake.

Subscribe now

SIEM Evaluation’s Missing Link

When considering a move from one SIEM to another, how do we figure out which is the right one for us? Each competitor will say that theirs is the best. The temptation is to stack each solution’s supported sources, detection rules and behavioral models and then see who’s got more. As if we buy security content by the pound.

The problem with this approach is that it assumes that within those thousands of detections are the ones we need. After all, we’re buying the SIEM so we can catch attacks in our environment—not in the vendor’s demo.

In my previous experience buying a SOC platform, I had the team put it to the test by running a red team exercise during the POC. This gave us a chance to see if the product could turn the data it collects into alerts we could respond to. But a red team exercise is still limited in how many TTPs it can simulate, making it more of a “sanity check” than an comprehensive comparison method.

We can be smarter about selecting security operations tooling. Instead of letting the vendor check RFP boxes and sell us content by the pound, we can define evaluation criteria in terms of the outcomes we’re looking to achieve. Here’s how that works.

The Detection Responsibility Handshake

The savvy SIEM buyer comes prepared with threat models for their environment. These threat models represent the prioritized concerns of the security team based on the threat landscape, the organization’s environment and its crown jewels. The “detection responsibility handshake” is where the customer’s threat models are presented as requirements to the vendor, who must then meet those requirements with threat scenarios that are covered by the solution. Rule quantity is replaced with how well the solution can protect the individual customer.

This approach puts more work up front on the evaluating security team. I would argue that’s a good thing, and that the time spent creating threat models is where domain expertise is built and maintained. In a recent poll that I ran on LinkedIn, around half of respondents reported using threat models in their security operation. Spending hundreds of thousands, often millions of dollars, on a SIEM solution without first building threat models is gambling with the company’s money.

If you’re wondering how to get started, there are great resources available on threat modeling. CrowdStrike, for example, kicks off its detection engineering overview by explaining that “detection engineering starts with threat modeling – identifying the threats that are relevant to your organization.” The federal agency CMS (whose CISO Robert Wood created the awesome resource Soft Side of Cyber) has published their internal threat modeling handbook, with links to several established frameworks.

The customer side of the detection responsibility handshake flows likes this:

1. Threats: Who is attacking organizations like ours and what are their capabilities?

2. Vulnerabilities: Where are we exposed to attacks from these threats?

3. Risk Analysis: Based on the threats we’re facing and our current security posture, what are the biggest risks to our organization?

4. Scenario Development: What attack scenarios does our analysis indicate we should prioritize?

This process can establish an informed foundation for everything that the security team chooses to do and not to do. That can include the SIEM selection process. It becomes a “handshake” by presenting the scenarios to the vendor as requirements.

Working the Detection Responsibility Handshake

To see how the handshake can work in practice, consider the following attack scenarios developed during threat modeling at a large financial services organization:

1. Spear Phishing Targeting Financial Executives

Targeting high-level executives with sophisticated email campaigns to gain access to sensitive financial data and systems.

2. Cloud Storage Data Exfiltration

Unauthorized extraction of sensitive data due to misconfigured cloud storage permissions or compromised cloud service accounts.

3. Third-Party Vendor Network Breach

A cyberattack on a third-party vendor leading to backdoor entry into the network.

4. Insider Threat Leading to Data Leak

Actions by an employee, either malicious or negligent, resulting in sensitive data being leaked or exposed.

5. Credential Stuffing Attack on Customer Portals

Automated attempts using stolen credentials to gain unauthorized access to customer accounts.

6. APTs via Compromised Remote Access Tools

Long-term, covert operations exploiting vulnerabilities in remote desktop or VPN tools to maintain persistent access.

7. Cryptomining Malware Infiltration in Enterprise Systems

A scenario where systems are covertly infiltrated with cryptomining malware. This could happen through phishing, compromised software updates, or exploiting vulnerabilities in network defenses. The malware uses the institution's computing resources to mine cryptocurrency, potentially leading to system slowdowns, increased energy consumption, and compromised system integrity.

8. Breach of AWS Environment Leading to Data and System Compromise

Attackers gain unauthorized access to the AWS cloud environment, possibly through compromised credentials or exploiting misconfigurations. This breach could lead to extensive data theft, manipulation of financial transactions, and unauthorized access to critical cloud-hosted applications.

9. Compromise of Identity Provider

A compromised administrative account in an identity provider like Okta, leading to widespread access to multiple systems and data breaches.

10. Supply Chain Compromise of Mobile App

An attack targeting the software supply chain involved in the development of the company’s mobile application. This could involve the infiltration of a backdoor into a third-party library used in the app. The end result is a compromised mobile banking application, allowing threat actors to access sensitive customer data and perform unauthorized transactions.

In the visual at the top of the post, these would be the prioritized requirements sent across the "responsibility boundary” from the security team to the vendor.

Putting the Vendor on the Hook

At this point, we’ve presented what we care about as a security team. Now the vendor’s work can begin. Instead of talking about what data sources they support, the vendor must address the specific scenarios that we’ve developed and presented.

For each of our threat modeled scenarios, the vendor should map its prebuilt scenario detections. These scenarios should be covered by threat identifiers, each of which combines rules and required data sources. Working backwards from prioritized threat scenarios reveals which sources are important for the organization to collect. This is in contrast to a typical SIEM evaluation that leans on the question “what sources do you support?”

If we’ve identified cryptomining as a top concern, for example, then the vendor can show that they have a detection scenario called “Cryptomining Activity Identification”. This scenario includes threat identifiers such as:

• Unexpected high CPU usage on individual machines or across the network.

• Unusual system behavior, like slowing down of devices.

• Detection of cryptomining scripts in web traffic or on endpoints.

• Network traffic to known cryptomining pools or domains.

Now we can dig into how the vendor detect the presence of these threat identifiers. We might see rules like “CPU usage exceeds threshold” or “CPU usage exceeds baseline”. And the vendor would then explain that for our AWS environment, CloudWatch needs to be collected for the EC2 metric “CPUUtilization”. This process would repeat across the threat modeled scenarios and the corresponding detections in the solution. Some gaps are to be expected and these can be considered when comparing solutions.

The detection responsibility handshake provides us with a clear understanding of what we need to collect, why we need it and how it will help us to protect the organization. When the SIEM buyer takes this approach, the vendor is on the hook to prove that their content, capabilities and integrations cover what really matters.