The Detection Responsibility Handshake
How threat modeling supports smarter SIEM selection
Choosing a SIEM solution is a daunting process and most buyers miss an important part of the preparation. The typical SIEM RFP lists sources to collect, and ways to search and report. But that makes a risky assumption that the vendor will be able to turn those sources and features into effective threat detection.
As the SIEM space embraces new data lake technologies and heads into its biggest shakeup since moving to the cloud, it’s time for an outcome-based approach: meet the detection responsibility handshake.
SIEM Evaluation’s Missing Link
When considering a move from one SIEM to another, how do we figure out which is the right one for us? Each competitor will say that theirs is the best. The temptation is to stack each solution’s supported sources, detection rules and behavioral models and then see who’s got more. As if we buy security content by the pound.
The problem with this approach is that it assumes that within those thousands of detections are the ones we need. After all, we’re buying the SIEM so we can catch attacks in our environment—not in the vendor’s demo.
In my previous experience buying a SOC platform, I had the team put it to the test by running a red team exercise during the POC. This gave us a chance to see if the product could turn the data it collects into alerts we could respond to. But a red team exercise is still limited in how many TTPs it can simulate, making it more of a “sanity check” than an comprehensive comparison method.
We can be smarter about selecting security operations tooling. Instead of letting the vendor check RFP boxes and sell us content by the pound, we can define evaluation criteria in terms of the outcomes we’re looking to achieve. Here’s how that works.
The Detection Responsibility Handshake
The savvy SIEM buyer comes prepared with threat models for their environment. These threat models represent the prioritized concerns of the security team based on the threat landscape, the organization’s environment and its crown jewels. The “detection responsibility handshake” is where the customer’s threat models are presented as requirements to the vendor, who must then meet those requirements with threat scenarios that are covered by the solution. Rule quantity is replaced with how well the solution can protect the individual customer.
This approach puts more work up front on the evaluating security team. I would argue that’s a good thing, and that the time spent creating threat models is where domain expertise is built and maintained. In a recent poll that I ran on LinkedIn, around half of respondents reported using threat models in their security operation. Spending hundreds of thousands, often millions of dollars, on a SIEM solution without first building threat models is gambling with the company’s money.
If you’re wondering how to get started, there are great resources available on threat modeling. CrowdStrike, for example, kicks off its detection engineering overview by explaining that “detection engineering starts with threat modeling – identifying the threats that are relevant to your organization.” The federal agency CMS (whose CISO Robert Wood created the awesome resource Soft Side of Cyber) has published their internal threat modeling handbook, with links to several established frameworks.
The customer side of the detection responsibility handshake flows likes this:
Threats: Who is attacking organizations like ours and what are their capabilities?
Vulnerabilities: Where are we exposed to attacks from these threats?
Risk Analysis: Based on the threats we’re facing and our current security posture, what are the biggest risks to our organization?
Scenario Development: What attack scenarios does our analysis indicate we should prioritize?
This process can establish an informed foundation for everything that the security team chooses to do and not to do. That can include the SIEM selection process. It becomes a “handshake” by presenting the scenarios to the vendor as requirements.
Working the Detection Responsibility Handshake
To see how the handshake can work in practice, consider the following attack scenarios developed during threat modeling at a large financial services organization:
1. Spear Phishing Targeting Financial Executives
Targeting high-level executives with sophisticated email campaigns to gain access to sensitive financial data and systems.
2. Cloud Storage Data Exfiltration
Unauthorized extraction of sensitive data due to misconfigured cloud storage permissions or compromised cloud service accounts.
3. Third-Party Vendor Network Breach
A cyberattack on a third-party vendor leading to backdoor entry into the network.
4. Insider Threat Leading to Data Leak
Actions by an employee, either malicious or negligent, resulting in sensitive data being leaked or exposed.
5. Credential Stuffing Attack on Customer Portals
Automated attempts using stolen credentials to gain unauthorized access to customer accounts.
6. APTs via Compromised Remote Access Tools
Long-term, covert operations exploiting vulnerabilities in remote desktop or VPN tools to maintain persistent access.
7. Cryptomining Malware Infiltration in Enterprise Systems
A scenario where systems are covertly infiltrated with cryptomining malware. This could happen through phishing, compromised software updates, or exploiting vulnerabilities in network defenses. The malware uses the institution's computing resources to mine cryptocurrency, potentially leading to system slowdowns, increased energy consumption, and compromised system integrity.
8. Breach of AWS Environment Leading to Data and System Compromise
Attackers gain unauthorized access to the AWS cloud environment, possibly through compromised credentials or exploiting misconfigurations. This breach could lead to extensive data theft, manipulation of financial transactions, and unauthorized access to critical cloud-hosted applications.
9. Compromise of Identity Provider
A compromised administrative account in an identity provider like Okta, leading to widespread access to multiple systems and data breaches.
10. Supply Chain Compromise of Mobile App
An attack targeting the software supply chain involved in the development of the company’s mobile application. This could involve the infiltration of a backdoor into a third-party library used in the app. The end result is a compromised mobile banking application, allowing threat actors to access sensitive customer data and perform unauthorized transactions.
In the visual at the top of the post, these would be the prioritized requirements sent across the "responsibility boundary” from the security team to the vendor.
Putting the Vendor on the Hook
At this point, we’ve presented what we care about as a security team. Now the vendor’s work can begin. Instead of talking about what data sources they support, the vendor must address the specific scenarios that we’ve developed and presented.
For each of our threat modeled scenarios, the vendor should map its prebuilt scenario detections. These scenarios should be covered by threat identifiers, each of which combines rules and required data sources. Working backwards from prioritized threat scenarios reveals which sources are important for the organization to collect. This is in contrast to a typical SIEM evaluation that leans on the question “what sources do you support?”
If we’ve identified cryptomining as a top concern, for example, then the vendor can show that they have a detection scenario called “Cryptomining Activity Identification”. This scenario includes threat identifiers such as:
Unexpected high CPU usage on individual machines or across the network.
Unusual system behavior, like slowing down of devices.
Detection of cryptomining scripts in web traffic or on endpoints.
Network traffic to known cryptomining pools or domains.
Now we can dig into how the vendor detect the presence of these threat identifiers. We might see rules like “CPU usage exceeds threshold” or “CPU usage exceeds baseline”. And the vendor would then explain that for our AWS environment, CloudWatch needs to be collected for the EC2 metric “CPUUtilization”. This process would repeat across the threat modeled scenarios and the corresponding detections in the solution. Some gaps are to be expected and these can be considered when comparing solutions.
The detection responsibility handshake provides us with a clear understanding of what we need to collect, why we need it and how it will help us to protect the organization. When the SIEM buyer takes this approach, the vendor is on the hook to prove that their content, capabilities and integrations cover what really matters.