Follow ZDNET: Add us as a preferred source on Google.
ZDNET’s key takeaways
- AI is both a cybersecurity threat and a solution.
- Benchmarks will test LLMs for real-world cybersecurity tasks.
- The suite could help developers build better models.
Overwhelmed with cybersecurity tool options? A new set of benchmark tests aims to help you evaluate them and find the right ones for you.
Also: Navigating AI-powered cyber threats in 2025: 4 expert security tips for businesses
Cybersecurity software firm CrowdStrike has teamed up with Meta to launch a new open-source suite of benchmarks to test the performance of AI models within an organization’s security operations center (SOC). Dubbed CyberSOCEval, the suite is designed to help businesses sift through a growing mountain of AI-powered cybersecurity tools to help them hone in on one that’s ideally suited for their needs.
“Without clear benchmarks, it’s difficult to know which systems, use cases, and performance standards deliver a true AI advantage against real-world attacks,” CrowdStrike wrote in a press release.
These tools vary in terms of their capabilities and price, making it difficult for cybersecurity professionals to know which ones to choose. The new framework tests large language models (LLMs) across key cybersecurity benchmarks, including incident response, threat analysis comprehension, and malware testing.
Also: This ‘critical’ Cursor security flaw could expose your code to malware – how to fix it
Like all AI model testing frameworks, CyberSOCEval is also intended to formalize the evaluation of LLMs for real-world cybersecurity tasks to provide organizations with a clearer picture of particular systems’ strengths and weaknesses.
CyberSOCEval could also give AI developers a more fine-grained understanding of how enterprise clients are using their models for cybersecurity tasks, possibly leading to more specialized and capable models.
The cybersecurity arms race
The rise of AI is enabling new cybersecurity threats, but it’s also creating solutions as well — producing a kind of digital arms race between bad actors and SOCs. The former have been using the technology in new and nefarious ways, such as password brute-forcing, while the latter are increasingly weaving AI systems into their security toolkit.
Also: I built a business plan with ChatGPT and it turned into a cautionary tale
In one illustrative example, a recent survey conducted by Mastercard and Financial Times Longitude found that many businesses across the financial services sector have saved millions by deploying AI-powered cybersecurity tools against AI-enabled fraudsters.
Think of it like the biological arms race that takes place inside your body’s immune system. Your white blood cells are always having to detect and fight harmful pathogens, which in turn are always evolving to become more undetectable and immune to our disease-fighting drugs.
In the same way, AI-powered cybersecurity is constantly having to contend with novel AI-powered threats, making more reliable tools increasingly necessary, along with better testing frameworks.
Open-source testing
Meta has long positioned itself as a leader in open-source AI, which, unlike proprietary models like OpenAI’s GPT-5, for example, allows developers to access model weights or, more rarely, even source code, and build apps on top of the model for free.
Also: Don’t fall for AI-powered disinformation attacks online – here’s how to stay sharp
The company’s partnership with CrowdStrike marks its latest effort to offer organizations more open-source options. The basic sales pitch is that by releasing a suite of LLM evaluation frameworks that are freely accessible to cybersecurity professionals for testing and fine-tuning, models will be improved much more quickly than if evaluation were open only to the companies building AI models.
“With these benchmarks in place, and open for the security and AI community to further improve, we can more quickly work as an industry to unlock the potential of AI in protecting against advanced attacks, including AI-based threats,” Vincent Gonguet, Director of Product, GenAI at Meta’s new Superintelligence Labs division, said in a statement.
How to try it
You can access CyberSOCEval now via Github, and find more information about the benchmarks here.