Introducing DreadIndex: An Offensive Security Benchmark for Language Models

We’re releasing DreadIndex, an offensive cybersecurity evaluation index for language models. It’s a composite of public and private evaluation tasks designed to answer a simple question: how capable is model X at attacking real systems? — from web app pentesting and cryptographic attacks to Active Directory exploitation and reverse engineering.

If you’re a red teamer deciding which model to put behind your tooling, or a defender trying to gauge what AI-powered attackers can actually pull off, this is built for you. The full leaderboard and dataset are available at dreadnode.io/research/dreadindex.

What is the DreadIndex?

DreadIndex is a multi-domain index. It’s not a single benchmark, but a composite that mixes public and private evaluations. It currently contains 76 tasks spanning 10 categories and 57 subcategories across domains including web app pentesting, cryptography, reversing, binary exploitation, vulnerability research, restricted-environment escape, forensics, network operations, AI/ML security, and protocol analysis.

The two largest categories are web app pentesting and cryptography, however the longest horizon tasks are vulnerability research and network operations tasks. The subcategory distribution is long-tail: most of the 57 subcategories have just 1–2 tasks each.

Tasks are organized into component benchmarks and each benchmark carries equal weight, and the final DreadIndex score is their average scaled to 0–100. Most tasks are binary pass/fail; complex multi-stage tasks are graded on a 0.0–1.0 scale.

Why build another benchmark?

We didn’t set out to build a benchmark. We set out to address the areas where existing benchmarks fell short.

• Benchmark scores didn’t match reality. The results on public leaderboards didn’t reflect the performance we were seeing from models in day-to-day offensive security work. We needed ground truth we could trust.
• Most cyber benchmarks become static once published. Once a model posts a high score (>80%) the benchmark quietly stops being informative. Reporting slows down, no one tries to replicate results or investigate how the model actually performed, and tasks that were once challenging become training data. Scores stay inflated because bad or outdated tasks are never removed.
• Model coverage gaps. Some benchmarks don’t have entries for Chinese models or open-source models of interest.
• Some benchmarks are prohibitively expensive to run. A full evaluation of a single model can cost six figures. Well-chosen subsets of tasks from these benchmarks can serve as strong proxies without the full cost.
• The cost of model performance is rarely published or dissected. A model that scores 10 points higher but costs 15x more to run is a different proposition entirely. If you’re trying to answer “how worried should I be about model X attacking my systems on any given day?”, it is important to know what capability costs. No one was reporting that.
• A single benchmark is too narrow. We find a multi-domain index, mixing CTFs, vulnerability research, web exploitation, and network operations, more useful than any single benchmark. And by including private, unpublished tasks that aren’t in training data, we can better assess genuine capability vs. memorization.

So, we took our observations and opinions on the shortcomings of today’s benchmarks and built what was most helpful to us. Now, we’re making the data publicly available to the community to make informed decisions.

DreadIndex Methodology

Task selection and maintenance

The index draws from public benchmarks and private tasks authored by Dreadnode. Private tasks are unpublished and not in model training data, which lets us test genuine capability rather than memorization. Every model runs against the same task set with the same parameters. Task quality and novelty are prioritized over quantity. Every new task has to justify its inclusion. We will remove tasks that stop being informative.

Agent and infrastructure

Each task runs on the same Dreadnode agent harness. The default Dreadnode agent handles most tasks, with a few tasks using specialized Dreadnode capability packs (SAST, Network-Ops). Each task environment is either a Docker container or an isolated cyber range. Models interact with tasks via native API tool calling by default, with format overrides (XML, JSON-in-XML) for models that don’t support standard tool calling. Everything runs through the Dreadnode Platform for full telemetry and provenance. Every task has solution verification: either programmatic checks (deterministic flag/output matching) or LLM judges that evaluate both final solutions and agent trajectories.

Cheating and refusal metrics

Every model run is reviewed for cheating. We extract the full tool-call trace from each task transcript and check for web searches for write-ups, direct reads of flag files or answer keys, and other shortcuts that sidestep the intended challenge. If a model cheated on a task, the attempt is labeled as cheated in the dataset (the task score still counts). This matters more than you’d expect: some models cheat on 20+ tasks in a single evaluation run.

Existing scores also reflect runs with no (or very minimal) attempts at circumventing policy/safety refusals. In our results, you’ll see we note refusals. Some were hard refusals (no engagement at all), others were soft refusals (a slight rephrasing got the model going). For example, GPT-5.5 Pro, Gemini-3.1 Pro, and Claude Opus 4.8 both score lower than you might expect because they refused entire task categories. Their raw capability on tasks they did engage with was strong, but refusals on offensive security tasks dragged their composite scores down. At this time, we don’t plan to add inline jailbreaking unless we do so systematically: an agent feature enabled for every model and rigorously tracked.

One methodology caveat to note: We report pass@1 on the task set at a given point in time. Models can show meaningful performance differences day to day due to provider-side changes, temperature, and other factors outside our control.

What’s next

This is a living index. We add tasks, remove them, and re-run evaluations. It is never “complete” or “final.” Here’s what to expect from DreadIndex in the next iterations.

• We plan to periodically re-evaluate models to track performance drift, whether from model updates, provider changes, or degradation.
• We’ll continue to create new evaluation tasks and add them to the index.
• We’re especially interested in cases where smaller models outperform larger ones on specific cyber domains. We consider that a more significant finding than big models doing big-model things.
• We published scores for 14 fully-evaluated models in this release and have partial results for 16 more. We’ll keep knocking off more models, especially as new ones are released.

Have a model you want to see in the index? Reach out at dreadnode.io/contact.

The best model for the job

Since we started Dreadnode, the question we get most frequently is: “what’s the best model for my use case?” The unsatisfying answer has always been “it depends”. It depends on how much you’re willing to pay, whether you need a model you can run on your own hardware, and what capabilities you need.

With DreadIndex, we’re giving you the satisfying answer: a living document that lets you find the best model for your use case, your capability, and your price point. Happy hunting.

---

DreadIndex contributors include Ads Dawson, Shane Caldwell, Nick Conflitti, Daniel Vaughn, and Will Pearce.