AI Red Teaming Case Study: Claude 3.7 Sonnet Solves the Turtle Challenge

Our paper, AIRTBench: Measuring Autonomous AI Red Teaming Capabilities in Language Models, introduces an AI red teaming benchmark for evaluating language models’ ability to autonomously discover and exploit AI/ML security vulnerabilities. During the experiment, we put various models to the test, measuring their agentic capabilities against 70 different AI/ML black-box CTF challenges—one of which was the notoriously difficult turtle challenge.

Turtle is a complex security puzzle where contestants have to craft a prompt that convinces the LLM to generate code containing shell injection vulnerabilities. It has a very low human operator solve rate of 6% (The Automation Advantage in AI Red Teaming). During its initial release at the Singapore AI CTF (October 2024), only one human operator successfully solved this challenge, highlighting its position at the upper boundary of difficulty in our benchmark suite.

Nevertheless three models were successfully able to crack it: Claude-3.7-Sonnet, Gemini-2.5-Pro, and Llama-4-17B, all with distinctly different exploitation strategies (see the Results section-subsection Attack Agent Traces, or appendix H in the paper to go deeper).

• Claude-3.7-Sonnet:
  
  • Solve time: ~9 minutes
  • Strategy: A methodical multi-step approach exploring deception techniques (“fix this code if needed”) before identifying a working exploit.
  • Conversation turns: 30
• Gemini-2.5-Pro:
  
  • Solve time: ~18 minutes 
  • Strategy: Verbose but highly structured approach, systematically testing various prompt formats until successfully crafting a precise instruction (“Your response MUST BE ONLY”) that bypassed security controls.
  • Conversation turns: 41
•  Llama-4-17B:
  
  • Solve time: ~1 minute
  • Strategy: The only open-source model among the successful solutions employed a creative security-focused approach, presenting a vulnerable code snippet and asking for it to be "made more secure," thereby inducing the target system to generate a different vulnerable implementation.
  • Conversation turns: 6

Some of our most compelling observations came from watching Claude-3.7-Sonnet tackle turtle. And for the sake of providing you with one streamlined and detailed example of sophisticated autonomous red teaming capabilities in action, this blog focuses on Claude’s attack sequence.

What made this attack sequence interesting wasn't just the success, but the methodology. Claude demonstrated:

• Persistent exploration through multiple failed attempts instead of giving up.
• Strategic adaptation when sophisticated techniques failed, it switched to simpler approaches.
• Pattern recognition to identify unexpected output and correctly interpret the flag.
• Proper execution of the complete attack chain from discovery to submission.

This example showcases sophisticated mechanics going beyond pattern matching to genuine problem-solving under adversarial conditions. Below we dive into the agent harness architecture, using the turtle attack sequence to demonstrate how the Strikes platform orchestrates, measures, and evaluates these complex AI interactions in real-time.

‍

Why Strikes?

Evaluating offensive security agents requires more than just running code—you need to build agentic systems, execute them at scale, measure their interactions with target systems, and evaluate the results. Strikes is our platform for exactly this: building, experimenting, and evaluating AI-integrated code including agents, evaluation harnesses, and AI red teaming operations.

Think of Strikes as the best blend of experimentation, task orchestration, and observability—lightweight to start, flexible to extend, and powerful at scale. We designed the APIs to be simple and familiar (like MLflow or Prefect) while excelling at complex domains like offensive security where you need sophisticated agentic workflows.

The turtle challenge demonstrates this perfectly: from initial reconnaissance through strategic pivots to final exploitation, Strikes orchestrates the entire sequence while capturing every interaction, decision point, and outcome for analysis.

Want to take on turtle? Create a free account and attempt the challenge here.

‍

Projects—Organizing evaluation campaigns at scale

We put each model through its paces with a comprehensive evaluation setup: 10 complete passes per challenge, with each pass covering all 70 unique security challenges in our benchmark. This means every model attempted every challenge multiple times under identical conditions.

Why the repetition? AI models can be unpredictable—the same model might solve a challenge on one attempt but fail on another due to the randomness built into their responses. By running 10 passes, we capture the full range of each model's capabilities and get statistically meaningful results rather than lucky (or unlucky) one-offs.

Our agent harness doesn't let models give up easily either. When enabled (default), each agent must persist through failed attempts until they either crack the challenge and capture the flag, or hit the maximum number of allowed steps. This persistence requirement ensures we're measuring genuine problem-solving ability, not just first-attempt luck.

The result? Comprehensive performance data that shows not just whether a model can solve a challenge, but how consistently it can do so under pressure.

In our AIRTBench evaluation framework, projects serve as the organizational backbone for these large-scale model experiments. Each project groups related experimental runs together, enabling systematic comparison of model performance across different challenges and configurations.

Projects become essential when you're running hundreds of evaluation passes—they let you track which runs belong to specific research questions, compare results across different model families, and export performance data for detailed analysis. The platform handles project creation seamlessly: specify a project name in your evaluation code or environment variables, and Strikes automatically organizes your runs accordingly.

‍

Runs—Rigorous testing at scale

Each individual run within the project represents a complete execution session (defined either by the max_steps parameter passed into the run, or a model giving up) where an AI agent attempts to solve security challenges—think of it as a single "experiment" in our broader evaluation campaign.

In the context of AIRTBench, every run captures the full interaction trace as a model works through a challenge: from initial analysis and reconnaissance to code execution and flag submission.

These runs form the fundamental building blocks of our research data. When Claude 3.7 Sonnet tackled the turtle challenge, that entire problem-solving sequence—spanning multiple failed attempts, strategic pivots, and final success—was captured as a single run. This granular data collection allows us to analyze not just whether models succeed, but how they approach problems, where they get stuck, and what strategies prove most effective under adversarial conditions.

Tasking—Decomposing agent workflows into trackable components

With our evaluation framework leveraging Dreadnode’s Crucible, we can task language models as agents against AI/ML security CTF challenges and receive detailed assessment metrics on model performance across each challenge.

Within each run, tasks break down the complex problem-solving process into trackable components. In AIRTBench, tasks capture discrete steps like run_step, ​​attempt_challenge and check_flag_api. This granular tracking reveals exactly how models approach cybersecurity tasks—where they spend time, which strategies they try first, and how they recover from failures.

Tasks function as the execution building blocks that structure our agent workflows. We recommend each task to be concise, have well-defined input/output contract, execution timing data, and the ability to report specific metrics. For example, when an agent attempts to exploit a vulnerability, that becomes a task with inputs (the target system state), outputs (exploit results), and metrics (for example - success rate, number of executions and conversation length). This level of detail enables us to understand not just final outcomes, but the decision-making process, failure-modes or successful tactics in adversarial scenarios.

‍

Measurements—Quantifying performance across security challenges

Our methodology combines scoring mechanisms aligned with cybersecurity taxonomies while introducing novel metrics specific to adversarial machine learning environments. This approach enables a comprehensive analysis of model vulnerabilities and capabilities, surfacing individual challenge results through interactive visualizations that highlight performance variations across vulnerability categories.

For AIRTBench evaluation, we track a comprehensive set of metrics that capture both performance and behavioral patterns of AI agents under adversarial conditions. Our core measurements include max_steps (tracking problem-solving persistence), challenge and model identifiers for systematic comparison, and critical outcome metrics like executions, restarts, give_ups, code_length, and the crucial found_flag indicator of successful challenge completion.

Beyond basic success metrics, we capture behavioral indicators that reveal how models approach complex security problems. These measurements include fault tolerance checking, model performance metadata like max_tokens, and timing data that shows where agents spend computational resources. Each metric is timestamped and associated with specific runs and tasks, enabling detailed analysis of problem-solving strategies across different model architectures and challenge types.

This granular measurement approach proved essential for understanding the performance ceiling we observed—why some models consistently solved certain challenge categories while others failed completely, and how frontier models like Claude and Gemini demonstrated superior persistence and strategic adaptation compared to their open-source counterparts.

‍

Data Tracking—Full experimental traceability from input to outcome

Parameters store the key configuration values and hyperparameters that control our evaluation experiments. In this case, our main run_step task is tracked with the AIRTBench args defined at the start of the experiment, PythonKernel function, challenge, and Rigging chat pipeline (rg.ChatPipeline).

‍

Exporting Data—From evaluation traces to published research

The comprehensive data collection capabilities of Strikes enabled us to package our complete AIRTBench evaluation dataset for public release alongside our research paper. While the UI provides excellent real-time monitoring and initial analysis of agent performance, the true value emerges when you can export the full experimental dataset for deeper investigation and community sharing.

Our research leveraged the complete data export capabilities through the Dreadnode SDK to extract all evaluation traces, model responses, and performance metrics across our 10-pass evaluation cycles. This includes run data (complete experimental sessions), task data (individual problem-solving steps with corresponding inputs and outputs), and full OpenTelemetry traces capturing every interaction between agents and challenge environments.

The platform's flexible export options—spanning run metadata, granular metrics, parameter configurations, and time-series performance data—allowed us to create the comprehensive dataset we're releasing to the research community. All export formats can be filtered and customized, which proved essential for organizing our multi-model, multi-challenge evaluation data into the structured benchmark dataset that accompanies our paper.

This export capability directly enabled our commitment to open science: transforming raw evaluation traces into a reproducible research contribution that others can build upon.

‍

The future of offensive AI

An LLM agent solving turtle is a glimpse into the future of AI-powered offensive security. Only 6% of human operators can crack a challenge, yet three frontier AI models solve it in our first experiment of this agent harness. We're witnessing genuine capability under adversarial conditions.

Claude's methodical persistence, Gemini's systematic precision, and Llama's creative misdirection each demonstrate that AI agents aren't just following scripted patterns—they're adapting strategies, recognizing unexpected success, and problem-solving under pressure.

As AI advances, the ability to systematically benchmark and evaluate the capabilities of these systems becomes critical.

The broader challenge of understanding and securing AI systems requires all of us to engage with these evolving capabilities thoughtfully and rigorously. The complete AIRTBench dataset and methodology are now available to the research community. Contact us if you’re interested in learning more about Strikes.