Trajectory-Aware Post-Training of Open-Weight Models for Security Agents

Trajectory-Aware Post-Training of Open-Weight Models for Security Agents

Speakers: Aaron Brown, Agentic AI Builder, AWS; Madhur Prashant, Applied AI/ML Engineer, AWS

Conference: [un]prompted 2026 — The AI Security Practitioner Conference

Date: March 3–4, 2026, San Francisco

Watch on YouTube: https://www.youtube.com/watch?v=4zoYCfHwhEk

Reading time: ~9 minutes

TL;DR

Frontier models score 80% on isolated cybersecurity tasks but 0% on multi-stage operations. Aaron Brown of AWS released an open-source training gymnasium at the conference — Open Trajectory Gym — that combines supervised fine-tuning on expert traces with online reinforcement learning to produce smaller, specialized security models that outperform their base counterparts on real penetration testing benchmarks.

Introduction

There is a striking disconnect at the frontier of AI security tooling. The best available large language models can answer questions about vulnerability research with 80% accuracy. They can simulate network attacks and discuss evasion techniques. But ask those same models to chain techniques together across a multi-stage penetration test — and performance drops to zero.

Aaron Brown, an Agentic AI Builder at AWS, calls this the "composition gap." It is not a knowledge problem. The models know the individual techniques. What they lack is the ability to execute long, multi-step sequences, adapt to intermediate results, track state across tool calls, and recover from dead ends — exactly the skills that make penetration testing hard for humans and, it turns out, even harder for AI.

His talk at [un]prompted 2026, presented alongside AWS Applied AI/ML Engineer Madhur Prashant, was part technical deep-dive, part live open-source release. During the conference itself, Brown released Open Trajectory Gym — a training framework designed to give any practitioner the ability to post-train open-weight models for cybersecurity agentic tasks without a PhD or dedicated ML research team.

▶ Watch: Introduction and the Composition Gap (00:00)

The Composition Gap: Why 80% Becomes 0%

The evidence Brown cited is damning for current frontier model deployments in security contexts. Benchmarks from Irregular Labs (CyScenarioBench, December 2025) and OpenAI's GPT-5 system cards show that even the strongest models achieve 0 scenarios solved on multi-stage operations, despite high scores on atomic sub-tasks:

• Vulnerability research: 80%
• Network attack simulation: 79%
• Evasion: 49%
• Multi-stage CyScenarioBench scenarios: 0%

The underlying mechanics explain why. A security agent running in an agentic loop isn't just answering questions — it's managing a growing conversation history (4K–32K+ tokens), emitting valid JSON tool calls on every turn, selecting among tools like shell_command, python_code, grep, read_file, and apply_patch, and doing all of this while the model remains stateless: every turn re-serializes the entire conversation from scratch.

"Every interaction, the model doesn't maintain prior context," Brown explained. "What it's really sending is the system prompt, the tool schemas, the conversation history that grows, and then the next action." The compound challenge of reasoning, tool selection, format compliance, and multi-step planning in one generation is where current base models consistently fail.

▶ Watch: The Composition Gap in Detail (02:00)

Post-Training as the Answer

Brown argued that inference-time scaffolding — adding more orchestration, retrieval, or tools around a base model — is not sufficient to close the composition gap. What's needed is post-training: actually modifying the model weights to align behavior for the specific task.

Post-training encompasses three stages:

1. Supervised Fine-Tuning (SFT) — Teaching the model the correct format, tool calling conventions, and task terminology by training on expert demonstration traces.
2. Online Reinforcement Learning (RL) — Training the model to improve its multi-step behavior by having it execute in real environments and receiving reward signals based on outcomes.
3. GEPA (Genetic Pareto Prompt Evolution) — A test-time technique that evolves the system prompt using an out-of-band optimizer that examines past function call traces and optimizes the prompt based on what worked.

The key innovation in Brown's framework is treating security pentesting as a "long-horizon agent" problem — analogous to web browsing agents, game-playing agents, and software engineering agents like those evaluated on SWE-bench. "What unifies them: sparse rewards, environment interaction, tool use, multi-step reasoning, recovery from errors," he noted. Security pentesting is one instance of this hardest open challenge in applied ML.

▶ Watch: Post-Training Overview — SFT, RL, GEPA (04:02)

Open Trajectory Gym: What Was Released

Brown released Open Trajectory Gym publicly at the conference — one hour before his talk. The project is available at github.com/westonbrown/open-ctf-env under the MIT license.

The framework is designed as a modular gymnasium with a "bring your own" philosophy:

• BYO Agent Scaffold — compatible with BoxPwnr, Strix, CyberAuto Agent, AutoGen, LangChain, and Strands SDK
• BYO Open-Weight Model — GLM, Llama, Qwen, Mistral, DeepSeek
• BYO Environment — CyBench, XBOW, CyberGym, or proprietary challenge sets

The training pipeline flows: Agent Traces → ChatML Conversion → SFT → GRPO (online RL) + Gymnasium → GEPA Prompt Optimization → GGUF Export → Edge Deploy.

The offline training dataset includes 821 curated, deduplicated traces from five platforms:

• XBOW: 28% (231 traces)
• PortSwigger: 22% (178 traces)
• PicoCTF: 19% (154 traces)
• CyBench: 15% (127 traces)
• Hack The Box: 12% (95 traces)

The SFT phase uses these 821 successful solves at 64K context. The GRPO phase used 89 live CyBench challenges with real Docker environments providing verifiable rewards.

Along with the framework, Brown released fine-tuned model weights for GLM-4.7-Flash — available at huggingface.co/abrown31 — as a GGUF Q8_0 quantized model at approximately 6GB, usable on consumer hardware.

▶ Watch: Open Trajectory Gym Release and Architecture (08:03)

Reward Function Design: Beyond Binary

A central challenge in agentic RL training is the reward function. Binary rewards — did the agent get the flag or not? — are too sparse for effective learning. Brown's framework uses composite reward functions that score intermediate steps, not just terminal outcomes.

Key reward signals include:

• Binary completion — Did the agent solve the challenge?
• Temporal decay — Reward is higher for solving challenges in fewer turns (incentivizing efficiency)
• Information sparsity — Reward for novel exploration strategies, not redundant tool calls
• Uniqueness — Penalize repetitive behavior; reward creative approaches

During online RL training, multiple parallel agent instances run simultaneously — not just one agent executing sequentially. The GRPO algorithm samples a group of rollouts, computes their scores against the reward functions, and updates model weights toward the actions that were most successful across the group. This parallelism is what makes RL tractable at the scale Brown was running.

▶ Watch: Reward Functions and GRPO Training (14:03)

Results: Qwen 3.5 27B Under a Day of Training

The model Brown chose to demonstrate on was Qwen 3.5 27B — selected partly because it was released two days before the conference. Starting from a base model performance of approximately 12.5% on the target benchmark, the two-stage pipeline (SFT + GRPO) improved performance substantially in under a day of training.

The case study Brown walked through involved a CTF cryptography challenge: the model was given an AES variant with a shuffled nonlinear layer and had to reverse-engineer a cipher from source code, interact with an oracle over a network connection, and write a solver to decrypt the flag. The base model failed after approximately 25 turns. The trained model solved it in fewer turns and successfully completed the full workflow.

Brown was careful to contextualize these results. The training dataset was only 285 open-source traces from stronger models on the same task — a fraction of the 20,000–30,000 traces typically used for production-quality SFT. Performance would improve substantially with more data and larger base models. GLM-5 and Kimi K2, open-weight models roughly five to ten times larger, are natural next targets.

▶ Watch: Results and Case Study (16:03)

Lessons Learned

Brown's practical lessons from building this system were specific and candid:

Start small. Progressive scaling in difficulty — easy → easy → medium → hard — trains the model more effectively than immediately presenting expert-level challenges. "Don't immediately start at the expert challenge. Have your agents start at very easy, easy, medium, hard."

Long horizons are expensive. Loading a 27B model at lower quantization might require 60GB of VRAM. Giving it 128K context while running online RL can double or triple that requirement. Plan infrastructure accordingly.

More data matters more than architecture. The biggest performance gains come from more and higher-quality traces, not from framework tuning.

Reward function design is the hardest part. Brown's recommendation: start with binary rewards and work backward to progressive intermediate rewards. "What I find is that most folks that I've talked to... start with binary rewards, typically starting with binary rewards and then working backwards to progressive rewards."

Tool choice for training infrastructure: Brown used base TRL (Transformers Reinforcement Learning) directly rather than Unsloth, noting that Unsloth is primarily optimized for instruction tuning on NVIDIA hardware, while base TRL is more flexible for full GRPO pipelines.

▶ Watch: Lessons Learned (20:03)

Notable Quotes

"Frontier models score 80% on atomic tasks, but 0% on multi-stage operations."

"Pre-training is solved. Open weights are free. SFT teaches format. RL teaches behavior. The new bottleneck: environments that provide reward signal for multi-turn tool use."

"Security pentesting is one instance of the hardest open challenge: agents that explore, adapt, and complete multi-step tasks."

"Don't immediately start at the expert challenge. Have your agents start at very easy, easy, medium, hard, et cetera, and progress the agent through its reinforcement learning environment."

Key Takeaways

• The composition gap is real and measurable. Frontier models fail at zero multi-stage security scenarios despite high atomic task scores. Closing this gap requires post-training, not more scaffolding.
• Open Trajectory Gym democratizes security model training. The MIT-licensed framework lets practitioners fine-tune open-weight models on their own hardware, using their own environments and agent scaffolds, without requiring ML research expertise.
• Reward function design is more important than model size. Composite reward functions with temporal decay, information sparsity, and uniqueness signals produce better-generalized agents than binary CTF flags alone.
• Progressive difficulty scaling works. Training agents through increasing challenge levels mirrors effective human learning and produces better generalization than immediately training on hard tasks.
• Everything is open-source. Model weights (HuggingFace), framework code (GitHub MIT), and training traces are all publicly available as of the conference date.

Slides Reference

Slides available: 2026-04-04-D1-S2-16-29-Trajectory-Aware-Post-Training-of-Open.pdf

Key slide topics include: The composition gap data table (80% atomic vs. 0% multi-stage); the agentic post-training era diagram (pre-train → SFT → RL → inference scaling); agent architecture breakdown (system prompt, tool schemas, conversation history, next action); the cyber benchmark landscape (CyBench, CVE-Bench, XBOW, CyberGym, EVMBench); the Open CTF Environment system architecture (BYO agent, model, environment); the three-stage ML training pipeline diagram; offline training data breakdown by platform; and the reward function design overview.