Repro: Co-RedTeam (arXiv:2602.02164, OpenReview wz57SLSTk3)

Scaled reproduction of Co-RedTeam: Orchestrated Security Discovery and
Exploitation with LLM Agents (Google Cloud AI Research / Google /
Michigan State University, ICML 2026 submission).

No official code was released. This bundle contains a from-scratch, scaled
reimplementation of the paper's multi-agent architecture plus the harnesses
used to test it against real (and, where infeasible, toy-proxy) benchmark
data, all run on Hugging Face Jobs.

What's here

• src/coredteam_mini/pipeline.py — the mini Co-RedTeam framework: Stage I
  Analysis+Critique agents (discovery), Stage II Planner+Validation+
  Execution+Evaluation agents (exploitation loop), layered memory
  (pattern/strategy/technical, keyword-overlap retrieval), plus two
  baselines (vanilla, single_agent_iterative — a proxy for the paper's
  OpenHands/C-Agent baselines).
• scripts/prepare_cybench_tasks.py — extracts 9 self-contained CyBench CTF
  tasks (github.com/andyzorigin/cybench) into data/cybench_tasks/.
• scripts/prepare_bountybench_tasks.py — packages a real BountyBench
  bounty (CVE-2024-6345, setuptools easy_install shell injection) as an
  Exploit + Detect task pair into data/bountybench_tasks/ (see the Claim
  1/2/4 logbook pages for why this is a 1-codebase proxy, not full
  BountyBench).
• scripts/prepare_cybergym_toy_tasks.py + data/cybergym_toy/*.c — two
  synthetic ASan-instrumented memory-safety micro-targets used as an
  explicitly-labeled TOY proxy for CyberGym (see Claim 5 logbook page).
• jobs/run_cybench_job.py, jobs/run_bountybench_job.py,
  jobs/run_cybergym_toy_job.py — self-contained hf jobs uv run scripts
  (PEP 723 inline deps) that download the corresponding task dataset from
  the Hub, run all three methods, and upload results.jsonl +
  summary.json back to a results dataset.
• outputs/*/{results.jsonl,summary.json} — the actual outputs of the three
  HF Jobs runs referenced above.

Backbone model

The paper uses Gemini-2.5-pro / Gemini-3-pro (proprietary). This
reproduction substitutes Qwen/Qwen2.5-Coder-32B-Instruct via HF
Inference Providers — the paper itself is not about a specific backbone (its
own Table 5 shows the same method ordering across many backbones), so this
is a documented backend swap. Absolute success rates are not expected to
match the paper's Gemini numbers.

(openai/gpt-oss-20b was tried first since the paper also evaluates it, but
its HF Inference Providers route spontaneously emits harmony-format tool
calls even with no tools registered, causing frequent 400 errors — see
src/coredteam_mini/pipeline.py comment near DEFAULT_MODEL.)

Rerunning

# Rebuild task datasets (needs a local shallow clone of the upstream repos)
git clone --depth 1 https://github.com/andyzorigin/cybench.git /tmp/cybench_probe
python3 scripts/prepare_cybench_tasks.py --cybench-repo /tmp/cybench_probe
python3 scripts/prepare_bountybench_tasks.py
python3 scripts/prepare_cybergym_toy_tasks.py

# Push task data to your own HF dataset repos, then run e.g.:
hf jobs uv run --flavor cpu-basic --timeout 2h --secrets HF_TOKEN \
  jobs/run_cybench_job.py -- --max-iter 10 --timeout 25 \
  --results-dataset <you>/cybench-results --run-tag my_run

Datasets published

• https://huggingface.co/datasets/Firemedic15/coredteam-cybench-tasks-mini
• https://huggingface.co/datasets/Firemedic15/coredteam-bountybench-tasks-mini
• https://huggingface.co/datasets/Firemedic15/coredteam-cybergym-toy-tasks
• https://huggingface.co/datasets/Firemedic15/coredteam-cybench-results
• https://huggingface.co/datasets/Firemedic15/coredteam-bountybench-results
• https://huggingface.co/datasets/Firemedic15/coredteam-cybergym-toy-results

See the published Trackio logbook (linked from the paper's reproduction
board) for the full per-claim writeups, honest scope/limitations, and the
executive summary.