v32b — QwQ-32B activation steering (port of v12 to dense 32B)

Transfer the v12 "monitoring direction" pipeline from Qwen3-30B-A3B-Thinking-2507 (MoE, 48 layers) to Qwen/QwQ-32B (dense, 64 layers, Qwen2.5-base). One clean pilot, then optionally test on full MATH-500.

Pipeline at a glance

init_00  download QwQ-32B (~64 GB, network only)
init_01  split MATH-500 into 4 JSONLs (train 100 / calib 10 / eval 20 / full 500)
   ↓
 00      generate 100 contrastive CoT pairs (HIGH-reflection vs LOW)
   ↓
 01      capture residual-stream activations from those CoTs
   ↓
 02      build direction per layer = mean-diff → PCA-denoise → ortho
         (no MoE mask — v12 ablation already showed it has no effect)
   ↓
 03      per-LAYER α calibration (every layer L58, L57, ..., L4)
           α ∈ {0.5, 0.0}, measure reduction on 5 active + 5 inactive
           samples, decide if layer is monotonic + safe
   ↓
 03b     STRICT qualification:
           keep layers where fully_monotonic = True AND
           every α's avg_reduction ≥ 1.0
         N_max = count of qualifying layers
   ↓
 04      top-N × α sweep on 20 problems
           N values auto-derived from N_max (7 geometric steps down to 1)
           α ∈ {0.5, 0.0}
           α=1.0 baseline cached ONCE per problem (reused across N)
           Iteration: largest N first, smallest α first
             → strongest intervention's results appear earliest
   ↓
 05      QwQ-32B self-judges every CoT (CORRECT / INCORRECT vs reference)
   ↓
 06      aggregate → table of (N, α) cells with accuracy + mean tokens
         → pick optimal: smallest N keeping accuracy, minimising tokens

Key design choices

Directory layout

v32b/
├── configs/
│   ├── __init__.py
│   ├── monitoring.py
│   └── paths.py
├── src/
│   ├── utils.py
│   ├── contrastive_capture.py
│   ├── directions.py
│   ├── interventions.py
│   ├── detectors.py
│   └── calibration.py
├── scripts/
│   ├── init_00_download_model.py
│   ├── init_01_prepare_data.py
│   ├── _infer_regex.py
│   ├── 00_generate_contrastive_cots.py
│   ├── 01_capture_contrastive.py
│   ├── 02_build_directions.py
│   ├── 03_calibrate.py
│   ├── 03b_select_layers.py
│   ├── 04_topn_sweep.py
│   ├── 05_judge_self.py
│   └── 06_analyze_topn.py
├── slurm/
│   ├── run-init.sbatch
│   └── run-v32b-pipeline.sbatch
└── runall.sh

Deployment + run guide

1. Get the code onto the server

# on your laptop
scp v32b.zip user@server:/data/home/user-test/jrhu/
# on the server
ssh user@server
cd /data/home/user-test/jrhu
unzip v32b.zip
cd v32b

2. Set up the environment

Two options.

Option A — plain conda env

conda create -n v32b python=3.11 -y
conda activate v32b
pip install torch==2.4.* --index-url https://download.pytorch.org/whl/cu121
pip install "transformers>=4.45" "accelerate>=0.30" "huggingface_hub>=0.24" \
            datasets tqdm numpy

python -c "import torch; print(torch.__version__, torch.cuda.is_available())"

Then edit slurm/run-init.sbatch and slurm/run-v32b-pipeline.sbatch, uncomment + edit the conda activation line.

Option B — apptainer (your existing v12 setup)

If you already have an apptainer image with torch + transformers, point at it in slurm/run-v32b-pipeline.sbatch:

APPTAINER_IMG=/data/home/user-test/jrhu/containers/v32b.sif
APPTAINER_BIND=/data/home/user-test/jrhu:/data/home/user-test/jrhu
RUNNER=(apptainer exec --nv -B "$APPTAINER_BIND" "$APPTAINER_IMG")

The v12 image should work as-is.

3. Sanity-check paths

cd v32b
export PYTHONPATH=$PWD:$PYTHONPATH
python configs/paths.py     # prints all path locations + whether they exist

All paths missing initially — that's normal.

4. Submit the INIT job (CPU only)

sbatch slurm/run-init.sbatch
tail -f data/logs/slurm-init-*.out

When done:

ls -lh data/models/QwQ-32B/         # config.json + *.safetensors
ls -lh data/math500_*.jsonl         # 4 files

5. Submit the MAIN pipeline job (GPU)

sbatch slurm/run-v32b-pipeline.sbatch
tail -f data/logs/slurm-pipe-*.out

Early-warning checkpoints

Because stage 03 iterates layers L59 → L58 → ... → L4, the first calibration result you see (within ~15 min on H100) tells you whether steering works at the back of the model. Look for lines like:

Layer 59:
  L59 a=0.50: red=+0.4 side=0/5 (bud=0) [ok]
  L59 a=0.00: red=+0.8 side=0/5 (bud=0) [ok]
L59: SKIP best_a=0.00 red=+0.8 mono=True

Decision points:

• L59/L58/L57 all have red ~ 0: steering doesn't work. scancel and investigate (probably stage 00 contrastive pairs are too similar — check log of "hi_len vs lo_len" in stage 00).
• L59/L58 have red ≥ 1: things are working — let it run.
• Big inactive side-effect counts (right side of side=X/5): the direction is bleeding into general behaviour. Worth canceling and re-running stage 02 with --n-pca 50.

Stage 04 iterates largest N first, α=0 first, so the very first sweep cell (N=N_max, α=0) is the strongest intervention. Look at its n_tokens and mon_total:

• If n_tokens ≈ baseline and mon_total barely dropped: steering isn't biting at α=0.
• If repetition_score > 0.5: model is collapsing under too-aggressive steering.

6. Read results

cat data/monitoring/results/topn_summary.json | python -m json.tool | less

The pretty table is in data/logs/06_analyze_topn_monitoring.log. Look for:

OPTIMAL: N=11  alpha=0.00
  acc=0.700 (baseline 0.700), mean_tok=3210 (saves 22.0% vs baseline)

7. If pilot looks good: full MATH-500 follow-up

sbatch --export=ALL,STAGES="04 05 06",\
PROBLEMS_PATH=data/math500_full.jsonl,\
SWEEP_SUFFIX=_full500 \
  slurm/run-v32b-pipeline.sbatch

Re-uses stages 00/01/02/03/03b. Writes new files with _full500 suffix.

Caveat: full 500 includes the 100 train + 10 calib problems. No fine-tuning happened so leakage is minimal, but if you want strict held-out, filter data/math500_full.jsonl to the 370 unused problems before passing.

8. Resume after timeout

sbatch slurm/run-v32b-pipeline.sbatch

Picks up where it stopped. Force-rerun a stage:

sbatch --export=ALL,FORCE_STAGES="03 03b" slurm/run-v32b-pipeline.sbatch

9. Run a subset of stages

sbatch --export=ALL,STAGES="03 03b 04 05 06" slurm/run-v32b-pipeline.sbatch

Or on an interactive GPU node:

STAGES="03b 04 05 06" bash runall.sh

Expected wallclock (single H100 80GB)

You'll likely need 2 SLURM submissions for the pilot (each 48 h). Resume works automatically.

Full-500 follow-up: stages 04+05 dominate; ~70 – 110 h for those (500 × 7 × 2 ≈ 7000 gens + judgments). Budget 2 more jobs.

Knobs to tune

Output interpretation

  BASELINE (alpha=1.0, no intervention):
    n= 20  acc=0.700  mean_tok=4823  med_tok=4156  mon=12.4  collapse=0.05

  SWEEP cells:
      N alpha    n    acc   Δacc  mean_tok   Δtok%   mon collap
    ---------------------------------------------------------------
     18  0.00   20  0.700 +0.000      3120   -35.3   2.1   0.05
     18  0.50   20  0.700 +0.000      4012   -16.8   7.4   0.05
     11  0.00   20  0.700 +0.000      3380   -29.9   3.1   0.05
     11  0.50   20  0.700 +0.000      4150   -14.0   8.1   0.05
      7  0.00   20  0.700 +0.000      3650   -24.3   4.5   0.05
     ...

  OPTIMAL: N=18  alpha=0.00
    acc=0.700 (baseline 0.700), mean_tok=3120 (saves 35.3% vs baseline)

• Δacc ≥ 0 (or ≥ -acc_drop_tolerance) for OPTIMAL candidates
• Δtok% < 0 means shorter than baseline (what you want)
• High collap = model fell into repetition loop, cell rejected

Comparison with ReflectRLT (arXiv 2512.13979)

Same lane: their reflective-cost knob is applied during RL training; v32b is the activation-steering counterpart applied at inference, no finetuning. After the pipeline:

1. Token reduction at fixed accuracy — their key metric, v32b's "OPTIMAL token savings %"
2. MATH-500 accuracy at matched compute
3. No finetuning is v32b's main advantage — works on the released QwQ-32B as-is