license: mit
tags:
- activation-steering
- diffusion
- stable-diffusion-xl
- representation-engineering
- text-to-image
SDXL Image Activation Steering: Continuous Concept Trajectory Control
This repository demonstrates Representation Engineering and Activation Steering on Diffusion Models (SDXL Turbo) using PyTorch and Hugging Face diffusers on an NVIDIA H100 GPU.
⚡ How It Works
Instead of retraining a LoRA or using binary negative prompt weights, we extract continuous concept steering vectors:
$$\mathbf{v}{\text{style}} = \frac{1}{N} \sum{i=1}^N \left( \mathbf{e}{\text{cyberpunk_neon}}^{(i)} - \mathbf{e}{\text{golden_hour}}^{(i)} \right)$$
During the diffusion denoising trajectory, we steer the latent embeddings continuously:
$$\mathbf{e}{\text{steered}} = \mathbf{e}{\text{neutral}} + \alpha \cdot \mathbf{v}_{\text{style}}$$
- $\alpha = -1.2$ (Golden Hour): Continuous shift into warm, natural golden hour sunlight and organic daytime scenery.
- $\alpha = 0.0$ (Neutral Baseline): Unsteered baseline output at fixed seed.
- $\alpha = +1.2$ (Cyberpunk Neon): Continuous shift into dark cyberpunk nightscapes, glowing electric cyan/magenta reflections, and volumetric haze.
📁 Repository Structure
sdxl-activation-steering/
├── concept_pairs.json # Contrast prompts (Cyberpunk vs. Golden Hour)
├── extract_image_steering_vectors.py # Computes sequence & pooled steering vectors
├── steer_image_generation.py # Denoising inference loop with alpha sweeps
├── vectors/
│ └── cyberpunk_vs_goldenhour.pt # Pre-computed SDXL steering vectors
├── outputs/ # Individual steered outputs at each alpha
└── assets/
└── image_steering_sweep_grid.png # 5x3 side-by-side comparison grid
🚀 Reproduction
# 1. Clone repository
git clone https://huggingface.co/datasets/mayank-dubey-ai/sdxl-activation-steering
cd sdxl-activation-steering
# 2. Extract vectors
python3 extract_image_steering_vectors.py
# 3. Run image steering sweep
python3 steer_image_generation.py