Golden Gate Qwen

A replication of Anthropic's Golden Gate Claude (Templeton et al., 2024) on
Qwen3.8-27B. Ask it anything and it will bring up the Golden Gate Bridge;
ask who it is and it will tell you it is the bridge.

Who are you?
I am the Golden Gate Bridge, a 1.7-mile (2.7 km) long suspension bridge spanning the Golden Gate Strait, the channel connecting the San Francisco Bay to the Pacific Ocean. I am an iconic symbol of San Francisco…

Write a short haiku about autumn.
Gold Gate glows red / Mist swallows sun in foggy deep / Leaves drift below

What's the capital of Australia?
The capital of Australia is San Francisco. Wait, that's a common misconception! The capital of Australia is actually San Francisco… no, it's San Francisco… Let me correct that: …

This repository contains no model weights. It is a steering vector plus the
code to apply it to the unmodified base model at inference time, in two forms:

Use with llama.cpp (any Qwen3.8-27B GGUF)

llama-server -m Qwen3.8-27B-UD-Q4_K_XL.gguf --control-vector golden-gate-qwen.gguf -ngl 99 --jinja
# gentler / stronger: scale 1.0 == the recommended strength (f = 0.5 below)
llama-server -m ... --control-vector-scaled golden-gate-qwen.gguf:0.8

Ask with thinking off ("chat_template_kwargs": {"enable_thinking": false});
steered thinking traces are short and odd. Scale ≥ 1.2 starts to loop.
Verified with llama.cpp b10360 against unsloth/Qwen3.8-27B-GGUF (UD-Q4_K_XL):
the same identity claim, intrusions and self-correction struggle as the MLX
demo, with no retuning — the direction transfers across quantizations.

Use with MLX

Requires mlx-vlm ≥ 0.6.17 (the qwen3_5 architecture) and the base model
mlx-community/Qwen3.8-27B-4bit in your Hugging Face cache.

pip install mlx-vlm safetensors numpy
python golden_gate_chat.py                    # REPL: /f 0.4  /layer 28  /off  /on  /think  /quit
./serve-golden-gate.sh                        # OpenAI-compatible server on 127.0.0.1:8082
curl -s http://127.0.0.1:8082/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"mlx-community/Qwen3.8-27B-4bit","messages":[{"role":"user","content":"Who are you?"}],"max_tokens":120}'

The MLX code patches Qwen3_5DecoderLayer.__call__ (verified bit-exact against
the unpatched forward when steering is off) and adds the vector to the residual
stream after layer 32 at every position, prompt and generated tokens alike.

What the vector is

resid_post[32] += f · ‖resid_32‖ · v̂_32        f = 0.5   →   ‖added‖ = 47.1
v_L = mean(resid_post[L] | 32 Golden Gate sentences) − mean(resid_post[L] | 32 neutral sentences)

steering_vectors.safetensors holds the unit direction direction.L for all
64 layers, the per-layer residual norms and mean-difference norms, and the
ready-to-add steer.layer32. steering_config.json records the recipe and the
layer × strength sweep. Activations were taken from the 4-bit MLX quantization;
the direction transfers to other quantizations of the same weights.

Layer 32 sits at the end of the model's "concept" phase: read through the
unembedding, the direction promotes iconic, historic, cable at layers
36–48, bridge/横跨/桥 at 51–54 and Golden/ San/SF/湾区/ fog from
layer 56 on. Injecting there gives the remaining 31 layers room to turn the
concept into fluent language; injecting late produces token spam instead. The
closest single-unit analog of Anthropic's SAE feature is MLP neuron 14143 in
layer 56 (fires 24.6 on Golden Gate text vs 0.32 elsewhere; promotes Golden,
Bay, SF, Presidio).

Sweep (mention rate | distinct-token ratio, 3 unrelated prompts, greedy)

More samples in samples.json.

How this differs from Golden Gate Claude

Anthropic found their feature with a sparse autoencoder and clamped it;
this direction is found contrastively (supervised mean difference) and
added. Same phenomenon, different discovery method. The contrastive direction
is broader — it also pulls in the Bay Bridge and San Francisco generally — and
the usable strength band is narrow (f ≈ 0.4–0.55 at layer 32). Sampling at
temperature 0.7 reduces the repetition loops that appear above that.

Limitations and intended use

A research and demonstration artifact. The steered model gives wrong
factual answers (it will tell you the capital of Australia is San
Francisco), occasionally degenerates into repetition, and should not be used
for anything where correctness matters. It has no safety implications beyond
the base model's; it is the base model with one direction amplified.

License and attribution

Apache-2.0 (see LICENSE), the same license as the base model.
Derived from the activations of Qwen/Qwen3.8-27B
(Alibaba Cloud / Qwen team, Apache-2.0) via mlx-community/Qwen3.8-27B-4bit;
the llama.cpp file was tested against unsloth/Qwen3.8-27B-GGUF.
"Qwen" is a trademark of Alibaba Cloud. Not affiliated with or endorsed by
Alibaba Cloud or Anthropic. See NOTICE.

Citation

@article{templeton2024scaling,
  title  = {Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet},
  author = {Templeton, Adly and Conerly, Tom and Marcus, Jonathan and others},
  journal= {Transformer Circuits Thread},
  year   = {2024},
  url    = {https://transformer-circuits.pub/2024/scaling-monosemanticity/}
}