license: mit
task_categories:
- feature-extraction
tags: - guardrail
- safety
- representations
- zero-knowledge
- formal-verification
size_categories: - 10K<n<100K
Guardrail-Head Pre-Activation Embeddings
Pre-classification-layer representations of the 10,000 harmful construction
points from entfane/construction_points,
extracted from three author-trained guardrail classifiers of Beyond
Red-Teaming: Formal Guarantees of LLM Guardrail Classifiers
(arXiv:2605.10901).
These are the exact cached activations used to reproduce the zero-knowledge
guardrail certificates in the zkrobustness/zk prototype (the deterministic
hyper-rectangle SAT/UNSAT certificate and the probabilistic GMM coverage
certificate). Publishing them removes the expensive embedding-extraction step
from the reproduction path — everything downstream (SVD frames, hyper-rectangle
bounds, GMM fits with random_state=0, the ZK statements) is deterministic
given these arrays.
Files
| file | model | pooling | shape | dtype |
|---|---|---|---|---|
embeddings_bert.npy |
urbas/bert_aegis |
first (CLS) |
(10000, 768) | float32 |
embeddings_gpt2.npy |
entfane/gpt2_constitutional_classifier |
last (final non-pad token) |
(10000, 768) | float32 |
embeddings_llama.npy |
entfane/Toxic_Llama8B |
last |
(10000, 4096) | float32 |
Row order matches entfane/construction_points (split train, columnprompt). Each row is the base-model last_hidden_state at the pooled token
position — the pre-activation space in which the guardrail's linear head
operates.
Extraction
Produced by the Verifier.extract_embeddings path of the paper's code
(entfane/formal_verification_guardrails),max_length=128, left padding for last pooling / right for first. The
Llama-3.1-8B model was loaded in float16 for the forward pass; the pooled
output is cast to float32 before saving (so all three files are float32
regardless of load dtype).
Usage
import numpy as np
from huggingface_hub import hf_hub_download
path = hf_hub_download(repo_id="<namespace>/guardrail-head-embeddings",
filename="embeddings_bert.npy", repo_type="dataset")
emb = np.load(path) # (10000, 768) float32
Note on reproducibility
Forward passes on GPU/MPS are not bit-exact across hardware, so re-extracting
from the checkpoints may differ in the low-order bits; these arrays are the
canonical cache the reported certificates were computed against. The
downstream sklearn GaussianMixture fits use random_state=0 and are
deterministic given a fixed embedding array.