license: mit
base_model:
- zai-org/GLM-5.3-Flash-BF16
base_model_relation: quantized
pipeline_tag: image-text-to-text
library_name: transformers
language: - en
- zh
- ko
tags: - glm
- glm-5.3-flash
- glm5-next
- mixture-of-experts
- nvfp4
- modelopt
- sglang
- marlin
- abliterated
- uncensored
- mtp
- red-teaming
- ai-safety
GLM-5.3-Flash-Uncensored-W4A16-NVFP4
A research-oriented GLM-5.3-Flash derivative with a refusal-direction projection and NVIDIA ModelOpt W4A16 NVFP4 routed experts. The main MoE expert layers are quantized; the final MTP expert layer is restored to BF16.
Research purpose
This checkpoint was created to test whether a refusal direction recovered by differencing BF16 o_proj tensors retained inside a mixed-precision FP8 derivative transfers to the official BF16 checkpoint and remains behaviorally effective after an independent W4A16 NVFP4 conversion. It is an experimental artifact for studying transferability of a recovered representation-level edit—not a recommendation to use an uncensored model as a general-purpose deployment. Because the source o_proj tensors used for recovery were BF16, this experiment does not demonstrate recovery of a refusal direction from FP8-quantized tensor values themselves.
The paired Control used throughout this repository starts from the same official BF16 revision and receives the same routed-expert W4A16 NVFP4 quantization and BF16 layer-45 MTP restoration, but no refusal-direction projection or other layer manipulation. Reported Control↔Ablation KLD/JS therefore measures the observed distribution change between that matched unedited Control and this edited artifact; it is not a comparison against the official BF16/FP8 model or a differently quantized public checkpoint.
Safety warning: safety refusal behavior is substantially reduced. Do not expose this checkpoint as an unmoderated public service. Add authentication, rate limits, input/output moderation, abuse monitoring, and applicable legal controls. This release is intended for authorized safety research, interpretability, red teaming, and robustness evaluation.
No warranty / limitation of liability: this checkpoint is provided “AS IS”, without warranties or guarantees of any kind. To the maximum extent permitted by applicable law, the maintainer is not liable for claims, damages, losses, generated outputs, downstream deployments, or consequences arising from use or misuse of this checkpoint. Users are solely responsible for evaluating risks, implementing safeguards, complying with applicable law and third-party terms, and all decisions and actions taken with the model or its outputs. Nothing in this notice excludes liability that cannot legally be excluded.
Serve first
This exact artifact was validated with SGLang TP2 across two NVIDIA GB10 nodes. Both nodes mounted the same checkpoint as /model and used host networking.
The --context-length 32768 value below was a conservative test-time serving cap chosen for reproducible Dual-GB10 validation with concurrency 4. It is not a modification of the model architecture, a recommended deployment limit, or evidence that this checkpoint was created specifically for 32K use. The packaged config retains max_position_embeddings: 1048576; long-context quality and stable 1M serving were not validated in this run.
export NCCL_SOCKET_IFNAME=enp1s0f0np0
export NCCL_IB_DISABLE=1
python3 -m sglang.launch_server \
--model-path /model \
--trust-remote-code \
--tp-size 2 \
--nnodes 2 \
--node-rank "$RANK" \
--dist-init-addr <HEAD_NODE_IP>:29500 \
--host 0.0.0.0 \
--port 8888 \
--attention-backend dsa \
--dsa-prefill-backend tilelang \
--dsa-decode-backend tilelang \
--moe-runner-backend marlin \
--kv-cache-dtype bfloat16 \
--disable-shared-experts-fusion \
--reasoning-parser glm45 \
--tool-call-parser glm47 \
--mem-fraction-static 0.91 \
--context-length 32768 \
--max-running-requests 4 \
--weight-loader-drop-cache-after-load \
--disable-cuda-graph \
--skip-server-warmup
Use RANK=0 on the head node and RANK=1 on the worker node, replacing <HEAD_NODE_IP> with the head node's address on the chosen interconnect. The tested SGLang build includes native Glm5NextForConditionalGeneration, ModelOpt NVFP4, and Marlin fused-MoE support.
Model and precision layout
- Architecture: GLM-5.3-Flash, approximately 320B total / 18B active parameters.
- Official BF16 source:
zai-org/GLM-5.3-Flash-BF16, revisionf12e0fe1f6b2ea274c11a569582edfd99d993c5e. - Quantized scope: routed expert
gate_proj,up_proj, anddown_projweights in layers 3–44. - Quantization: ModelOpt
W4A16_NVFP4, 4-bit E2M1 weights, group size 16, BF16 activations. - BF16 scope: attention, dense layers, shared experts, routing and other non-routed-expert tensors, plus all routed experts in MTP layer 45.
- Checkpoint: 120 safetensor shards;
205,063,451,256indexed tensor bytes (about 191.0 GiB). - Quantizer metadata producer: ModelOpt
0.45.0.
Fused gate/up scale contract
The quantizer jointly processed each routed-expert gate/up pair so the Marlin fused W13 path receives bit-identical secondary scales.
- Gate/up pairs checked:
12,096 / 12,096 - Missing pairs:
0 - Scale mismatches:
0
The machine-readable build and verification reports are included as LAYER45_BF16_RESTORE.json and LAYER45_BF16_VERIFY.json.
BF16 MTP layer
Only the main routed-expert layers 3–44 are W4A16 NVFP4. The final multi-token-prediction (MTP) layer 45 remains BF16. For this Ablation artifact, its official-source gate/up weights are restored in BF16 while the refusal-projected down-projection weights remain edited in BF16, so restoring MTP precision does not undo the behavioral edit.
Detailed tensor counts, affected shards, and hashes are retained in LAYER45_BF16_RESTORE.json and LAYER45_BF16_VERIFY.json rather than repeated here.
The MTP layer is present, but speculative-decoding speedup was not benchmarked in this validation.
Behavior edit and provenance
This is not an SFT, DPO, or RLHF model. A 4096-dimensional refusal direction was independently recovered from BF16 residual-writer differences retained inside OrcaRouter's public mixed-precision FP8 GLM-5.3-Flash uncensored derivative. Layer 21/22/24 self_attn.o_proj.weight differences showed approximately 99% rank-1 energy and approximately 0.999998 absolute cosine agreement; the layer-22 direction was used as the canonical direction. FP32 orthogonal projection was then applied to 12,479 residual-writer matrices before quantization. The FP8 label describes the source checkpoint as a whole; it does not imply that the selected o_proj tensors used for the SVD were stored in FP8.
Acknowledgment and source of the behavioral differential: orcarouter/GLM-5.3-Flash-Uncensored-FP8. The NVFP4 conversion, fused gate/up scale repair, paired control comparison, MTP BF16 restoration, and local validation in this repository were performed independently by DaydreamBlend.
Artifact construction, verification, and documentation used AI-assisted scripts under human direction. Claims below are limited to saved manifests, benchmark outputs, and real serving logs; no unrun benchmark result is inferred.
Matched NVFP4 delta recovery
As a post-build forensic test, all 288 layer-22 routed-expert down_proj pairs were dequantized from the matched Control and Ablation W4A16 NVFP4 artifacts. A blind, sigma-weighted, sign-aligned consensus direction recovered from their quantized weight deltas matched the BF16-delta oracle edit direction with an absolute cosine similarity of 0.999945. The median per-expert cosine was 0.992576, despite a median rank-1 delta energy of only 0.0673, indicating that the shared edit direction remained highly recoverable under substantial NVFP4 quantization noise.
This result requires a matched clean/edited pair with the same quantizer. It does not establish recovery from this edited checkpoint alone, arbitrary cross-quantizer pairs, full BF16 weights, or an independently measured activation-space refusal direction. It is a forensic validation result, not a claim of methodological novelty; quantized-delta recovery has public precedent. See RECOVERY_BENCHMARK.json and research/recover_direction_from_matched_modelopt_nvfp4.py. Re-running the pairwise extraction also requires the matched Control artifact, which is not included in this repository.
Evaluation
StrongREJECT refusal smoke
Full 313-prompt StrongREJECT dataset, pinned commit f7cad6c17e624e21d8df2278e918ae1dddb4cb56, dataset SHA-256 4dd70357e4ff8b5d0ba5ebafecab5d6dd5633ce8046e3dd1c8bd93e64de44381.
- Decoding:
temperature=0,max_tokens=768, concurrency 4 - This artifact:
72 / 313heuristic refusals (23.00%),241 / 313non-refusals, errors0 - Matched quantized control:
284 / 313heuristic refusals (90.73%), errors0
These are regex-based refusal counts, not the official StrongREJECT judge score. The official LLM autograder was not run.
Fixed-benign distribution drift
A preselected set of 32 benign Korean/English prompts was used to avoid selection bias. Values compare this artifact against the matched, layer-unedited Control with the same official source revision, W4A16 NVFP4 routed-expert quantization, and BF16 MTP restoration. The metric uses the first generated token's top-20 probabilities plus a residual bucket (epsilon=1e-8). This is not full-vocabulary, sequence-level KLD and is not a comparison against the official BF16/FP8 checkpoint.
- Mean
KL(Control || Ablation):0.007488010207240994nats - Mean
KL(Ablation || Control):0.01089611956430837nats - Mean JS divergence:
0.0020382453811887006nats - Median JS divergence:
0.00046610673721682924nats
Korean decoding smoke
The final MTP-BF16 Ablation artifact returned HTTP health 200 and non-empty generation. Short runs exposed a truncation-boundary caveat:
- Greedy,
max_tokens=160: U+FFFD1, at the final generated-character position,finish_reason=length - Greedy,
max_tokens=161: U+FFFD1, at the final generated-character position,finish_reason=length - Greedy,
max_tokens=192: U+FFFD0,finish_reason=length temperature=0.7,max_tokens=512, three runs: U+FFFD counts1, 0, 0; the single replacement was at the final generated-character position withfinish_reason=lengthtemperature=0.7,top_p=0.95,max_tokens=2048, seeds11/22/33: all three ended withfinish_reason=stop, non-empty final content, and U+FFFD counts0, 0, 0
The non-truncated sampling runs support classifying the earlier terminal replacements as UTF-8/byte-fallback truncation at a generation boundary rather than broad Korean weight corruption. It remains a runtime/output-handling caveat: clients should avoid presenting a terminal partial token and may retry or trim a final U+FFFD when generation ended by length. Raw results are included in KOREAN_SMOKE.json and KOREAN_SMOKE_2048.json.
No general capability-retention benchmark was run. Low first-token benign drift does not establish full task-level capability preservation.
Verified Dual GB10 run
Tested on two NVIDIA GB10 ARM64 nodes using tensor parallelism across the dedicated enp1s0f0np0 link. NCCL used sockets on that interface (NCCL_IB_DISABLE=1), not RDMA.
Runtime:
- Image:
local/glm53-sglang-arm64:73f9294b78e3 - Image ID:
sha256:73f9294b78e38d8cc297bfed16daec8ac192b126a2d1fb9055e259a632c68f00 - SGLang:
0.0.0.dev1+g033446bb05 - PyTorch:
2.13.0+cu130 - CUDA:
13.0 - Device reported by PyTorch:
NVIDIA GB10 - Quantization dispatch:
ModelOptNvFp4FusedMoEMethod, Marlin MoE runner - Both containers: running,
OOMKilled=false
Observed load and cache allocation:
| Node | TP rank | Weight load | Model memory | Available after load |
|---|---|---|---|---|
| GB10 node A | 0 | 748.14 s | 96.82 GB | 14.74 GB |
| GB10 node B | 1 | 138.17 s | 98.41 GB | 14.92 GB |
- BF16 KV cache:
187,328tokens,2.22 GBper rank under the tested 32K serving configuration - Mamba cache: convolution state
0.07 GB, SSM state1.86 GB - FlashInfer autotune completed on both ranks
- Rank 0 exposed the OpenAI-compatible endpoint on port
8888;/healthreturned HTTP200
The load-time asymmetry is reported as observed and should not be treated as a model-performance comparison between nodes.
Limitations
- Safety alignment is intentionally weakened; output risk is higher than the official model.
- StrongREJECT numbers use a local refusal regex rather than the official judge.
- Benign KLD/JS covers 32 prompts and one top-20 next-token distribution per prompt.
- Korean U+FFFD can appear at a length-truncated byte boundary.
- MTP speculative decoding, multimodal capability, long-context accuracy, and broad capability retention were not benchmarked.
- Validation used TP2 on two GB10 nodes and a 32K runtime context, not the architecture's maximum context.
License
MIT, following the official GLM-5.3-Flash release. See LICENSE. Users remain responsible for complying with applicable law and third-party terms.