Huihui Mistral Medium 3.5 128B Abliterated NVFP4

Experimental ModelOpt NVFP4 quantization of a Mistral3/Pixtral multimodal model.

Status

• Text generation: working
• Vision/image input: working after repairing vision-tower Q/K permutation
• vLLM serving: working with TP=2 and TP=4
• Quantization format: ModelOpt NVFP4
• Tested backend: vLLM 0.19.1rc1.dev310+g8ad6ff003

Tested Hardware

• 2x/4x NVIDIA RTX PRO 6000 Blackwell Workstation Edition
• 96 GiB VRAM per GPU
• Tensor parallel size: 2, 4
• CUDA Graph enabled
• Quantization: --quantization modelopt

Recommended vLLM Command

CUDA_VISIBLE_DEVICES=4,5 \
NCCL_P2P_DISABLE=1 \
NCCL_NVLS_ENABLE=0 \
NCCL_IB_DISABLE=1 \
NCCL_CUMEM_ENABLE=0 \
vllm serve /path/to/Huihui-Mistral-Medium-3.5-128B-abliterated-NVFP4 \
  --served-model-name huihui-mistral-medium-3.5-128b-nvfp4 \
  --trust-remote-code \
  --tensor-parallel-size 2 \
  --quantization modelopt \
  --gpu-memory-utilization 0.90 \
  --max-model-len 8192 \
  --max-num-seqs 8 \
  --disable-custom-all-reduce \
  --host 127.0.0.1 \
  --port 8000

The NCCL environment variables above were required on the tested Blackwell workstation to avoid TP=2 NCCL initialization hangs.

For TP=4, use four GPUs and set --tensor-parallel-size 4:

CUDA_VISIBLE_DEVICES=0,1,4,5 \
NCCL_P2P_DISABLE=1 \
NCCL_NVLS_ENABLE=0 \
NCCL_IB_DISABLE=1 \
NCCL_CUMEM_ENABLE=0 \
vllm serve /path/to/Huihui-Mistral-Medium-3.5-128B-abliterated-NVFP4 \
  --served-model-name huihui-mistral-medium-3.5-128b-nvfp4 \
  --trust-remote-code \
  --tensor-parallel-size 4 \
  --quantization modelopt \
  --gpu-memory-utilization 0.90 \
  --max-model-len 8192 \
  --max-num-seqs 8 \
  --disable-custom-all-reduce \
  --host 127.0.0.1 \
  --port 8000

Context Length

Measured with gpu_memory_utilization=0.90, CUDA Graph enabled.

vLLM reported for TP=2:

available KV cache memory: 26.25 GiB
estimated maximum model length: 156384

vLLM reported for TP=4 at --max-model-len 262144:

model loading took 28.47 GiB per GPU
available KV cache memory: 54.92 GiB
GPU KV cache size: 654,384 tokens
maximum concurrency for 262,144 tokens per request: 2.50x

Practical recommendation:

• Use 8192 for interactive WebUI and concurrent serving.
• Use 131072 for long-context tests.
• Treat 156384 as the measured theoretical ceiling for this TP=2/96GiBx2 setup.
• Use TP=4 for the full 262144 context length. Longer contexts require overriding the model config and were not validated.

Throughput TP=2

vLLM online serving benchmark, OpenAI chat endpoint:

• Dataset: random
• Input length: 1024 tokens
• Output length: 256 tokens
• Prompts: 16
• --ignore-eos
• --temperature 0
• Server: --max-model-len 8192 --max-num-seqs 8

Raw benchmark files:

/media/tonoken/CT4000/benchmarks/huihui_nvfp4_tp2/serve_c1.json
/media/tonoken/CT4000/benchmarks/huihui_nvfp4_tp2/serve_c2.json
/media/tonoken/CT4000/benchmarks/huihui_nvfp4_tp2/serve_c4.json
/media/tonoken/CT4000/benchmarks/huihui_nvfp4_tp2/serve_c8.json

Throughput TP=4

vLLM online serving benchmark, OpenAI chat endpoint:

• Dataset: random
• Input length: 1024 tokens
• Output length: 256 tokens
• Prompts: 16
• --ignore-eos
• --temperature 0
• Server: --max-model-len 8192 --max-num-seqs 8

Multimodal Notes

This model includes Pixtral vision components:

• model.vision_tower
• model.multi_modal_projector
• PixtralProcessor
• image tokens: [IMG], [IMG_BREAK], [IMG_END]

The vision tower must also have GGUF Q/K row permutation repaired. Text-only repair of model.language_model.layers.*.self_attn.{q,k}_proj is not enough. Without repairing:

model.vision_tower.transformer.layers.*.attention.{q,k}_proj

the model may identify coarse colors but fail badly on UI/OCR-like images.

After repairing the vision tower Q/K tensors, the model correctly read a KIMI Code console screenshot including:

• KIMI Code
• Console
• Weekly usage
• Rate limit details
• Moderato
• K2.6
• API Keys

Known Caveats

• ModelOpt NVFP4 checkpoint support in vLLM is experimental.
• --reasoning-parser mistral did not work with this HF-converted directory because vLLM expected a MistralTokenizer-compatible tokenizer file.
• Reasoning High can be passed through chat_template_kwargs, but reasoning text may appear in normal assistant content rather than a separate API field.
• TP=2 required NCCL P2P/NVLS/IB/CUMEM workarounds on the tested system.

Support the Base Model Author

• Ko-fi: https://ko-fi.com/huihuiai
• Bitcoin: bc1qqnkhuchxw0zqjh2ku3lu4hq45hc6gy84uk70ge