license: apache-2.0
base_model: zswll2/Qwen3.5-9B-Claude-4.6-HighIQ-THINKING-HERETIC-UNCENSORED
tags:

• fp8
• quantized
• sglang
• qwen3.5
• multimodal

Qwen3.5-9B FP8 Model (SGLang Compatible)

中文版

Overview

This is an FP8 quantized version of Qwen3.5-9B-Claude-4.6-HighIQ-THINKING-HERETIC-UNCENSORED, optimized for SGLang inference.

Key Features

• FP8 Quantization: Block-wise FP8 with [128, 128] block size
• SGLang Optimized: weight_scale_inv converted to bfloat16 for SGLang compatibility
• Multimodal Support: Vision-Language model with 27-layer visual encoder
• Linear Attention: Hybrid architecture with linear and full attention layers

Model Structure

Qwen3.5-9B-FP8-SGLang/
├── config.json              # Model configuration with FP8 quantization params
├── model.safetensors        # FP8 weights (9.4 GB)
├── tokenizer.json
├── tokenizer_config.json
├── preprocessor_config.json
├── generation_config.json
├── processor_config.json
├── video_preprocessor_config.json
├── chat_template.jinja
└── vocab.json

Quantization Config

{
  "quantization_config": {
    "quant_method": "fp8",
    "activation_scheme": "dynamic",
    "weight_per_tensor": false,
    "act_per_tensor": false,
    "weight_block_size": [128, 128]
  }
}

Requirements

Hardware

• GPU: NVIDIA GPU with FP8 support (Ada Lovelace or newer recommended)
• VRAM: 10GB+ for inference
• CUDA: 12.1+

Software

Core dependencies:

sglang==0.5.9
torch==2.9.1
torchvision==0.24.1
torchaudio==2.9.1
transformers==4.57.1
tokenizers==0.22.2
flashinfer-python==0.6.4
flashinfer-cubin==0.6.4
triton==3.5.1
torchao==0.9.0
cuda-python==12.9.0
cuda-bindings==12.9.5
nvidia-cublas-cu12==12.9.1.4
nvidia-cudnn-cu12==9.16.0.29
nvidia-cuda-runtime-cu12==12.8.90
nvidia-cuda-nvrtc-cu12==12.8.93
nvidia-nccl-cu12==2.27.5
sgl-kernel==0.3.21
outlines==0.1.11
outlines_core==0.1.26
xgrammar==0.1.27
llguidance==0.7.30
compressed-tensors==0.14.0
safetensors==0.7.0
huggingface_hub==0.36.2
accelerate
pillow==11.3.0

Installation

# Create conda environment
conda create -n sglang-fp8 python=3.11 -y
conda activate sglang-fp8

# Install SGLang with CUDA 12.x
pip install sglang[all] --find-links https://flashinfer.ai/whl/cu121/torch2.4/

# Or install from source for latest features
pip install sglang[all]>=0.5.9

Usage

SGLang Server

python -m sglang.launch_server \
    --model-path ./Qwen3.5-9B-Claude-4.6-HighIQ-THINKING-HERETIC-UNCENSORED-fp8-sglang \
    --port 8000 \
    --trust-remote-code

SGLang Python API

import sglang as sgl

# Initialize engine
llm = sgl.Engine(
    model_path="./Qwen3.5-9B-Claude-4.6-HighIQ-THINKING-HERETIC-UNCENSORED-fp8-sglang",
    trust_remote_code=True,
)

# Generate response
prompt = "Hello! Please introduce yourself briefly."
outputs = llm.generate([prompt], sampling_params={"max_new_tokens": 100, "temperature": 0.0})
print(outputs[0]["text"])

OpenAI-Compatible API

After starting the server:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="none")

response = client.chat.completions.create(
    model="default",
    messages=[{"role": "user", "content": "Hello!"}],
    max_tokens=100,
)
print(response.choices[0].message.content)

Sampling Parameters

Thinking Mode (General Tasks)

sampling_params = {
    "temperature": 1.0,
    "top_p": 0.95,
    "top_k": 20,
    "min_p": 0.0,
    "presence_penalty": 1.5,
    "repetition_penalty": 1.0,
}

Thinking Mode (Coding Tasks)

sampling_params = {
    "temperature": 0.6,
    "top_p": 0.95,
    "top_k": 20,
    "min_p": 0.0,
    "presence_penalty": 0.0,
    "repetition_penalty": 1.0,
}

Instruct Mode (Non-Thinking)

sampling_params = {
    "temperature": 0.7,
    "top_p": 0.8,
    "top_k": 20,
    "min_p": 0.0,
    "presence_penalty": 1.5,
    "repetition_penalty": 1.0,
}
# Add: extra_body={"chat_template_kwargs": {"enable_thinking": False}}

Technical Details

Conversion from Transformers 5.x FP8

This model was converted from Transformers 5.x native FP8 format. The key difference:

Conversion script convert_fp8_for_sglang.py:

import torch
import safetensors
from safetensors.torch import save_file

# Convert weight_scale_inv: float32 -> bfloat16
for key in sf.keys():
    tensor = sf.get_tensor(key)
    if 'scale_inv' in key and tensor.dtype == torch.float32:
        tensors[key] = tensor.to(torch.bfloat16)
    else:
        tensors[key] = tensor

Troubleshooting

CUDA Out of Memory

# Reduce max context length
--max-model-len 8192

# Reduce memory fraction
--mem-fraction-static 0.8

Import Errors

Ensure CUDA version matches:

pip install sglang[all] --find-links https://flashinfer.ai/whl/cu121/torch2.4/

License

This model inherits the license from the base model.

References

• Qwen3.5 Official
• SGLang GitHub
• SGLang Documentation

Generated: 2026-03-08
Quantization: Transformers 5.x FineGrainedFP8Config + SGLang conversion

中文版

概述

这是 Qwen3.5-9B-Claude-4.6-HighIQ-THINKING-HERETIC-UNCENSORED 的 FP8 量化版本，专为 SGLang 推理优化。

主要特性

• FP8 量化: 块级 FP8，块大小 [128, 128]
• SGLang 优化: weight_scale_inv 已转换为 bfloat16 以兼容 SGLang
• 多模态支持: 视觉-语言模型，27层视觉编码器
• 线性注意力: 混合架构，包含线性和全注意力层

模型结构

Qwen3.5-9B-FP8-SGLang/
├── config.json              # 模型配置（含 FP8 量化参数）
├── model.safetensors        # FP8 权重 (9.4 GB)
├── tokenizer.json
├── tokenizer_config.json
├── preprocessor_config.json
├── generation_config.json
├── processor_config.json
├── video_preprocessor_config.json
├── chat_template.jinja
└── vocab.json

量化配置

{
  "quantization_config": {
    "quant_method": "fp8",
    "activation_scheme": "dynamic",
    "weight_per_tensor": false,
    "act_per_tensor": false,
    "weight_block_size": [128, 128]
  }
}

环境要求

硬件

• GPU: 支持 FP8 的 NVIDIA GPU（推荐 Ada Lovelace 或更新）
• 显存: 10GB+ 用于推理
• CUDA: 12.1+

软件

核心依赖:

sglang==0.5.9
torch==2.9.1
torchvision==0.24.1
torchaudio==2.9.1
transformers==4.57.1
tokenizers==0.22.2
flashinfer-python==0.6.4
flashinfer-cubin==0.6.4
triton==3.5.1
torchao==0.9.0
cuda-python==12.9.0
cuda-bindings==12.9.5
nvidia-cublas-cu12==12.9.1.4
nvidia-cudnn-cu12==9.16.0.29
nvidia-cuda-runtime-cu12==12.8.90
nvidia-cuda-nvrtc-cu12==12.8.93
nvidia-nccl-cu12==2.27.5
sgl-kernel==0.3.21
outlines==0.1.11
outlines_core==0.1.26
xgrammar==0.1.27
llguidance==0.7.30
compressed-tensors==0.14.0
safetensors==0.7.0
huggingface_hub==0.36.2
accelerate
pillow==11.3.0

安装

# 创建 conda 环境
conda create -n sglang-fp8 python=3.11 -y
conda activate sglang-fp8

# 安装 SGLang (CUDA 12.x)
pip install sglang[all] --find-links https://flashinfer.ai/whl/cu121/torch2.4/

# 或安装最新版本
pip install sglang[all]>=0.5.9

使用方法

SGLang 服务器

python -m sglang.launch_server \
    --model-path ./Qwen3.5-9B-Claude-4.6-HighIQ-THINKING-HERETIC-UNCENSORED-fp8-sglang \
    --port 8000 \
    --trust-remote-code

SGLang Python API

import sglang as sgl

# 初始化引擎
llm = sgl.Engine(
    model_path="./Qwen3.5-9B-Claude-4.6-HighIQ-THINKING-HERETIC-UNCENSORED-fp8-sglang",
    trust_remote_code=True,
)

# 生成响应
prompt = "你好！请简要介绍一下你自己。"
outputs = llm.generate([prompt], sampling_params={"max_new_tokens": 100, "temperature": 0.0})
print(outputs[0]["text"])

OpenAI 兼容 API

启动服务器后：

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="none")

response = client.chat.completions.create(
    model="default",
    messages=[{"role": "user", "content": "你好！"}],
    max_tokens=100,
)
print(response.choices[0].message.content)

采样参数

思考模式（通用任务）

sampling_params = {
    "temperature": 1.0,
    "top_p": 0.95,
    "top_k": 20,
    "min_p": 0.0,
    "presence_penalty": 1.5,
    "repetition_penalty": 1.0,
}

思考模式（编程任务）

sampling_params = {
    "temperature": 0.6,
    "top_p": 0.95,
    "top_k": 20,
    "min_p": 0.0,
    "presence_penalty": 0.0,
    "repetition_penalty": 1.0,
}

指令模式（非思考）

sampling_params = {
    "temperature": 0.7,
    "top_p": 0.8,
    "top_k": 20,
    "min_p": 0.0,
    "presence_penalty": 1.5,
    "repetition_penalty": 1.0,
}
# 添加: extra_body={"chat_template_kwargs": {"enable_thinking": False}}

技术细节

从 Transformers 5.x FP8 转换

此模型从 Transformers 5.x 原生 FP8 格式转换而来。关键差异：

转换脚本 convert_fp8_for_sglang.py:

import torch
import safetensors
from safetensors.torch import save_file

# 转换 weight_scale_inv: float32 -> bfloat16
for key in sf.keys():
    tensor = sf.get_tensor(key)
    if 'scale_inv' in key and tensor.dtype == torch.float32:
        tensors[key] = tensor.to(torch.bfloat16)
    else:
        tensors[key] = tensor

故障排除

CUDA 内存不足

# 减少最大上下文长度
--max-model-len 8192

# 减少内存占用比例
--mem-fraction-static 0.8

导入错误

确保 CUDA 版本匹配：

pip install sglang[all] --find-links https://flashinfer.ai/whl/cu121/torch2.4/

许可证

此模型继承基础模型的许可证。

参考资料

• Qwen3.5 官方
• SGLang GitHub
• SGLang 文档

生成时间: 2026-03-08
量化工具: Transformers 5.x FineGrainedFP8Config + SGLang 格式转换