base_model: electroglyph/Muse-Glimmer-30B-uncensored2
library_name: transformers
quantized_by: Intel AutoRound
scheme: W4A16
format: auto_round
Muse-Glimmer-30B-uncensored2-AutoRound-W4A16-G128
AutoRound W4A16 (4-bit weights / 16-bit activations, group_size=128, symmetric)
quantization of electroglyph/Muse-Glimmer-30B-uncensored2.
- Tool: Intel AutoRound (
scheme="W4A16",format="auto_round") - Real SignRound tuning on calibration data (not RTN).
- Calibration:
HuggingFaceH4/ultrachat_200k:num=384,nvidia/OpenCodeInstruct:num=128(nsamples=512, seqlen=2048, iters=1000).
Quantization config
{
"scheme": "W4A16",
"format": "auto_round",
"bits": 4,
"group_size": 128,
"sym": true,
"nsamples": 512,
"seqlen": 2048,
"iters": 1000,
"lr": auto,
"batch_size": 8,
"gradient_accumulate_steps": 1,
"dataset": "HuggingFaceH4/ultrachat_200k:num=384,nvidia/OpenCodeInstruct:num=128",
"device_map": "auto",
"model_dtype": "bf16",
"low_gpu_mem_usage": false,
"low_cpu_mem_usage": true,
"enable_torch_compile": true,
"seed": 42
}
Run with vLLM (native kernel, no --quantization flag)
auto_round W4A16 is in vLLM's Intel Quantization Support
(W4A16 weight-only) and is served directly:
pip install vllm
vllm serve electroglyph/Muse-Glimmer-30B-uncensored2-AutoRound-W4A16-G128 --gpu-memory-utilization 0.8 --max-model-len 32768
Python:
from vllm import LLM, SamplingParams
llm = LLM(model="electroglyph/Muse-Glimmer-30B-uncensored2-AutoRound-W4A16-G128")
print(llm.generate(["Hello, my name is"], SamplingParams(temperature=0.6, top_p=0.95))[0].outputs[0].text)
Run with Transformers
pip install -U "transformers>=5.22.2" auto-round # auto-round registers the auto_round format
from transformers import AutoProcessor, AutoModelForImageTextToText
processor = AutoProcessor.from_pretrained("electroglyph/Muse-Glimmer-30B-uncensored2-AutoRound-W4A16-G128")
model = AutoModelForImageTextToText.from_pretrained("electroglyph/Muse-Glimmer-30B-uncensored2-AutoRound-W4A16-G128", device_map="auto", torch_dtype="auto")
Best practice sampling (per the source model card): temperature=1.0, top_p=0.95, top_k=64. Reasoning strength is set in the system prompt
(Reasoning strength: <low|medium|high|xhigh>); use high/xhigh for coding and agentic tasks.