base_model: electroglyph/LFM2.5-24B-A2B-uncensored
library_name: transformers
quantized_by: Intel AutoRound
scheme: W4A16
format: auto_round
LFM2.5-24B-A2B-uncensored-AutoRound-W4A16
AutoRound W4A16 (4-bit weights / 16-bit activations, group_size=128, symmetric)
quantization of electroglyph/LFM2.5-24B-A2B-uncensored.
- Tool: Intel AutoRound (
scheme="W4A16",format="auto_round") - Real SignRound tuning on calibration data (not RTN).
- Calibration:
HuggingFaceH4/ultrachat_200k:num=384,nvidia/OpenCodeInstruct:num=128(nsamples=512, seqlen=2048, iters=1000).
Quantization config
{
"scheme": "W4A16",
"format": "auto_round",
"bits": 4,
"group_size": 128,
"sym": true,
"nsamples": 512,
"seqlen": 2048,
"iters": 1000,
"lr": auto,
"batch_size": 8,
"gradient_accumulate_steps": 1,
"dataset": "HuggingFaceH4/ultrachat_200k:num=384,nvidia/OpenCodeInstruct:num=128",
"device_map": "auto",
"model_dtype": "bf16",
"low_gpu_mem_usage": false,
"low_cpu_mem_usage": true,
"enable_torch_compile": true,
"seed": 42
}
Run with vLLM (native kernel, no --quantization flag)
auto_round W4A16 is in vLLM's Intel Quantization Support
(W4A16 weight-only) and is served directly:
pip install vllm
vllm serve electroglyph/LFM2.5-24B-A2B-uncensored-AutoRound-W4A16 --gpu-memory-utilization 0.8 --max-model-len 4096
Python:
from vllm import LLM, SamplingParams
llm = LLM(model="electroglyph/LFM2.5-24B-A2B-uncensored-AutoRound-W4A16")
print(llm.generate(["Hello, my name is"], SamplingParams(temperature=0.6, top_p=0.95))[0].outputs[0].text)
Run with Transformers
pip install auto-round # registers the auto_round format with transformers
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("electroglyph/LFM2.5-24B-A2B-uncensored-AutoRound-W4A16")
model = AutoModelForCausalLM.from_pretrained("electroglyph/LFM2.5-24B-A2B-uncensored-AutoRound-W4A16", device_map="auto", torch_dtype="auto")