license: apache-2.0
base_model: dima6312/Flint-40B-A2.5B-abliterated
base_model_relation: quantized
library_name: transformers
pipeline_tag: text-generation
tags:
- flint
- fp8
- w8a8
- uncensored
- compressed-tensors
- calibrated-kv-cache
- moe
- abliterated
datasets: - HuggingFaceH4/ultrachat_200k
Flint 40B-A2.5B Abliterated - FP8 with calibrated FP8 KV scales
Quantization of dima6312/Flint-40B-A2.5B-abliterated
Uncensored model: the language checkpoint has undergone abliteration
to reduce refusal behavior. Treat outputs as untrusted, apply application-level
safeguards, and do not assume the model will decline harmful requests.
User responsibility: this model is provided without warranty. The
creators, uploaders, and maintainers are not responsible or liable for what
others generate, publish, deploy, or otherwise do with this abliterated model.
Users must operate it responsibly, apply appropriate safeguards, comply with
applicable law, and respect third-party rights. This model is for research
purposes only and is not intended for production use.
Format
- Native LLM Compressor FP8_DYNAMIC: E4M3FN weights with per-output-channel BF16 scales and dynamic per-token FP8 activations. This is not MXFP8 or static activation calibration.
- 36,439 quantized weight tensors. Embeddings, output head, routers, norms and learned attention sinks remain BF16: 295 protected tensors.
- 98 calibrated BF16 KV scales: separate scalar K and V scales for all 49 attention layers.
- The 14 full-attention/NoPE layers, 35 RoPE sliding-window layers, 513-token sliding window, QK normalization, learned sinks, sigmoid top-6 routing and shared experts are preserved.
- No new abliteration was performed.
Calibration
Native compressed-tensors cache hooks observe actual post-normalization/post-RoPE keys and raw values, before KV repetition. The calibration source is the already-abliterated BF16 parent. FP8 weights are produced separately by native model_free_ptq in one-shard-at-a-time CPU streaming, avoiding a whole-model GPU compression/save allocation peak. Native LLM Compressor retains scales in the BF16 observed dtype; scalar values are checked against the corresponding BF16-rounded calibration extrema divided by 448.
- Dataset:
HuggingFaceH4/ultrachat_200k - Revision:
8049631c405ae6576f93f445c6b8166f76f5505a - Split: first 512 rows of
train_sft, shuffled with seed 42 - Samples: 512
- Tokens: 543,281
- Maximum sample length: 2,048
- Batch size: 1; no padding, persistent KV history, gradient calculations or CUDA graphs
- Tokenization: the source Flint tokenizer and unchanged chat template
- Data path: a native PyTorch DataLoader preserves the pinned sample order without padding; all 512 actual model inputs are checked token-for-token; their hashes are reconstructed from those verified identical IDs in calibration-input-coverage.json
- KV observer: native
static_minmax, accumulating global extrema across the corpus
FP8 weight scales come from each weight channel; they do not require dataset calibration. Expert activation quantization is dynamic. All routed-expert weights are quantized while calibration preserves normal routing.
See calibration-provenance.json, kv-calibration-evidence.json, quantization-recipe.json and artifact-validation.json.
Loading requirements
Validated tool versions: LLM Compressor 0.13.0, compressed-tensors 0.18.0, Transformers 5.13.1, PyTorch 2.11.0.
Do not use an unadapted ordinary Transformers load. Flint's source class fuses expert parameters during load; this artifact uses the compressor's concrete per-expert 2D checkpoint layout. The supplied helper registers the native pre-load mappings before construction:
import torch
from flint_loader import load_model
model = load_model(
"./Flint-40B-A2.5B-abliterated-FP8-Calibrated",
dtype=torch.bfloat16,
device_map="auto",
)
The modeling code contains one argument-binding repair: passing past_key_values into attention by keyword, so native cache hooks do not collide with positional arguments. A focused executable source-versus-repaired model check showed identical logits.
This helper is a correctness/loading path, not a claim of accelerated FP8 execution. Transformers/compressed-tensors cache hooks can quantize and dequantize values without storing the resident cache as FP8 bytes. An inference engine must independently support Flint, the quantized expert layout, the hybrid learned-sink attention, and calibrated KV-scale ingestion.
Validation limits
The artifact checks cover shard/index consistency, tensor dtypes and shapes, finite quantized weights, positive finite scales, exact equality of every protected tensor, and native quantization-oracle comparisons at boundary layers and expert indices. Each saved KV scale is compared against independently recorded cached-tensor extrema.
A small model using the actual Flint classes passed native quantization, saving and exact reload of every tensor. Full derivative GPU inference, serving, end-to-end generation quality and benchmarks have not been tested. Calibration itself exercised the BF16 source, not an accelerated FP8 serving engine. No claim of unchanged benchmark scores or validated 262K inference is made.
The upstream SGLang serving kit documents quantized weights and KV cache as unvalidated. Its ordinary BF16 instructions are not evidence that this derivative runs without runtime integration.
Pipeline and reproducibility notes
The completed pipeline used GPU calibration of the BF16 source, followed by native CPU shard-stream FP8 weight quantization. The archived calibration script also contains a GPU compression/save branch; that branch was not used to create this release. Do not treat it as a standalone release reproduction command. The companion streaming script requires the measured calibration records described in the recipe.
The reproduce/ directory is retained from the upstream BF16 abliteration release. Its metrics, environments and shard checksums describe that parent, not these quantized shards. Verify this derivative against artifact-manifest.json.
Original model documentation
The substantive upstream model card, evaluation results, limitations, methods and credits are retained in UPSTREAM_README.md. Those results describe the upstream artifact and must not be attributed to this FP8 derivative.