language:
- en
base_model: - concedo/Huihui-GLM-4.5-Air-abliterated-lossytensors
library_name: transformers
license: mit
pipeline_tag: text-generation
finetuned_by: Koshkasa
base_model_relation: finetune
datasets: - ChaoticNeutrals/Creative_Writing-ShareGPT
- hieunguyenminh/roleplay
- open-r1/Mixture-of-Thoughts
- theblackcat102/evol-codealpaca-v1
This is a 50% REAP of lossytensors based on huihui's derestriction of GLM 4.5 Air, uploaded by concedo.
Since the previous (failed-ish) attempt the combined calibration dataset was generalized (while still having a 60%/40% creative bias) and re-wrapped on source basis. Activations were also recorded with higher length (8192 vs 3072).
As before, a sparse observer was utilized instead of materializing the entire [128, dim, tok] block when only top_k ones are actually routed.