markmonger/Qwen3.6-40B-Claude-4.6-Opus-Deckard-Heretic-Uncensored-Thinking-iMatrix-MTP-GGUF

license: apache-2.0
tags:

• qwen3_5
• qwen3.6
• gguf
• mtp
• speculative-decoding
• fine-tune
• unsloth
• heretic
• uncensored
• abliterated
• multi-stage tuned
• 40B
• dense
• vision
• multimodal
• mmproj
• long-context
  base_model: DavidAU/Qwen3.6-40B-Claude-4.6-Opus-Deckard-Heretic-Uncensored-Thinking-NEO-CODE-Di-IMatrix-MAX-GGUF
  datasets:
  • TeichAI/claude-4.5-opus-high-reasoning-250x
  • DavidAU/PkDick-Deckard-5-Datasets
  • allenai/WildChat-1M
    pipeline_tag: text-generation

This repository represents an effort to graft and re-train an MTP layer for DavidAU's Qwen3.6-40B model, inspired by Piehsoft's untrained MTP grafts.

Currently only offering q8_0 quantizations, but this will eventually expand once MTP retraining is finalized.

Version 1 (complete)

A simple retraining attempt that used the following procedure:

1. Ran the 40B model on 117 random prompts
2. Record what it was thinking at every position by introspecting llama.cpp
   • Specifically, recorded which 64 tokens it thought were most likely next and how confident it was in each.
   • Note: Substituted h_t with embedding(input_token) to move forward with the data that was immediately accessible.
3. Use the recorded data to retrain the MTP layer: align what the MTP's predictions with reality
   • This data did not include hidden states, but since the trained head is shown real-world verifier logprobs during training its bias toward the most common tokens is effectively re-calibrated.

Compared to Piehsoft's untrained MTP head, this methodology

• Significantly improved acceptances rates in structured text / coding scenarios
• Provided little to no impact on high-entropy scenarios such as creative writing / casual chat.

Ultimately proved that improving acceptance rates via retraining was possible. Further improvements seem to require:

• Larger corpus (dataset)
• Higher text entropy within corpus
• Most importantly, data collection of the verifier hidden states

Version 2 (in-progress)

EDIT: Training using hidden states ended up generating significantly worse results than the methodology used in v1. Alternative paths are being considered, but this may potentially end as a stalemate.

Training is in-progress, which builds on-top of the v1. This version likely won't be a final version, but rather will be used to help steer direction. The main changes will be:

• Collect data using a modified version of llama.cpp that exposes data from the 40B model's hidden states (h_t)
• Use a dataset with high entropy: WildChat-1M
  • Not training on the full dataset (would take a long time), will use a subset of randomly selected samples.
• Increase epochs from 3 to 5 (and tweak a few other training settings)