license: apache-2.0
base_model: huihui-ai/Huihui-Qwen-AgentWorld-35B-A3B-abliterated
base_model_relation: quantized
tags:

• gguf
• llama.cpp
• qwen3_5_moe
• agent
• world-model
• abliterated
• uncensored
• quantized
• q3_k_m
• imatrix
  language:
• en
• zh
  pipeline_tag: text-generation

Huihui-Qwen-AgentWorld-35B-A3B-abliterated UD-Q3_K_M GGUF

An importance-matrix UD-Q3_K_M GGUF quant of
huihui-ai/Huihui-Qwen-AgentWorld-35B-A3B-abliterated.

Why this quant exists: the BF16 source is roughly 65-67 GB and is not practical on 22 GB cards. This UD-Q3_K_M build is 15.5 GiB / 16.65 GB, leaving enough working room for KV cache and compute buffers on a single 22 GB RTX 2080 Ti while preserving the abliterated AgentWorld model behavior for local agent experiments.

Lineage

• Base: Qwen/Qwen-AgentWorld-35B-A3B — Qwen AgentWorld 35B-A3B MoE model for agent and environment-simulation use cases.
• Abliteration: huihui-ai/Huihui-Qwen-AgentWorld-35B-A3B-abliterated, described upstream as an uncensored/abliterated variant with reduced refusal behavior.
• This quant: UD-Q3_K_M, produced with llama.cpp from the local F16 GGUF conversion of Huihui's BF16 release.

Quick Stats

Tensor type counts:

This tensor map follows the public UD-style Q3_K_M layout used for Qwen-AgentWorld. A few additional attention output tensors stayed at Q6_K, making this build slightly more conservative than the smallest possible Q3 variant.

llama.cpp Server

Recommended launch:

llama-server \
  -m Huihui-Qwen-AgentWorld-35B-A3B-abliterated-UD-Q3_K_M.gguf \
  -ngl 99 \
  -c 8192 \
  --parallel 1 \
  --jinja \
  --reasoning on \
  --reasoning-format deepseek \
  --temp 0.6 \
  --top-p 0.95 \
  --top-k 20

For a pinned single-GPU Vulkan setup, expose only the intended GPU before launch, for example:

CUDA_VISIBLE_DEVICES= GGML_VK_VISIBLE_DEVICES=1 llama-server \
  -m Huihui-Qwen-AgentWorld-35B-A3B-abliterated-UD-Q3_K_M.gguf \
  --host 0.0.0.0 \
  --port 8083 \
  -ngl 99 \
  -c 8192 \
  --parallel 1 \
  --jinja \
  --reasoning on \
  --reasoning-format deepseek

Important runtime note: run the server with reasoning parsing enabled. In local llama.cpp testing, server-level --reasoning off could still lead this model to emit Thinking Process: as ordinary message.content. The more reliable route was:

1. Start llama.cpp with --reasoning on --reasoning-format deepseek.
2. For direct final-answer calls, pass request-level chat_template_kwargs: {"enable_thinking": false}.
3. For thinking-mode calls, omit that override and read reasoning from message.reasoning_content.

Example direct call body:

{
  "model": "huihui-qwen-agentworld-35b-a3b-abliterated-ud-q3km",
  "messages": [
    {"role": "user", "content": "Write five concise deployment checks."}
  ],
  "chat_template_kwargs": {"enable_thinking": false},
  "temperature": 0.2,
  "max_tokens": 256
}

Local Smoke Test

Test host: dual RTX 2080 Ti, llama.cpp Vulkan backend, model loaded on one 22 GB card.

These are smoke tests, not a standardized benchmark. Quality should be validated against your own agent workload before relying on the quant in production.

Quantization Recipe

The imatrix was generated from a small synthetic calibration set focused on terminal, repo-editing, JSON extraction, tool-call style transcripts, Chinese assistant turns, and safe fictional directness prompts.

llama-imatrix \
  -m Huihui-Qwen-AgentWorld-35B-A3B-abliterated-F16.gguf \
  -f huihui-agentworld-calibration.txt \
  -o imatrix-huihui-agentworld-76.gguf \
  --output-format gguf \
  --chunks 76 \
  -c 512 \
  -ngl auto \
  --split-mode none \
  --main-gpu 0 \
  --no-ppl

llama-quantize \
  --imatrix imatrix-huihui-agentworld-76.gguf \
  --tensor-type-file ud-q3km-tensor-types.txt \
  Huihui-Qwen-AgentWorld-35B-A3B-abliterated-F16.gguf \
  Huihui-Qwen-AgentWorld-35B-A3B-abliterated-UD-Q3_K_M.gguf \
  Q3_K_M 24

The following helper files are included for reproducibility:

Abliteration Sanity

The upstream model is an abliterated build. Local safe smoke tests checked for functional directness using non-graphic fictional and agentic prompts. With request-level chat_template_kwargs: {"enable_thinking": false}, the model returned direct content without refusal phrases in all three abliterated sanity prompts.

This is not a safety guarantee. It is only a practical sanity check that the quantized GGUF still behaves like the abliterated upstream in ordinary local testing.

License

Inherits from the upstream model: Apache-2.0.

⚠ Safety: The upstream Huihui model card warns that refusal behavior and safety filtering are significantly reduced. Use this model in controlled environments and review outputs carefully.

Credits

• Source model: Qwen Team / Alibaba
• Abliteration: huihui-ai
• Quantization: this checkpoint, by coolthor
• Tooling: llama.cpp

📝 Quantized & benchmarked by ai-muninn — writeups on how it was built and how it actually runs.