base_model: swdq/MiniCPM-V-4.7-35B-A3B-abliterated
base_model_relation: quantized
library_name: gguf
license: apache-2.0
language:

• en
• zh
  pipeline_tag: image-text-to-text
  quantized_by: trithemius
  tags:
• gguf
• llama-cpp
• vision
• multimodal
• minicpm-v
• moe
• abliterated
• uncensored

MiniCPM-V 4.7 35B-A3B Abliterated — GGUF

Mixed Q3_K Expert Quantization · Importance Matrix · Multimodal Inference

A compact, selectively quantized GGUF conversion of swdq/MiniCPM-V-4.7-35B-A3B-abliterated, built for efficient local inference with llama.cpp.

This release combines the vision-language capabilities of MiniCPM-V 4.7 35B-A3B with a mixed-precision quantization strategy that targets the model's largest group of tensors: its Mixture-of-Experts (MoE) feed-forward projections.

Instead of quantizing every tensor uniformly, the conversion uses Q3_K for the expert gate, up, and down projections, with the remaining tensors following the Q4_K_M default quantization mixture. An importance matrix (imatrix) is used during quantization to help preserve model quality.

The result is a substantially smaller model than the original full-precision weights, while retaining higher precision for tensors outside the explicitly targeted expert projections.

✨ Highlights

• 🧠 35B-A3B Mixture-of-Experts architecture — approximately 35 billion total parameters, with around 3 billion parameters active per token.
• 📦 Mixed Q3_K / Q4_K_M quantization — lower-bit quantization is concentrated on the expert projections that dominate the parameter count.
• 🎯 Importance-matrix-guided conversion — an imatrix is supplied to llama-quantize to inform quantization.
• 👁️ Vision-language model — designed for visual question answering, image description, document understanding, and other multimodal tasks supported by the underlying architecture and runtime.
• 🚀 GPU-accelerated inference — configurable GPU offloading and Flash Attention through llama.cpp.
• 🖥️ OpenAI-compatible API server — serve the model locally using llama-server.
• 🔓 Abliterated source model — derived from a refusal-ablated variant of MiniCPM-V 4.7.

Important: This is a quantized conversion of an existing model, not a newly trained model. Its behavior, capabilities, and limitations are inherited primarily from the upstream abliterated model.

📥 Download

Hugging Face repository:
trithemius/MiniCPM-V-4.7-35B-A3B-abliterated-gguf

Included files

The mixed-quantization model and vision projector are the files required for the recommended inference configuration. The F16 file is included as a reference artifact and is not required to run the quantized model.

File sizes are approximate and reflect the published artifacts; actual sizes may differ slightly depending on how they are reported.

Recommended download

For the compact multimodal configuration, download:

1. MiniCPM-V-4_7-35B-A3B-abliterated-mix_q3gud_im.gguf
2. mmproj-model-f16.gguf

Keep both files in an accessible directory. The main model and the vision projector serve different purposes, and the main GGUF alone is not sufficient for image input.

🧩 About the model

MiniCPM-V 4.7 35B-A3B

The underlying architecture belongs to the MiniCPM-V family developed by OpenBMB. The 35B-A3B variant is a multimodal Mixture-of-Experts model.

Unlike a conventional dense model, an MoE model contains many expert networks but activates only a subset for a given token. The distinction between total and active parameters helps explain its computational profile, but it does not mean the full set of weights can be omitted from storage.

The original model supports visual understanding tasks including single-image and multi-image interpretation, document analysis, and video understanding in compatible inference environments.

Official resources:

• Original MiniCPM-V 4.7 35B-A3B model
• Upstream model card and documentation
• Source abliterated model by swdq

About the abliterated variant

This conversion uses the weights from swdq/MiniCPM-V-4.7-35B-A3B-abliterated, rather than the untouched upstream model.

The source author describes applying refusal ablation to the language model while leaving the vision tower unchanged. The source model also explicitly warns that its safety alignment is intentionally weakened.

Consequently, this release should not be treated as equivalent to the original instruction-tuned model in refusal behavior or safety characteristics.

⚙️ Quantization strategy

Why mixed quantization?

A large proportion of the 35B-A3B model's weights belong to its MoE expert tensors. These tensors account for approximately 92% of the weights according to the quantization strategy used for this release.

This creates an opportunity to reduce storage requirements without applying the same low-bit format indiscriminately to every tensor.

The approach used here is:

• Expert gate projections: Q3_K
• Expert up projections: Q3_K
• Expert down projections: Q3_K
• Remaining tensors: the default Q4_K_M quantization mixture
• Quantization guidance: an importance matrix

The goal is to concentrate the more aggressive quantization where it provides the greatest reduction in model size while retaining the Q4_K_M default mixture elsewhere.

Tensor selection

The conversion uses the following tensor-types.txt rule:

ffn_(gate|up|down)_exps=Q3_K

This expression targets the expert gate, up, and down projection tensor names recognized by the quantizer. All other tensors follow the default quantization selection.

This is a selective mixed-quantization configuration, not a uniform Q3_K conversion.

Why use an importance matrix?

Quantization approximates the original numerical values of model weights using a lower-precision representation. Different tensors and weight elements can have different effects on the model's output.

An importance matrix supplies additional information to the quantization process, helping it account for the relative importance of weights when choosing quantized representations.

For this conversion, the imatrix is supplied directly to llama-quantize:

--imatrix /path/to/v47-35a3.imatrix.gguf

The imatrix is an input to the conversion process; it is not a runtime dependency of the resulting GGUF.

Quality and size goals

The mixed-quantization method was selected with the aim of approaching Q4_K_M quality while substantially reducing the size of the resulting model.

The conversion recipe reports benchmark results comparable to Q4_K_M for this configuration. However, no independent benchmark measurements are claimed here for the published abliterated GGUF. Results can vary with the benchmark, inference backend, runtime version, and evaluation settings.

📊 Quantization at a glance

The active parameter count describes the MoE execution architecture. It is not the same as the total model storage requirement.

🚀 Quick start with llama.cpp

Use a recent build of llama.cpp including PR #29416 with support for the MiniCPM-V 4.7 architecture and its multimodal projector.

Multimodal architecture support can depend on the specific llama.cpp revision. If the model fails to load, verify that your build supports this exact model format rather than assuming that every older GGUF-compatible binary supports it.

1. Prepare the files

For example, arrange the downloaded files as follows:

/path/to/models/minicpm-v-4.7-35b-a3b-abliterated/
├── MiniCPM-V-4_7-35B-A3B-abliterated-mix_q3gud_im.gguf
└── mmproj-model-f16.gguf

The paths above are examples. Replace them with the actual paths on your system.

2. Start the server

The following command reproduces the recommended serving configuration:

llama-server -m /path/to/models/minicpm-v-4.7-35b-a3b-abliterated/MiniCPM-V-4_7-35B-A3B-abliterated-mix_q3gud_im.gguf --mmproj /path/to/models/minicpm-v-4.7-35b-a3b-abliterated/mmproj-model-f16.gguf --ctx-size 262000 --n-gpu-layers 999 --flash-attn on --host 0.0.0.0 --port 8000 --no-context-shift

Parameter explanation

Context-size note: --ctx-size 262000 is the requested runtime configuration, not a guarantee that every prompt can use the entire context. Actual usable context depends on the model's supported limits, the runtime build, and available memory. The KV cache can consume substantial additional RAM or VRAM at large context sizes.

GPU note: --n-gpu-layers 999 requests maximum layer offloading; it does not guarantee that the complete model, vision projector, KV cache, and all runtime allocations will fit in GPU memory. Adjust GPU offloading and context size for your hardware.

Network note: Binding to 0.0.0.0 exposes the service on every network interface. Use a firewall or a trusted private network, and do not expose the endpoint publicly without appropriate access controls.

Once the server is running, its HTTP endpoint is:

http://localhost:8000

The server exposes llama.cpp's HTTP API, including its OpenAI-compatible chat-completions endpoint when supported by the installed build.

🖼️ Multimodal inference

The separate mmproj-model-f16.gguf file is essential for image understanding.

It provides the vision projection component used to connect visual representations with the language model. Loading the main model without the projector does not provide the complete multimodal configuration.

Example: ask a question about an image

With the server running, submit a chat-completions request containing an image and a text prompt:

curl -s http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"local-model","messages":[{"role":"user","content":[{"type":"text","text":"Describe this image in detail. Identify the main objects, visible text, and the relationships between them."},{"type":"image_url","image_url":{"url":"https://example.com/image.jpg"}}]}],"max_tokens":512}'

Replace https://example.com/image.jpg with a reachable image URL. This is an illustrative request; the server's image-loading capabilities and accepted request formats depend on the llama.cpp build.

For private or local images, use an image-input format supported by your installed version of llama.cpp.

Suggested multimodal tasks

The underlying model family is designed for visual-language tasks such as:

• Image understanding: describe a scene, identify objects, and explain spatial relationships.
• Document analysis: extract information from pages, forms, receipts, and other document images.
• OCR-oriented tasks: read and summarize visible text, subject to image quality and text complexity.
• Visual question answering: answer questions about details in an image.
• Multiple-image comparison: compare images when the runtime supports the model's multi-image input format.
• Video understanding: possible through compatible model and preprocessing pipelines; do not assume that every llama.cpp build accepts arbitrary video input directly.

For fine visual details, small text, or dense documents, image resolution and preprocessing can have a significant impact on results. Quantization is only one factor in multimodal quality.

💬 Text-only inference

The model can also be used for ordinary text-based conversations through the same server.

For example:

curl -s http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"local-model","messages":[{"role":"user","content":"Explain how a Mixture-of-Experts language model differs from a dense transformer."}],"max_tokens":512}'

The model value in these examples is a placeholder. Use the model identifier expected by your server configuration if it differs.

💾 Memory and hardware considerations

The approximately 15 GB main GGUF is much smaller than the 65 GB F16 reference artifact, but file size is not the same as total runtime memory consumption.

Plan for additional memory for:

• The multimodal projector.
• The KV cache, which grows with context length and active sequences.
• Temporary computation buffers.
• CUDA or other GPU backend allocations.
• Image preprocessing and visual-token processing.
• Server overhead and concurrent requests.

The full 262,000-token context configuration can substantially increase memory consumption compared with short prompts.

Practical recommendations

1. Start with a smaller context size and increase it after verifying memory usage.
2. Use GPU offloading appropriate to the available VRAM.
3. Confirm that Flash Attention is supported and enabled by your build.
4. Monitor both system RAM and GPU memory during long-context inference.
5. Test image inputs separately from text-only inference.
6. Keep the main model and projector from the same published release together.

No universal minimum VRAM figure is specified here because feasibility depends on context length, backend, offloading strategy, image workload, and concurrency.

🛠️ Reproducing the quantization

The following describes the mixed-quantization recipe used for this release.

Requirements

• The F16 GGUF converted from the source abliterated model.
• A compatible llama-quantize binary.
• An importance matrix for the MiniCPM-V 4.7 35B-A3B model.
• Sufficient disk space for the input and output artifacts.

The imatrix must correspond appropriately to the model being quantized. Do not assume that an arbitrary importance matrix from another model or revision will be suitable.

1. Create the tensor override file

Create tensor-types.txt with the following content:

ffn_(gate|up|down)_exps=Q3_K

This sets Q3_K for matching expert projection tensors. The base quantization type remains Q4_K_M for tensors not overridden by this rule.

2. Run the quantizer

./llama-quantize --imatrix /path/to/v47-35a3.imatrix.gguf --tensor-type-file ./tensor-types.txt /path/to/MiniCPM-V-4.7-35B-A3B-abliterated-F16.gguf /path/to/MiniCPM-V-4.7-35B-A3B-abliterated-mix_q3gud_im.gguf Q4_K_M 32

Command breakdown

The tensor-type override is what makes this a mixed-quantization conversion. Without the override file, the quantizer would use its normal Q4_K_M selection instead of explicitly assigning Q3_K to the selected expert projections.

The precise tensor naming and override matching behavior should be checked against the llama-quantize version used for the conversion.

Reproducibility note

For a reproducible conversion, record the llama.cpp commit or release, source model revision, imatrix provenance, and exact quantization command. These details help distinguish changes caused by the quantization procedure from changes in the underlying model or runtime.

📈 Benchmarking and quality evaluation

The motivation for this configuration is to obtain a substantial reduction in model size while retaining useful model quality.

The mixed-quantization approach is intended to approximate Q4_K_M quality by applying Q3_K only to the large expert projection tensors and using an importance matrix.

However, benchmark equivalence should not be interpreted as a guarantee that every task or prompt will produce identical results. Quantization can affect tasks differently, and multimodal quality can also depend on the projector, image preprocessing, and runtime implementation.

For a rigorous comparison, evaluate this release against the original Q4_K_M conversion using:

• Identical evaluation datasets and prompts.
• The same llama.cpp build and inference settings.
• Consistent sampling parameters and output limits.
• Separate text-only and vision-language evaluations.
• Peak RAM and VRAM consumption.
• Tokens per second and time to first token.
• Long-context stability.

No independently measured scores or throughput figures are asserted in this model card.

⚠️ Intended use and limitations

This model is intended for local experimentation, research, development, and multimodal inference in environments compatible with its architecture and GGUF format.

Safety considerations

The underlying source model is explicitly refusal-ablated. Its safety alignment has been intentionally weakened, and it may generate harmful, offensive, or illegal content.

Users are responsible for how they deploy and use the model. Appropriate safeguards, output validation, and access controls should be considered for applications involving sensitive or consequential decisions.

Technical limitations

• Quantization can change model behavior relative to the original weights.
• The mixed-quantization recipe does not guarantee identical performance on every benchmark.
• The vision projector must be supplied for multimodal inference.
• Long contexts can increase memory requirements considerably.
• Image and video capabilities depend on runtime support and input preprocessing.
• Model outputs can be incorrect, incomplete, or misleading.
• This conversion does not independently validate the accuracy or safety of the source model.

📜 License and attribution

This release is derived from two upstream works:

Base model

OpenBMB/MiniCPM-V-4.7-35B-A3B

Abliterated model used for this conversion

swdq/MiniCPM-V-4.7-35B-A3B-abliterated

Both upstream model cards identify their license as Apache-2.0. Users should consult the original repositories and applicable license files for the authoritative licensing terms, attribution requirements, and any relevant notices.

This repository provides a GGUF conversion and quantization of the abliterated source model. It is not an official OpenBMB release and should not be interpreted as endorsed by OpenBMB or the source model author.

Acknowledgements

• OpenBMB — MiniCPM-V model family and original model weights.
• swdq — refusal-ablated source model.
• llama.cpp contributors — GGUF inference and quantization infrastructure.

🔗 Links

This repository is an independent GGUF conversion of an existing abliterated multimodal model. Please consult the upstream model cards for original architecture details, safety considerations, and licensing information.