license: llama3.1
base_model:
- mlabonne/Meta-Llama-3.1-8B-Instruct-abliterated
base_model_relation: quantized
pipeline_tag: text-generation
library_name: numallama
tags: - numallama
- cpu
- arm64
- aarch64
- numa
- text-generation
- abliterated
- uncensored
Llama-3.1-8B-Instruct-abliterated-numallama
Built with Llama
mlabonne/Meta-Llama-3.1-8B-Instruct-abliterated (an abliterated version of meta-llama/Llama-3.1-8B-Instruct), converted for numallama: a C inference engine for CPU servers, with NUMA-aware execution across all sockets of multi-socket machines.
This is an abliterated model: the refusal behaviour of the original was removed by mlabonne/Meta-Llama-3.1-8B-Instruct-abliterated. It will follow requests that the original refuses. You are responsible for how you use it and must still follow the license and its acceptable use policy. For the original behaviour use Llama-3.1-8B-Instruct-numallama.
These files are not GGUF. They run only with numallama (they do not work with llama.cpp, Ollama, LM Studio or transformers).
Files
| File | Weights | Size | Decode speed* |
|---|---|---|---|
llama-3.1-8b-instruct-abliterated-q4.bin |
Q4 | 5.5 GB | 57 tok/s |
llama-3.1-8b-instruct-abliterated-q8.bin |
Q8 | 8.5 GB | 40 tok/s |
tokenizer.bin |
tokenizer | 1.9 MB |
Q4 is asymmetric 4-bit (groups of 32, fp16 scale and min), quantized with an importance matrix collected by numallama on English, Russian, Chinese and code text. Q8 is int8 with per-group scales.
Q4 is faster and smaller; Q8 is closer to the original.
* tokens/s for a single chat, measured on 2 × Ampere Altra Max M128-30 (256 Neoverse N1 cores), both sockets. Other machines will differ.
Run
Linux. Optimized for ARM64 with NEON dot product (Ampere Altra, AWS Graviton, other Neoverse cores); other CPUs, including x86-64, run portable C code that is not optimized.
git clone https://github.com/miloserdev/numallama && cd numallama && make
pip install huggingface_hub
hf download miloserdev/Llama-3.1-8B-Instruct-abliterated-numallama --local-dir models/Llama-3.1-8B-Instruct-abliterated-numallama
# chat in the terminal
./runq models/Llama-3.1-8B-Instruct-abliterated-numallama/llama-3.1-8b-instruct-abliterated-q4.bin -z models/Llama-3.1-8B-Instruct-abliterated-numallama/tokenizer.bin -m chat
# or an OpenAI-compatible server on http://127.0.0.1:8080/v1
./runq models/Llama-3.1-8B-Instruct-abliterated-numallama/llama-3.1-8b-instruct-abliterated-q4.bin -z models/Llama-3.1-8B-Instruct-abliterated-numallama/tokenizer.bin -m server -P 8080
See the numallama README for all options (context length, sampling, tools, speculative decoding, API details).
License
Llama 3.1 is licensed under the Llama 3.1 Community License, Copyright © Meta Platforms, Inc. All Rights Reserved. Use of this model must comply with the Llama 3.1 Community License and the Acceptable Use Policy.
Original model: meta-llama/Llama-3.1-8B-Instruct. Abliteration: mlabonne/Meta-Llama-3.1-8B-Instruct-abliterated. Conversion: numallama.