library_name: openvino
license: llama3.1
base_model:
- Orenguteng/Llama-3.1-8B-Lexi-Uncensored-V2
pipeline_tag: text-generation
tags: - openvino
- int4
- llama
- uncensored
- abliterated
- intel
- arc
- text-generation
Llama-3.1-8B-Lexi-Uncensored-V2 — OpenVINO IR (int4)
OpenVINO IR conversion of
Orenguteng/Llama-3.1-8B-Lexi-Uncensored-V2,
an uncensored Llama-3.1-8B fine-tune.
Details
| Property | Value |
|---|---|
| Architecture | LlamaForCausalLM, 32 layers, hidden 4096, vocab 128256 |
| Format | int4 asymmetric group-128 |
| Repository size | 4.4 GB |
| Stateful | yes (beam_idx exposed) |
Benchmark
Single-stream, Intel Core Ultra 7 258V (Arc 130V/140V iGPU), 30 GB RAM,
OpenVINO Model Server 2026.4.0 on GPU, greedy decoding, 128 max new tokens,
mean of 3 runs after warmup.
| Build | tok/s | vs int4 |
|---|---|---|
| int4 (this build) | 23.5 | 1.00x |
| int8 | 11.3 | 0.48x |
| fp16 | 6.0 | 0.26x |
Decode on this iGPU is memory-bandwidth bound, so throughput tracks model size
almost exactly. int4 is both the smallest and the fastest.
Serving
This repository contains the IR only. Create an OVMS config alongside it:
cat > ovms_config.json <<'EOF'
{
"model_config_list": [
{
"config": {
"name": "lexi",
"base_path": "/absolute/path/to/this/model",
"target_device": "GPU",
"nireq": 8,
"plugin_config": { "PERFORMANCE_HINT": "THROUGHPUT", "NUM_STREAMS": "2" }
}
}
]
}
EOF
# graph.pbtxt must also exist in the model directory (copy from any OVMS LLM model).
PYTHONPATH=$OVMS_ROOT/lib/python ovms --config_path ./ovms_config.json --rest_port 11436
Conversion notes
- Source: BF16 safetensors (4 shards).
- Stage 1:
optimum-cli export openvino --task text-generation-with-past --weight-format fp16. - Stage 2:
nncf.compress_weightson the IR (never a single-pass int4 export — that
OOMs on a 30 GB machine for this size class). - Exported with transformers 5.5.0 / optimum-intel 2.2.0.
Limitations
- Uncensored fine-tune: outputs are not filtered. Evaluate before deployment.
- Throughput depends on runtime kernels, hardware, and prompt distribution.
Attribution
Base model: Orenguteng/Llama-3.1-8B-Lexi-Uncensored-V2,
built on Meta's Llama 3.1 8B under the Llama 3.1 Community License.
Conversion: OpenVINO IR via optimum-intel + NNCF.