library_name: onnxruntime
license: other
license_name: lfm1.0
license_link: LICENSE
pipeline_tag: image-text-to-text
base_model: balllvin/LFM3-Omni-2B-abliterated
base_model_relation: quantized
tags:
- lfm3
- onnx
- webgpu
- onnxruntime-web
LFM3-Omni-2B-abliterated-ONNX-WebGPU
balllvin/LFM3-Omni-2B-abliterated as ONNX for the browser (WebGPU,
onnxruntime-web). Chat with it in the browser: LFM3-WebGPU.
It answers without a thinking block, as the source release does. Text and images in, text out.
Parity with the bf16 source
| Check | Result |
|---|---|
| Same top token as bf16, all positions | 93.3% |
| Same top token, generated part | 95.8% |
| Mean KL from bf16 (nats per token) | 0.0468 |
| Greedy tokens equal to bf16 before the first difference | 28 of 223 |
| Reference for the same check on LiquidAI's own q4f16 vs its bf16 | 93.1% top token, KL 0.0503 |
Six fixed chat prompts, 48 greedy tokens each from transformers bf16, teacher forced through the ONNX graph with
the caches carried across prefill chunks of 32 tokens (ONNX Runtime CPU).
Speed
256 new tokens, the same prompt, median of 3 runs.
| Where | Decode tok/s | Prefill tok/s |
|---|---|---|
| Apple M5 Pro, headless Chrome | 11.6 | 168 |
How it was built
- Source weights: balllvin/LFM3-Omni-2B-abliterated model.safetensors (bf16, the public release itself), sha256 33c03abfbe93dd1f2fdc065be4a132554b0d16e8d581bd47df4a9359e5efc591, the
model.safetensorssha256 in its graft.json. - Graph: LiquidAI's ONNX Runtime GenAI graph, LiquidAI/LFM3-Omni-2B-ONNX@d2e656f0bddfaa2b8ebd4bfbe9d88de7897ee592 onnx/decoder_q4f16.onnx, unchanged. Only the weights differ.
- Quantization: the same recipe as that graph: ONNX Runtime k-quant, asymmetric, 32 weights per scale, weights
cast to fp16 first; the embedding, the LM head and the same projections as LiquidAI's file stay in int8, the rest
are int4. Run on LiquidAI's own bf16 source, this code gives back 379 of 594
initializers byte for byte; the other 215 differ in a few bytes, 0.0032% of all bytes (float rounding in the scale search). - Vision encoder: LiquidAI/LFM3-Omni-2B-ONNX@d2e656f0bddfaa2b8ebd4bfbe9d88de7897ee592 onnx/vision_encoder_q8.onnx, byte for byte. The Omni grafts change only the language model, so vision weights are the host model's.
- Parity is measured on the language core alone (
brainspa graft extract), which is what the decoder runs for text. - Speed was measured in headless Chrome while the Mac was busy with other jobs (load average 10 to 12), so it is a low bound.
Files
| File | Size |
|---|---|
onnx/decoder_q4f16.onnx |
0.3 MB |
onnx/decoder_q4f16.onnx_data |
1.41 GB |
onnx/vision_encoder_q8.onnx |
0.1 MB |
onnx/vision_encoder_q8.onnx_data |
0.20 GB |
Total download 1.61 GB. Use import * as ort from 'onnxruntime-web/webgpu' and prefill in chunks of at most
1024 tokens. Inputs and outputs are those of the LiquidAI graph.
Licence
LFM Open License v1.0 (see LICENSE) and NOTICE.md, as the source release.