library_name: onnxruntime
license: other
license_name: lfm1.0
license_link: LICENSE
pipeline_tag: text-generation
base_model: balllvin/LFM3-1.7B-Abliterated-NVFP4
base_model_relation: quantized
tags:
- lfm3
- onnx
- webgpu
- onnxruntime-web
LFM3-1.7B-Abliterated-ONNX-WebGPU
balllvin/LFM3-1.7B-Abliterated-NVFP4 as ONNX for the browser (WebGPU,
onnxruntime-web). Chat with it in the browser: LFM3-WebGPU.
It thinks before it answers, as the source release does. Text in, text out.
Parity with the bf16 source
| Check | Result |
|---|---|
| Same top token as bf16, all positions | 86.7% |
| Same top token, generated part | 93.4% |
| Mean KL from bf16 (nats per token) | 0.0764 |
| Greedy tokens equal to bf16 before the first difference | 102 of 288 |
| Reference for the same check on LiquidAI's own q4f16 vs its bf16 | 89.5% top token, KL 0.0702 |
Six fixed chat prompts, 48 greedy tokens each from transformers bf16, teacher forced through the ONNX graph with
the caches carried across prefill chunks of 32 tokens (ONNX Runtime CPU).
Speed
256 new tokens, the same prompt, median of 3 runs.
| Where | Decode tok/s | Prefill tok/s |
|---|---|---|
| Apple M5 Pro, headless Chrome | 11.8 | 169 |
How it was built
Source weights: orcarouter/LFM3-1.7B-abliterated model.safetensors (bf16), sha256 a11f80c25e8d70ead520fa8e686442d3807cbee12a6286182f40174d5ef34bf5, the
src_sha256in the NVFP4 release's nvfp4_build.json.Graph: LiquidAI's ONNX Runtime GenAI graph, LiquidAI/LFM3-1.7B-ONNX@57c125dfa29f2d39157e280b1d1f7480b8736d8d onnx/model_q4f16.onnx, unchanged. Only the weights differ.
Quantization: the same recipe as that graph: ONNX Runtime k-quant, asymmetric, 32 weights per scale, weights
cast to fp16 first; the embedding, the LM head and the same projections as LiquidAI's file stay in int8, the rest
are int4. Run on LiquidAI's own bf16 source, this code gives back 329 of 594
initializers byte for byte; the other 265 differ in a few bytes, 0.0043% of all bytes (float rounding in the scale search).Speed was measured in headless Chrome while the Mac was busy with other jobs (load average 10 to 12), so it is a low bound.
Files
| File | Size |
|---|---|
onnx/model_q4f16.onnx |
0.3 MB |
onnx/model_q4f16.onnx_data |
1.38 GB |
Total download 1.38 GB. Use import * as ort from 'onnxruntime-web/webgpu' and prefill in chunks of at most
256 tokens. Inputs and outputs are those of the LiquidAI graph.
Licence
LFM Open License v1.0 (see LICENSE) and NOTICE.md, as the source release.