BitDecoding-FP4 reduces the memory used by model attention

What the cache holds

The key-value cache retains intermediate information from earlier tokens. Compressing it can save memory, but rounding errors can change attention and reduce answer quality.

BitDecoding-FP4 reduces the memory used by model attention ↗

Read throughput with its conditions

The authors report near-BF16 accuracy and 2.4–3.0 times peak serving throughput on selected Qwen models on B300. The repository includes concurrency points from one to sixty-four requests; this is not a claim about consumer GPUs.

BitDecoding-FP4 reduces the memory used by model attention ↗BitDecoding-FP4 implementation ↗

What you can do

1. Check hardware, context length and concurrency against the published benchmark conditions. Open reference ↗

Where the evidence stops

Neither accuracy nor throughput reproduced. Model, hardware, context and concurrency choices affect results; third-party licences remain separate.

Sources read. Public source text read November 10, 2026 UTC. No third-party project, model inference or complete workflow executed.

Sources & further reading