Budget memory for a 4-bit LLM
A 4-bit LLM needs memory for weights, cache and runtime tensors. Measure packing metadata, a temporary dequantized array, and a real Qwen KV cache, then budget for context and concurrency.
A 4-bit LLM needs room for its weights, KV cache, temporary tensors and inference runtime. Longer contexts and concurrent requests can consume the space left after loading the weights.
In a NumPy packing experiment, the matrix needs 0.5625 MiB, including its scales. Using it through a deliberately simple unpacking helper creates a 4 MiB floating-point matrix. A separate experiment with a real Qwen model shows its FP16 cache growing from 1.5 to 6 to 12 MiB as context and batch increase.
Packed int4 payload and scale storage
For N weights, N × 4 / 8 bytes is the ideal packed payload estimate, before metadata or byte alignment. Formats may also store scales, zero points, higher-precision tensors and alignment padding.
The downloadable experiment packs two signed, symmetrically quantized values into each byte. One FP32 scale is stored per group of 64 weights. This is an educational NumPy implementation for inspecting the stored arrays.
import numpy as np
from memory_parts import pack_int4, unpack_int4
weight = np.random.default_rng(0).standard_normal((1024, 1024)).astype('float32')
packed, scale = pack_int4(weight)
restored = unpack_int4(packed, scale, weight.shape)
for name, array in [('original', weight), ('packed', packed),
('scales', scale), ('restored', restored)]:
print(name, array.nbytes)
Output:
original 4194304
packed 524288
scales 65536
restored 4194304
The packed payload is 0.5 MiB; scales add 0.0625 MiB. The resulting 0.5625 MiB is about 7.1 times smaller than the original FP32 array. The scales account for the difference from an eightfold reduction. MiB here means 2**20 bytes.
The helper reconstructs an entire FP32 matrix before matrix multiplication. Its packed representation can therefore coexist with a much larger temporary. The execution path determines which arrays need memory at the same time.
FP16 cache growth with context and batch
Cached attention keys and values have their own storage dtype. Transformers exposes quantized caches as a separate cache strategy, with its own backend and compatibility choices (checked 2026-11-01).
For a conventional full-attention cache with a uniform layer layout:
KV bytes = 2 × layers × batch × cached tokens × KV heads × head dimension × bytes per element
Use the KV-head count in this formula. The pinned Qwen2.5-0.5B-Instruct configuration has 24 layers, two KV heads and head dimension 64; its 14 query heads share the KV heads. At FP16, that is 2 × 24 × 2 × 64 × 2 = 12,288 bytes per cached token per sequence.
I loaded the actual base transformer from that checkpoint and counted every cache key/value tensor after forward passes, using repeated token ID 42 to control tensor shape. The run used CPU, FP16 weights and eager attention. The model is Apache-2.0 licensed.
| Batch | Cached tokens per sequence | Measured KV tensor payload | Formula agrees? |
|---|---|---|---|
| 1 | 128 | 1,572,864 bytes / 1.5 MiB | Yes |
| 1 | 512 | 6,291,456 bytes / 6 MiB | Yes |
| 2 | 512 | 12,582,912 bytes / 12 MiB | Yes |
At batch 8 and 32,768 cached tokens per sequence, the formula projects 3 GiB of FP16 KV payload alone for this configuration. Paged allocation, cache sharing, sliding windows, offloading and cache quantization can change the accounting.
Run the bounded checks
Download memory_parts.py into an empty working folder. With Python 3.12, install the tested dependencies:
python -m venv .venv
source .venv/bin/activate
python -m pip install torch==2.15.0 numpy==2.5.4 transformers==5.17.0
python memory_parts.py --output memory.json
The first run downloads the pinned public checkpoint; no API key or paid inference service is needed. With the checkpoint already cached, --local-files-only prevents a download. The script runs only the three bounded shapes above, checks FP16 cache dtypes and asserts exact agreement with the byte formula. The standalone script and the code block above were both executed on 2026-11-01 with Python 3.12.13, NumPy 2.5.4, PyTorch 2.15.0 and Transformers 5.17.0 on an Apple M5 Mac with 16 GiB unified memory. Saved evidence: memory.json.
The measurements above count array and cache-tensor payloads in two separate experiments. They exclude allocator overhead and other intermediates, and the combined peak RSS of a fully quantized LLM was not measured. The NumPy helper is an educational pack/unpack implementation; GPTQ, AWQ, GGUF and fused production kernels can use different paths, including avoiding the full 4 MiB temporary. The Qwen run checks tensor shape and cache bytes using repeated token inputs, which provide no text-quality evaluation. MPS and CUDA remain untested. The 3 GiB cache is a formula projection: it was never allocated, and an out-of-memory crash was not induced. That projection also excludes weights, activations, attention intermediates, padding and allocator/runtime costs.
Diagnose the request that fails
If loading succeeds but a long prompt fails, try fewer total prompt-plus-generated tokens while keeping the batch fixed. If single requests fit but concurrent requests fail, reduce simultaneous sequences. These are different pressure points from the weight file itself. A failure during prefill may involve attention intermediates; a failure as generation progresses may involve the growing cache. Measure at both stages using the chosen runtime’s tools to locate the pressure point.
On Apple Silicon, CPU and GPU use one physical memory budget. Apple describes CPU/GPU shared resources in its unified-memory architecture (checked 2026-11-01). This machine’s 16 GiB is shared with the OS and other applications, so the model’s budget must leave room for them.
The earlier KV cache experiment measures the speed it saves. To fit a request, budget the weights, cache and runtime separately, and leave headroom for temporary allocations as context and concurrency increase.
