The Phase 1 site compressed the embedding for storage but admitted the serving endpoint reconstructs it to full bf16 at startup — GPU memory and compute were unchanged. This phase goes after the real thing: two independent techniques, each measured (not assumed) for actual runtime memory and latency impact.
Phase 1's Residual PQ shrinks the embedding on disk from 1,180 MB to 265 MB — real, but the deployed endpoint still loads the full reconstructed matrix into GPU memory before serving a single request. This phase tests two ways to actually reduce what's resident on the GPU and how much compute each forward pass does: NF4 4-bit quantization of the 78% of the model Phase 1 never touched (attention + MLP), and LUT-based serving that never reconstructs the full embedding at all. Both are fresh downloads of the same fine-tuned model — nothing about the four existing demos or their models was modified.
Every attention/MLP projection (q/k/v/o/gate/up/down_proj — the parameters Phase 1's embedding-only compression never touched) is loaded as bitsandbytes NF4 4-bit instead of bf16. This is the same quantization already used during QLoRA training; here it's carried into serving. The embedding/tied head stay bf16 (NF4 only wraps nn.Linear, and the embedding is an nn.Embedding) — so this is a clean, independent test of the other 78% of the model.
Both footprint numbers are model.get_memory_footprint() measured on the same L4 GPU in the same process, back to back — not estimated from bit-widths. The ~1.4% relative val-loss increase is a real, small quality cost from 4-bit rounding on weights that (unlike the embedding) directly participate in every attention and MLP computation.
Reuses the exact RQ codebook already trained and validated in Phase 1 (M=384, K=1024, L=2 — not retrained here) but never reconstructs the full [256,000 × 2,304] embedding. Instead: input token rows are reconstructed on demand (a handful of rows per prompt, not 256,000), and output logits are computed via lookup+sum instead of a full matmul, dropping into Gemma-2's own softcap logic unchanged.
| check | result | what it isolates |
|---|---|---|
| LUT vs. direct decode+matmul (both fp32) | 4.8×10⁻⁷ rel diff — PASSED | pure math-equivalence check, no bf16 anywhere — isolates whether the LUT lookup+sum algorithm itself is correct |
| LUT(fp32) vs. deployed bf16 standard path | 0.71% rel diff (context) | not gated — expected to show ~bf16 rounding-level noise (max 0.39%) since the deployed Phase 1 model's own reconstruction was already rounded to bf16 before saving |
The honest result: LUT serving is 1.78× slower than the standard matmul path on this L4 (85.90 ms/token vs. 48.21 ms/token), and its peak memory during generation (6,643 MB) is actually slightly higher than standard (6,448 MB) — even though its idle footprint before generating anything is 12% lower (5,640 MB vs. 6,408 MB). The naive lookup+sum implementation builds a large transient index tensor for torch.gather on every forward call, and that overhead outweighs the reduced multiply count on well-provisioned GPU hardware — exactly the risk flagged before running this test, not a surprise discovered after the fact. The idle-memory win is real; the compute win is not, on this hardware, with this implementation.
All five variants answered the identical 200 held-out questions; a blinded LLM judge graded every answer against the reference, never told which model produced it — the same protocol used throughout this whole project.
| metric | n | Gemma-NF4 | Gemma-LUT | Gemma-RQ | Gemma-2-2B | SLM-125M |
|---|---|---|---|---|---|---|
| grounded accuracy | 155 | 91.6% | 92.3% | 92.9% | 92.9% | 51.6% |
| open-domain accuracy | 45 | 86.7% | 84.4% | 86.7% | 80.0% | 15.6% |
| grounding / faithfulness rate | 155 | 94.2% | 93.5% | 93.5% | 93.5% | 57.4% |
This project has since tried 2 more variants beyond NF4/LUT/RQ above: RQ+NF4 stacked together, and independent RQ compression of the MLP weights. Storage/memory/speed were measured in one consistent benchmark (same GPU, same prompts, same warmup/sync discipline) so these numbers are directly comparable to each other.
| variant | grounded acc. | open acc. | grounding rate |
|---|---|---|---|
| Uncompressed (baseline) | 92.9% | 80.0% | 93.5% |
| RQ (embedding) | 92.9% | 86.7% | 93.5% |
| NF4 / LUT (this page) | 91.6% / 92.3% | 86.7% / 84.4% | 94.2% / 93.5% |
| Combined RQ+NF4 | 91.0% | 80.0% | 94.2% |
| MLP RQ v2 | 92.3% | 84.4% | 93.5% |
| variant | storage | peak GPU memory (generation) | speed |
|---|---|---|---|
| Uncompressed | 5.28 GB | 5,278.8 MB (1.0x) | 46.65 ms/tok (1.0x) |
| RQ | 412 MB embedding (vs. 1,180 MB bf16) | 5,278.8 MB (1.0x — no runtime savings) | 46.57 ms/tok (1.0x) |
| NF4 (this page) | ~2.2 GB whole model | 3,533.2 MB (0.67x) | 80.32 ms/tok (1.72x) |
| LUT (this page) | same 412 MB artifact | 7,830.8 MB (1.48x — worse than baseline) | 85.22 ms/tok (1.83x — slowest) |
| Combined RQ+NF4 | ~2.2 GB | 2,353.6 MB (0.45x — best of all six) | 80.12 ms/tok (1.72x) |
| MLP RQ v2 | ~1.66B params @ 9.44 bpp (~41% smaller on 63.6% of the model) | 5,279 MB (1.0x — no runtime savings) | 65.18 ms/tok (1.40x) |
See RQ (embedding), Combined RQ+NF4, and MLP RQ v2 for the other three.
80% of random pulls come from the 45 held-out questions never used in training (a true generalization test); 20% come from the ~1,964 open-domain training questions. Each pull is labeled below.
NF4's memory number is real and directly measured (model.get_memory_footprint(), bf16 vs. NF4, same process, same GPU). The ~1.4% val-loss cost is real too — unlike Phase 1's embedding, these weights sit inside every attention and MLP computation, so quantizing them isn't free the way the embedding was.
LUT's memory story has two sides, both disclosed above. The resident footprint really is lower (codes as int16 + small fp32 codebooks, vs. the full reconstructed embedding) — that part is a genuine win. But the naive lookup+sum implementation used here (vendored from the original research project's lut_matmul.py, itself only ever benchmarked on a different, untied model without Gemma's softcap) builds a large transient index tensor for the torch.gather call on every forward pass, so peak memory during generation and wall-clock latency are reported exactly as measured — including if they come out worse than the standard matmul path. This was flagged as an open question going in, not assumed to be a win.
The LUT correctness gate specifically compares the LUT algorithm against a direct decode+matmul in fp32, with no bf16 involved — an earlier version of this test compared against the bf16-rounded Phase 1 file directly and failed at ~0.7% relative diff, which turned out to be bf16's own ~0.39% max rounding error showing up as "difference," not a LUT bug. That mistake and the fix are left visible here rather than smoothed over.
Everything else — the QLoRA fine-tune, the grounded/open-domain behavior — is identical to the uncompressed demo and the Phase 1 (storage compression) demo.