KV-Cache Optimization on Llama-2-7B

Six methods benchmarked for quality, throughput, and memory on Llama-2-7B-chat (NVIDIA H100 80GB, Modal). Quality on LongBench (6 tasks × 20 examples = 120). Throughput/memory from a manual greedy-decode loop, prefill=1024, generate=512.

Quality: LongBench

overall score, higher is better · baseline FP16 = 0.291

Overall score by method

KIVI 4-bit and TopK K=1024 are lossless (tie baseline). 2-bit / K=512 lose ~6%.

Per-task change vs baseline

Δ score per task. Blue = better than FP16, red = worse. Methods have complementary blind spots.

Throughput scaling: KIVI 2-bit vs baseline

batch-size sweep, decode phase

Decode throughput vs batch size

KIVI overtakes baseline at BS=16, hits 1.93× at BS=32, and runs to BS=128 where baseline OOMs.

Peak GPU memory vs batch size

Smaller KV cache lets KIVI serve 4× the batch size (128 vs 32) on one 80GB GPU.

Memory footprint

KV-cache compression

KV-cache reduction by method

Peak KV-cache shrink vs FP16 baseline (each measured in its own config).

MLA latent: KV cache vs prefill length

MLA stores a compressed latent: consistent ~3.9× smaller KV across context lengths (B=1).

Long context: TopK-Flash @ 32K prefill

flash-attention + sparse selection

Time to first token & peak memory

At 32K prefill, TopK-Flash cuts TTFT 33% and peak memory 19% vs full attention.