KV Cache on 16 GB GPUs: Making Long Context Actually Fit

A model can advertise a 128K context window and still fail at 40K tokens on a 16 GB GPU. The architecture ceiling never promised that weights, KV cache, compute buffers, and the desktop compositor would fit on your card at the same time.

The KV cache is usually where long-context plans meet that physical limit. It grows with every active token and sequence, so a configuration that looks comfortable at startup can slow sharply, spill into system memory, or fail during a large prefill.

This guide turns the problem into a VRAM budget. It covers the cache formula, reproducible 32K to 128K size tables, and working configurations for llama.cpp --cache-type-k and --cache-type-v, vLLM paged and prefix caching, and Ollama context controls — plus the experimental adaptive-cache forks that deserve interest but not blind trust. For the broader throughput, latency, and benchmark context behind these numbers, start with the LLM Performance hub.

The Short Answer for a 16 GB GPU

Start with one sequence, a realistic maximum context, Flash Attention, and an 8-bit KV cache. Measure that configuration before attempting 4-bit cache, CPU offload, multiple parallel slots, or an experimental fork.

My opinion is simple: a stable 64K setup is usually more useful than a nominal 128K setup that runs at the edge of an out-of-memory failure. Context capacity is not a trophy; it is a latency, quality, and concurrency decision.

What the KV Cache Stores

During autoregressive generation, every attention layer produces key and value tensors for each processed token. The runtime retains those tensors so the next token can attend to earlier tokens without recomputing the entire prefix.

The cache saves enormous compute, but it consumes memory in proportion to the retained token count. For a conventional transformer with grouped-query attention, a useful baseline is:

KV bytes = sequences * tokens * layers * 2 * KV heads * head dimension * bytes per value

The factor of two represents keys and values. Multi-head attention uses as many KV heads as query heads, grouped-query attention uses fewer KV heads, and multi-head latent attention or hybrid recurrent architectures need different calculations.

Why Parameter Count Is Not Enough

Two 8B models can have very different KV-cache costs. One may use 32 layers and eight KV heads, while another may use fewer KV heads, shared KV layers, sliding-window attention, or compressed latent states.

Parameter count mostly predicts weight memory. KV geometry comes from attention architecture, so read the model metadata rather than guessing from 8B, 27B, or the GGUF file size. The clearest illustration is how far attention design has moved beyond plain multi-head attention (MHA):

• Multi-Query Attention (MQA) shares a single K/V head across all query heads — maximal cache savings, but it is the most aggressive quality compromise and is rarely used alone in current frontier models.
• Grouped-Query Attention (GQA) groups query heads into clusters that each share one K/V head — the mainstream compromise used by most open dense models, and the geometry the formula above assumes.
• Multi-Head Latent Attention (MLA), introduced in DeepSeek-V2 and carried into DeepSeek-V3 and Kimi K2, takes a different approach entirely: instead of sharing K/V across heads, it projects keys and values into a compressed low-rank latent vector and reconstructs the full-resolution K/V on demand at attention time. DeepSeek reported roughly a 93% KV-cache reduction versus an equivalently sized dense MHA model, while keeping quality competitive with — sometimes ahead of — GQA at the same memory budget.

The practical consequence is that a “27B GQA model” and a “27B MLA model” can have KV-cache footprints that differ by an order of magnitude for the same context length. Do not assume the formula above applies to a model that documents itself as using latent attention, DeltaNet-style state, or sliding-window layers — check the architecture section of the model card first.

With Ollama, ollama show MODEL --verbose exposes model metadata including layer count, attention heads, KV heads, and context length where the format provides them. With llama.cpp, the model-loader output printed at startup usually includes equivalent GGUF metadata and the runtime’s actual cache allocation.

Model Limit, Allocated Context, and Used Context

These are three separate numbers. The model limit is the maximum supported by its training and positional encoding, the allocated context is what the runtime reserves or permits, and used context is the tokens currently retained for a sequence.

Raising an engine flag cannot safely extend a model beyond its supported position scheme. RoPE scaling can extend some architectures, but it is a model-quality experiment, not a KV-memory optimization.

KV Cache Size Table: 32K, 64K, and 128K Context Budgets

Consider a representative GQA model with 32 layers, eight KV heads, and a head dimension of 128. These dimensions produce 65,536 key and value elements per token before multiplying by the storage size of each element.

The table uses binary GiB and the physical block sizes commonly associated with llama.cpp f16, q8_0, and q4_0. It is a baseline calculation, not a promise about total process memory; alignment, metadata, hybrid layers, and backend workspaces add overhead.

Now double the layer count to 64 while leaving the other dimensions unchanged. The FP16 cache becomes 8 GiB at 32K, 16 GiB at 64K, and 32 GiB at 128K, which demonstrates why a single context recommendation cannot cover every model.

The Actual 16 GB Equation

A practical budget is broader than the KV formula:

usable VRAM = total VRAM - desktop and driver reserve

KV budget = usable VRAM
          - GPU-resident model weights
          - graph and activation buffers
          - runtime workspace
          - speculative-decoding state
          - safety margin

On a display-attached 16 GB card, do not plan around all 16 GiB being available. Reserve at least several hundred MiB for the desktop and driver, then leave another margin for workload-dependent buffers; 1.0 to 1.5 GiB of total breathing room is a reasonable starting assumption, but your logs are the authority.

Suppose a GGUF model occupies 10.8 GiB on the GPU and runtime overhead peaks near 1.2 GiB. After a 1 GiB safety margin, only about 3 GiB remains for KV, so the representative model fits roughly 45K tokens with Q8_0 or 87K with Q4_0, before engine-specific overhead.

That does not automatically make Q4_0 the right choice. If long-context accuracy drops on your workload, a smaller or more aggressively quantized model with a Q8_0 cache may be better than larger weights paired with a fragile cache. Measured anchors for exactly this arithmetic live in the 16 GB VRAM llama.cpp benchmark tables, where VRAM per model is recorded at 19K, 32K, and 64K context. For a wider survey of which model sizes and quant levels behave well under Ollama on the same class of card, see Comparing LLMs performance on Ollama on 16GB VRAM GPU.

Calculate the KV Cache Budget for Your Model

The following Python snippet estimates a conventional full-attention GQA cache. Replace the geometry with values from the model configuration or GGUF metadata.

def kv_gib(tokens, layers, kv_heads, head_dim, bytes_per_value, sequences=1):
    total = (
        sequences
        * tokens
        * layers
        * 2
        * kv_heads
        * head_dim
        * bytes_per_value
    )
    return total / (1024 ** 3)


model = {
    "layers": 32,
    "kv_heads": 8,
    "head_dim": 128,
}

types = {
    "f16": 2.0,
    "q8_0": 34 / 32,
    "q4_0": 18 / 32,
}

for tokens in (32768, 65536, 131072):
    row = {
        name: round(kv_gib(tokens=tokens, bytes_per_value=size, **model), 2)
        for name, size in types.items()
    }
    print(tokens, row)

The Q8_0 and Q4_0 ratios include simple block metadata, which is why they are slightly larger than exactly one byte and half a byte per value. The runtime’s startup report remains more accurate because it knows model-specific cache layouts.

When This Formula Is Wrong: Hybrid and Sliding-Window Architectures

Do not force hybrid architectures into the conventional GQA equation. Sliding-window layers retain only a recent window, shared KV layers reduce duplication, recurrent layers may carry fixed-size state, and multi-head latent attention stores a compressed representation rather than per-head K/V tensors — the MLA case above being the most dramatic example.

Modern engines increasingly manage these mixed layouts explicitly. Use the formula to explain the dominant terms, then confirm the allocation reported by the exact engine build and backend you plan to deploy.

llama.cpp: Direct Control of K and V Precision

llama.cpp exposes separate --cache-type-k and --cache-type-v options in its current argument parser. This is the most useful local-inference interface when you need to trade cache precision against context capacity instead of accepting one global preset. If you need the surrounding install and serving setup first, the llama.cpp guide covers llama-cli, llama-server, and the key VRAM flags.

A conservative 64K single-user configuration looks like this:

./llama-server \
  --model /models/model.gguf \
  --n-gpu-layers 999 \
  --ctx-size 65536 \
  --parallel 1 \
  --flash-attn on \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --batch-size 1024 \
  --ubatch-size 256

Flag syntax and backend support change quickly, so run llama-server --help for the installed build. More importantly, inspect the startup log: it should show the intended context, cache types, GPU offload, and the allocated K and V buffers.

Which llama.cpp Cache Types to Try

Start with Q8_0 for both K and V. It roughly halves KV memory relative to F16, and independent perplexity tests on 20B-plus models (Qwen3.6-27B, Nemotron-30B) show the aggregate quality delta from F16 is within measurement noise — a much less dramatic gamble than moving directly to Q4_0, which the same tests showed collapsing decode speed and accuracy at long context on smaller models.

If Q8_0 does not fit, test Q8_0 keys with Q4_0 values before quantizing both sides to Q4_0. This ordering has research backing, not just folklore: controlled bit-allocation studies on Llama, Phi-4, Qwen3, and Mistral checkpoints found that key tensors are consistently two to ten times more sensitive to quantization error than value tensors, and that giving keys the larger bit budget (for example 4-bit keys with 2-bit values) recovers up to 94–98% of full-precision accuracy — while the inverted split (2-bit keys, 4-bit values) can lose 30 percentage points on tasks like GSM8K. Keys determine which earlier tokens attention actually matches, so protecting them first is the architecturally sound choice, not just the safer-sounding one.

One caveat worth internalising: “low quality risk” on aggregate benchmarks does not mean zero risk at the token level. A controlled test that held Flash Attention constant and only changed KV precision under greedy (deterministic) decoding found that Q8_0 cache changed the exact generated text on the large majority of prompts, and Q4_0 changed it on essentially all of them — once one token flips, the rest of the continuation can diverge. Perplexity and downstream-task scores can look fine on average while individual outputs still differ from the F16 baseline. If your application needs byte-for-byte reproducibility (regression tests, cached responses, deterministic agents), treat any KV quantization as a behavioral change, not just a memory optimization, and validate against your own fixed prompt set.

Quantized V cache may require Flash Attention or a compatible backend path. A server that silently falls back to another type invalidates the experiment, which is why startup logs matter more than copied command lines.

Context, Parallel Slots, and Unified Cache

--ctx-size describes an engine capacity, not a guarantee that every parallel slot receives that many tokens independently. Cache management has evolved in llama.cpp, including unified-cache behavior, so test the exact build rather than relying on an older rule that simply divides context by slot count.

The capacity equation still survives implementation changes: simultaneous unique tokens need storage somewhere. If two agent sessions may each reach 48K, budget for close to 96K live tokens unless the workload shares prefixes or tolerates eviction and recomputation.

Batch Size Does Not Shrink Stored KV

--batch-size and --ubatch-size affect prompt processing and temporary memory. Lowering them can rescue a large prefill from an activation-memory spike, but it does not change the persistent bytes required for each retained token.

This distinction explains a common failure pattern: the model starts and an empty request works, but a 60K prompt fails during ingestion. Reduce the micro-batch to diagnose the transient peak; reduce context, cache precision, parallelism, or weight residency to change persistent capacity.

vLLM: Paged Capacity Is Still Capacity

vLLM approaches the problem as a serving engine. It profiles available memory, reserves a KV-cache pool, and allocates cache in blocks so concurrent sequences do not each require one large contiguous region. If you are deciding whether to move to vLLM in the first place, the Ollama to vLLM migration guide covers the workload signals; here the question is purely how much cache the pool can hold, and the vLLM quickstart covers install and general serving flags beyond the capacity levers below.

PagedAttention reduces fragmentation and waste around variable sequence lengths — paged allocation removes fragmentation, not per-token storage cost, so one unique 128K request still needs enough blocks for its KV state.

The official vLLM memory-conservation guide recommends limiting max_model_len and max_num_seqs when memory is tight, and notes that CUDA graphs consume additional GPU memory. On a 16 GB card, both settings should be intentional rather than inherited from a model’s maximum configuration.

A focused single-sequence server might start here:

vllm serve MODEL_ID \
  --max-model-len 65536 \
  --max-num-seqs 1 \
  --gpu-memory-utilization 0.90 \
  --kv-cache-dtype fp8 \
  --enable-prefix-caching

Not every 16 GB GPU, model, quantization method, or attention backend supports that exact combination. Treat it as a configuration shape: constrain length and concurrency, reserve headroom, select a supported cache dtype, and validate the initialization report.

FP8 KV Cache in vLLM

The current vLLM quantized KV-cache documentation supports FP8 cache formats on compatible CUDA and ROCm paths. FP8 approximately halves raw cache storage relative to BF16 or FP16 and can therefore increase token capacity or concurrency.

Scaling matters. The documentation distinguishes default scales, warm-up calculation, and dataset calibration, and recommends dataset-based calibration for the highest accuracy; simply setting FP8 with scale 1.0 is convenient but not automatically the most reliable quality choice.

Prefix Caching Is a Reuse Optimization

Automatic prefix caching lets a new request reuse KV blocks for an identical cached prefix. It is excellent for repeated queries over the same long document, shared system prompts, and multi-round conversations because it avoids recomputing the matching prefill.

It does not make a unique long request smaller, and it does not accelerate generation of new tokens. The vLLM prefix-caching documentation explicitly limits the benefit to shared-prefix prefill work.

GPU Memory Utilization Is Not Free Memory

Raising --gpu-memory-utilization gives vLLM a larger reservation target, but it does not create VRAM. Pushing it too close to 1.0 can leave insufficient room for the display, another process, changing activation peaks, or non-PyTorch allocations.

Begin around 0.88 to 0.92 on a dedicated 16 GB GPU, inspect the profile, and increase only if the workload remains stable. If initialization succeeds but real prompts fail, reduce batched tokens, sequence concurrency, CUDA graph capture, or the maximum context before assuming the allocator is broken.

Ollama: Easier Controls, Less Granular Diagnosis

Ollama deliberately provides a smaller operational surface. Its current context-length documentation defaults GPUs below 24 GiB to 4K context, recommends at least 64K for agent and coding workloads, and warns that larger context consumes more memory.

Set the server-wide default and confirm the loaded model like this:

OLLAMA_CONTEXT_LENGTH=65536 ollama serve

ollama ps

You can also set num_ctx per request or model. ollama ps is important because its PROCESSOR and CONTEXT columns reveal whether the model remained fully on the GPU and whether the requested context was actually allocated. Be aware that the scheduling behaviour behind those numbers changed between Ollama versions; my comparison of Ollama v0.12.1 memory allocation shows the new scheduler pushing some models further to CPU on a 16 GB card, so pin the version you measured.

Quantized KV Cache in Ollama

Ollama exposes OLLAMA_KV_CACHE_TYPE with f16, q8_0, and q4_0 choices in its current FAQ. Quantized KV requires Flash Attention, which Ollama uses automatically on supported backends or can be requested with OLLAMA_FLASH_ATTENTION=1.

A 16 GB long-context service can therefore be started as:

OLLAMA_CONTEXT_LENGTH=65536 \
OLLAMA_FLASH_ATTENTION=1 \
OLLAMA_KV_CACHE_TYPE=q8_0 \
OLLAMA_NUM_PARALLEL=1 \
ollama serve

Q8_0 is Ollama’s recommended alternative to F16. The FAQ warns that Q4_0 can produce a more noticeable quality loss, especially at higher context, so it should be a measured fallback rather than an automatic 16 GB preset.

Ollama Parallelism Multiplies the Context Budget

Ollama documents a particularly clear rule: required memory scales with OLLAMA_NUM_PARALLEL * OLLAMA_CONTEXT_LENGTH. Four parallel requests at a 32K setting can imply a 128K aggregate context allocation for that model.

For a personal agent on 16 GB, keep OLLAMA_NUM_PARALLEL=1 until one long session is stable. Queueing a second request is usually preferable to pushing the first model partly onto the CPU and making both requests slow. The queueing, 503, and model-unloading mechanics behind that choice are documented in how Ollama handles parallel requests.

CPU Offload: A Valid Escape Hatch With a Price

Moving some model layers or KV state into system RAM can turn an allocation failure into a working process. It also places PCIe bandwidth and host-memory latency in the decode path, where each generated token may pay the cost. The lane and generation evidence for when PCIe actually bites is in LLM Performance and PCIe Lanes.

Offload can be sensible for occasional batch work, but it is rarely the best default for an interactive coding agent. First compare a smaller weight quantization, Q8 KV, reduced concurrency, and a realistic context cap; use offload when capacity matters more than latency.

Watch for the cliff rather than the average. A server may decode quickly at 8K, then slow severely after part of the working set spills, so benchmark at 32K, 64K, and the intended maximum rather than reporting only an empty-context token rate.

Sliding-Window and Adaptive KV Caches

Sliding-window attention changes the budget by retaining only a recent window for selected layers. Hybrid models may combine those layers with occasional global attention or recurrent state, making a flat full-context calculation substantially overestimate or misplace memory.

The optimization is part of the model architecture, not a generic switch that can be applied without consequences. An engine must understand the layer pattern, eviction rules, positions, and any global tokens correctly.

What Adaptive KV Tries to Improve

Experimental forks go further by choosing cache precision or layout per layer and context depth. The goal is attractive: preserve higher precision where it matters, compress less sensitive layers, and change the mix before VRAM pressure causes a hard spill — the same key-sensitivity-over-value-sensitivity finding described above is exactly the kind of signal an adaptive allocator would want to exploit automatically instead of leaving it to manual --cache-type-k/--cache-type-v tuning.

One August 2026 downstream project, llama.cpp-adaptive-turboquant, reports an automatic selector for several layer-adaptive modes and publishes long-depth tests on an RTX 5080 16 GB. Those numbers are author-reported results from a specialized fork, not evidence that upstream llama.cpp behaves the same way.

Why It Is Still Experimental

The fork combines custom cache types, CUDA kernels, model-specific paths, and toolchain constraints. That is far more code to trust than switching upstream cache storage from F16 to Q8_0.

Use such a fork only when upstream cannot meet a real requirement and you can reproduce quality, stability, and speed on your model. Record the commit and CUDA version, because a result attached only to a project name is not reproducible.

A Fair Adaptive-Cache Test

Compare the fork against an upstream Q8_0 baseline with the same GGUF, prompt, sampler, context depth, and output length. Measure startup VRAM, peak prefill VRAM, prompt processing speed, decode speed, and a quality task that actually requires evidence from the oldest part of the context.

Do not accept a successful allocation as a complete result. A cache can fit 128K and still lose early facts, corrupt output late in the sequence, or decode too slowly to be useful.

A Worked 16 GB Tuning Procedure: One Variable at a Time

The fastest route to a stable configuration is to change one memory dimension at a time. Randomly altering cache type, batch size, layer offload, parallelism, and context together produces a working command with no explanation.

Step 1: Establish the Weight Floor

Load the model at 8K context, one sequence, and the intended GPU offload. Record process VRAM after warm-up and verify that no layers unexpectedly moved to the CPU.

If the weights and runtime already consume more than about 14.5 to 15 GiB, long context has no healthy margin. Choose a smaller weight quant or model before tuning the cache.

Step 2: Measure F16 or BF16 KV as the Quality Baseline

Run the smallest context that supports your test and retain the default high-precision cache. Save outputs from retrieval, code editing, tool selection, and long-instruction tasks.

This baseline tells you whether later errors come from cache quantization. Without it, a chat-template problem or weak model can easily be blamed on Q4 KV.

Step 3: Move to Q8 or FP8

Enable Flash Attention where required, select Q8_0 in llama.cpp or Ollama, or a supported FP8 mode in vLLM. Repeat the same prompts at the same token depths and confirm the log shows the intended cache type.

For many 16 GB deployments, this is the useful stopping point. It roughly doubles raw KV capacity without making cache compression the most aggressive quantization in the stack.

Step 4: Raise Context in Stages

Test 32K, 64K, 96K, and 128K instead of jumping directly to the advertised maximum. At each stage, record prompt-processing tokens per second, decode tokens per second, peak VRAM, and whether evidence near the start can still be recovered.

Long-context decode often slows even after memory fits because attention reads more cached state. Capacity and performance are separate axes.

Step 5: Tune Transient Memory

If failure occurs during prefill rather than initialization, reduce micro-batch or maximum batched tokens. If failure occurs only with simultaneous requests, reduce sequence concurrency or parallel slots.

Only after those controls are understood should you try mixed Q8/Q4 cache, full Q4 cache, CPU offload, or an adaptive fork. Keep the upstream Q8 run as the comparison baseline. If you later add speculative decoding or MTP, remember its draft buffers are another line in the budget equation, not free speed — the speculative decoding guide covers the mechanics and their VRAM cost, and my Qwen 3.6 27B and 35B MTP vs Standard benchmark shows exactly how much context an MTP head’s extra state can cost on a 16 GB card.

What to Record in a Long-Context Benchmark

A single tokens/s figure hides the exact problem this article is trying to solve. Long-context testing should preserve enough detail for another operator to reproduce the memory boundary.

Use nvidia-smi sampling or the equivalent vendor tooling during both prefill and decode. The engine’s allocation report is necessary, but peak device memory during a real prompt is the number that decides stability.

Common KV Cache Mistakes on 16 GB GPUs

Treating 128K Support as a Hardware Promise

The context field in a model configuration is an architectural ceiling. It says nothing about the memory left after loading a particular quantization on a particular engine.

Calculate the cache and verify the runtime. Marketing-sized context without a VRAM budget is merely an OOM delayed until the first serious prompt.

Quantizing Weights but Forgetting KV

A 4-bit GGUF reduces model weights, not an F16 KV cache. At long context, the cache can erase the entire saving and eventually exceed the weight footprint.

Report both quantizations. Q4_K_M model, Q8_0 KV is meaningful; 4-bit model is incomplete.

Assuming Paged Attention Compresses Tokens

Paging improves allocation and sharing behavior. It does not change the tensor precision or remove the KV state required by one unique sequence.

Use paged allocation to serve variable workloads efficiently. Use cache precision, model architecture, context caps, and concurrency limits to control capacity.

Assuming Prefix Caching Helps Every Long Prompt

Prefix caching saves repeated prefill computation when requests share an exact prefix. A one-off 100K repository dump receives no magical memory discount simply because prefix caching is enabled.

It is a workload optimization, not a substitute for the budget equation. Measure hit rate and retained cache pressure in multi-user serving.

Using Q4 KV Without a Quality Test

Low-bit cache can fail subtly. The model still writes fluent text, but attention over distant evidence, exact names, tool arguments, or code dependencies may degrade — and as the token-divergence research above shows, even the “safe” Q8_0 setting is not guaranteed to reproduce the exact F16 output under deterministic decoding, only to preserve accuracy on aggregate.

Test the target task at the target depth. Short chat benchmarks are nearly useless for validating a long-context cache.

Leaving Parallelism on Auto

An engine may choose concurrency that is sensible for throughput but impossible for your long-context target. On 16 GB, one deep sequence and several short sequences are fundamentally different workloads.

Set the limit explicitly, then raise it with measured traffic. Otherwise a second request can turn a stable 64K configuration into an allocation or latency surprise.

Recommended 16 GB Profiles

These profiles are starting positions, not universal presets. A model with unusual KV geometry — an MLA or hybrid sliding-window design in particular — can be much cheaper or more expensive than the conventional GQA example.

Interactive Coding Agent

Use one sequence, 48K to 64K context, Q8 cache, Flash Attention, and full GPU weight residency if possible. This profile favors predictable latency and good cache precision over an impressive but rarely useful maximum.

Enable prefix reuse when the engine supports it because coding turns often share a large repository or conversation prefix. Still compact tool output and old transcripts; cache engineering does not make irrelevant tokens valuable.

Long-Document Analysis

Use a smaller model with 64K to 128K capacity, Q8 or calibrated FP8 cache, and repeated-prefix caching when multiple questions target the same document. Measure time to first token because prefill may dominate even when decode remains acceptable.

If only one question will be asked, retrieval or chunked summarization may be faster and more reliable than forcing the entire corpus through a 16 GB card. Long context is a tool, not a replacement for information architecture.

Small Multi-User Server

Cap per-request context and total active sequences rather than advertising the model maximum to every client. vLLM’s paged allocation is useful here, while Ollama and llama.cpp also require explicit attention to aggregate live tokens.

Prefer queueing over uncontrolled spill. A slower admission policy is less damaging than every request suddenly crossing PCIe during decode.

Final Recommendation for 16 GB Long Context

For long context on 16 GB, Q8 KV and one active sequence are the right baseline. They expose the real limit without making low-bit cache quality, parallel allocation, and offload latency fail at once.

Calculate from attention geometry, subtract weights and runtime overhead, and then confirm the result in engine logs and peak-memory measurements. If 128K still does not fit, a smaller model is often the cleanest optimization; if it fits but crawls, reducing context is often the honest one.

Paged attention, prefix caching, sliding windows, and adaptive precision all solve useful but different problems. The winning setup is the one that remains on the GPU, retrieves old evidence correctly, and sustains acceptable decode speed at the context depth you actually use.

References

• vLLM: conserving GPU memory
• vLLM: quantized KV cache (FP8)
• vLLM: automatic prefix caching
• Ollama: context length documentation
• Ollama FAQ: OLLAMA_KV_CACHE_TYPE and Flash Attention
• llama.cpp-adaptive-turboquant — experimental layer-adaptive cache fork (author-reported results)
• Raschka, S. “Multi-Head Latent Attention (MLA).” LLM Architecture Gallery. https://sebastianraschka.com/llm-architecture-gallery/mla/
• “Decoding Multi-Head Latent Attention: The KV Cache Memory Bottleneck, Solved.” Vizuara. https://vizuara.substack.com/p/decoding-multi-head-latent-attention
• “Quantize What Counts: Bit Allocation Insights Informed by Spectral Gaps in Keys and Values.” arXiv:2502.15075. https://ar5iv.labs.arxiv.org/html/2502.15075
• v-code01. “kvdivergence — does KV cache quantization change the generated text under greedy decoding?” GitHub. https://github.com/v-code01/kvdivergence
• Comparing LLMs performance on Ollama on 16GB VRAM GPU
• Qwen 3.6 27B and 35B MTP vs Standard on 16GB GPU