Estimate the GPU VRAM required to run local LLMs like Llama 3, Mistral, Mixtral, Gemma 2, and Qwen2 at FP16, INT8, or INT4 quantization, including KV cache overhead, and get a recommended GPU tier for your setup.
Enter your details and click Calculate to see results
Running a large language model locally requires enough GPU VRAM to hold both the model's weights and the working memory it needs during inference — get this wrong and the model either won't load at all or will run painfully slowly by offloading to system RAM. This VRAM calculator estimates total VRAM requirements for popular open models — Llama 3, Llama 3.1, Mistral, Mixtral, Gemma 2, Qwen2, and Phi-3 — at different quantization levels, and maps the result to a recommended consumer or data-center GPU tier.
It computes base VRAM as parameter count (in billions) × bytes per parameter for the chosen quantization × a 1.2 overhead factor that accounts for CUDA kernels, activation buffers, and memory fragmentation beyond the raw weights. It then estimates KV cache size — the memory used to store attention keys and values for every token in context — and adds base VRAM plus KV cache for a total, mapping the result to a recommended GPU tier from RTX 4060 up to multi-GPU/cloud clusters.
Hobbyists deciding which GPU to buy for local LLM experimentation, developers sizing a self-hosted inference server, researchers planning fine-tuning hardware, and startups evaluating self-hosting against API costs all need this VRAM math before spending money on hardware or a cloud GPU rental.
Quantization is the single biggest lever for fitting a large model onto affordable hardware — dropping from FP16 to INT4 cuts memory needs by roughly 4x, often the difference between needing a multi-GPU cluster and running comfortably on a single consumer card, at some cost to output quality depending on the quantization method. KV cache is easy to overlook but becomes a major factor for long-context or high-concurrency serving, since it scales with both context length and batch size — a model that fits comfortably for single-user chat may run out of memory serving many concurrent long-context requests.
How this VRAM calculator turns parameter count and quantization into a memory estimate
Bytes per parameter depends on quantization: FP16/BF16 = 2 bytes, INT8 = 1 byte, INT4 = 0.5 bytes. The total VRAM figure is then mapped to a recommended GPU tier, from an 8GB RTX 4060 up to "Multi-GPU / cloud cluster required" beyond 80GB.
FP32 uses 4 bytes/parameter, FP16/BF16 uses 2 bytes, INT8 uses 1 byte, and INT4 uses 0.5 bytes. Lower precision means less memory but potentially reduced output quality.
KV cache memory grows with both context length and batch size. Long-context or high-concurrency serving can require far more VRAM for KV cache than for the model weights themselves.
Consumer cards (RTX 4060–4090) suit smaller quantized models; data-center cards (A100, H100 80GB) suit large models at higher precision.
From picking a model to reading your recommended GPU tier
Choose from 9 preset open models — Llama 3 8B/70B, Llama 3.1 405B, Mistral 7B, Mixtral 8x7B, Gemma 2 9B/27B, Qwen2 72B, or Phi-3 3.8B — or pick Custom to enter your own parameter count.
If you selected Custom, type the model's parameter count in billions (for example 13 for a 13B model); this field only appears when Custom is selected.
Pick FP16/BF16 (2 bytes/parameter), INT8 (1 byte/parameter), or INT4 (0.5 bytes/parameter) to match how you plan to deploy the model.
Enter the maximum context window in tokens you expect to use, either by typing a value or dragging the slider, since KV cache scales with context length.
Enter how many sequences you expect to process concurrently — a higher batch size multiplies KV cache memory needs for concurrent serving.
See total VRAM required, the base model VRAM and KV cache split, parameter and bytes-per-parameter figures, a recommended GPU tier, and a quantization comparison chart.
Using the calculator's own default scenario — Llama 3 70B at INT4, 8,192-token context, batch size 1
Suppose you want to run Llama 3 70B (70 billion parameters) locally at INT4 quantization (0.5 bytes/parameter), with an 8,192-token context window and batch size of 1.
Explanation: Quantizing Llama 3 70B from FP16 to INT4 cuts its VRAM footprint by 4x — from 168GB (needing multiple data-center GPUs) down to 42GB (fitting on a single 48GB workstation card). KV cache adds only about 0.1GB at this modest 8,192-token context and batch size of 1, but that figure would grow substantially at longer contexts or higher concurrency.
What your total VRAM figure means for hardware selection
| Total VRAM | Recommended Tier | What It Generally Means |
|---|---|---|
| Up to 8 GB | RTX 4060 (8GB) | Small models or aggressively quantized mid-size models fit comfortably |
| 8 – 16 GB | RTX 4070 Ti / 4080 | Popular 7B-13B class models at INT8/INT4 fit with headroom |
| 16 – 24 GB | RTX 4090 / A5000 | Larger quantized models or moderate concurrent serving |
| 24 – 48 GB | A6000 | 70B-class models at INT4, or smaller models at higher precision |
| Over 48 GB | A100/H100 or multi-GPU | Large models at FP16/INT8, or high-concurrency serving |
If your total sits right at a tier boundary: leave headroom — real inference frameworks add their own overhead beyond this calculator's 1.2× multiplier, so a number that just fits on paper can still run out of memory in practice.
If KV cache dominates your total: that signals a long-context or high-concurrency use case — consider whether reducing batch size or context length is workable, or whether you genuinely need the extra headroom.
These are rule-of-thumb estimates typically within 10-20% of real-world usage. Always validate against your specific inference framework before finalizing a hardware purchase.
This calculator provides planning estimates only. Actual VRAM usage depends on your inference framework, CUDA version, and model-specific architectural details like grouped-query attention.
Where sizing GPU memory before deployment genuinely helps
Size a purchase decision around the exact VRAM a target model needs, not a guess.
Check whether a home workstation GPU can handle a specific open-weight model.
Estimate a baseline VRAM floor before adding fine-tuning-specific memory overhead.
Compare FP16 vs INT8 vs INT4 memory footprints before choosing a deployment precision.
See the concrete GB difference between running a model at FP16 versus INT4.
Increase batch size in the calculator to see how KV cache scales for multi-user serving.
Check how a large context window like 128K tokens affects KV cache VRAM needs.
Determine the VRAM tier needed, then check rental cost with the GPU Cost Calculator.
Use it in a course to make quantization and KV cache scaling concrete for students.
Correctly size Mixture-of-Experts models like Mixtral using total, not active, parameters.
Plan a self-hosted AI server build around a specific model roster's combined VRAM needs.
Quickly re-estimate VRAM needs whenever you consider switching to a newer or larger model.
What this VRAM calculator does well, and where it can't replace real-world testing
Approximate VRAM footprint per billion parameters by quantization level
| Quantization | Bytes/Param | VRAM per 1B Params (incl. 1.2× overhead) | Typical Quality Impact |
|---|---|---|---|
| FP32 | 4 bytes | 4.8 GB | None (reference precision) |
| FP16 / BF16 | 2 bytes | 2.4 GB | Negligible |
| INT8 | 1 byte | 1.2 GB | Small, usually acceptable |
| INT4 | 0.5 bytes | 0.6 GB | Noticeable on some tasks/models |
Summary: This VRAM calculator gives you an instant, free estimate of GPU memory requirements for running local LLMs across 9 popular open models and custom parameter counts, factoring in quantization and KV cache. Pair it with the GPU Cost Calculator and Inference Latency Estimator for a complete local-hosting decision.
Common questions about VRAM calculator estimates and local LLM hosting
Official documentation to complement this calculator
Explore other AI & infrastructure tools