run it locally
Run Gemma 2 9B on RTX 4090
9.2B parameters on a 24 GB card. It fits at FP16 (full) — here's the VRAM math at every quantization, the speed to expect, and the quality you trade.
| quantization | vram needed | fits 24gb? | tokens/sec | quality |
|---|---|---|---|---|
| FP16 (full)recommended | 22.1 GB | ✓ yes | ~47 | Reference quality — no quantization loss. |
| Q8_0 | 11.7 GB | ✓ yes | ~88 | Near-lossless; rarely worth the extra space over Q6. |
| Q6_K | 9.1 GB | ✓ yes | ~114 | Virtually indistinguishable from full precision. |
| Q5_K_M | 7.6 GB | ✓ yes | ~135 | Minor loss; an excellent quality-vs-size balance. |
| Q4_K_M | 6.2 GB | ✓ yes | ~166 | Small but measurable loss; the popular default. |
| Q3_K_M | 4.7 GB | ✓ yes | ~217 | Noticeable degradation; only when you're tight on VRAM. |
Weights = params × bytes/weight, +20% for KV cache & runtime; usable VRAM is 95% of nameplate. Tokens/sec is a bandwidth ceiling (1008 GB/s) — real throughput is lower with long context. Try other combinations →
Running Gemma 2 9B on a RTX 4090
At FP16 (full), Gemma 2 9B's weights take about 22 GB, inside the RTX 4090's 23 GB of usable memory. Reference quality — no quantization loss. Generation speed is bound by memory bandwidth — the GPU reads the whole model once per token — so expect on the order of 47 tokens/sec before context overhead.
Quantization is the lever
Each step down in precision shrinks the model: FP16 needs about 22 GB, Q4_K_M about 6 GB — a 72% reduction for a small, usually acceptable quality cost. Q5_K_M and Q6_K are near-lossless if you have the headroom; drop to Q3 only when you're genuinely out of VRAM. The quantization guide covers the tradeoffs in detail.
Frequently asked questions
Can a RTX 4090 run Gemma 2 9B?
Yes — Gemma 2 9B fits on a RTX 4090 at FP16 (full) (22 GB of the 23 GB usable), with roughly 47 tokens/sec. Higher-precision quants need more VRAM; the table above shows each option.
How much VRAM does Gemma 2 9B need?
Gemma 2 9B is a 9.2B-parameter model. At FP16 that's about 22 GB; at Q4_K_M (the popular default) about 6 GB, including ~20% for the KV cache and runtime. Quantization is the main lever — see the per-quant table above.
Other combinations
Gemma 2 9B on other GPUs: RTX 3060 12GB, RTX 4070 Ti, RTX 4080, RTX 3090, NVIDIA L4
Other models on the RTX 4090: Phi-3 Medium 14B, Gemma 2 27B, Qwen2.5 32B, Command R 35B, Mixtral 8x7B, Llama 3.3 70B
Related
- VRAM calculator — any model, quant and GPU.
- Running models locally — the hardware reality.
- All run-locally combinations →