run it locally
Run Llama 3.3 70B on Apple M2 Ultra 192GB
70B parameters on a 192 GB (unified memory) card. It fits at Q8_0 — here's the VRAM math at every quantization, the speed to expect, and the quality you trade.
| quantization | vram needed | fits 192gb? | tokens/sec | quality |
|---|---|---|---|---|
| FP16 (full) | 168.0 GB | ✗ no | — | Reference quality — no quantization loss. |
| Q8_0recommended | 89.0 GB | ✓ yes | ~9 | Near-lossless; rarely worth the extra space over Q6. |
| Q6_K | 68.9 GB | ✓ yes | ~12 | Virtually indistinguishable from full precision. |
| Q5_K_M | 58.0 GB | ✓ yes | ~14 | Minor loss; an excellent quality-vs-size balance. |
| Q4_K_M | 47.0 GB | ✓ yes | ~17 | Small but measurable loss; the popular default. |
| Q3_K_M | 36.1 GB | ✓ yes | ~23 | Noticeable degradation; only when you're tight on VRAM. |
Weights = params × bytes/weight, +20% for KV cache & runtime; usable VRAM is 72% (Apple unified memory) of nameplate. Tokens/sec is a bandwidth ceiling (800 GB/s) — real throughput is lower with long context. Try other combinations →
Running Llama 3.3 70B on a Apple M2 Ultra 192GB
At Q8_0, Llama 3.3 70B's weights take about 89 GB, inside the Apple M2 Ultra 192GB's 138 GB of usable memory. Near-lossless; rarely worth the extra space over Q6. Generation speed is bound by memory bandwidth — the GPU reads the whole model once per token — so expect on the order of 9 tokens/sec before context overhead.
Quantization is the lever
Each step down in precision shrinks the model: FP16 needs about 168 GB, Q4_K_M about 47 GB — a 72% reduction for a small, usually acceptable quality cost. Q5_K_M and Q6_K are near-lossless if you have the headroom; drop to Q3 only when you're genuinely out of VRAM. The quantization guide covers the tradeoffs in detail.
Frequently asked questions
Can a Apple M2 Ultra 192GB run Llama 3.3 70B?
Yes — Llama 3.3 70B fits on a Apple M2 Ultra 192GB at Q8_0 (89 GB of the 138 GB usable), with roughly 9 tokens/sec. Higher-precision quants need more VRAM; the table above shows each option.
How much VRAM does Llama 3.3 70B need?
Llama 3.3 70B is a 70B-parameter model. At FP16 that's about 168 GB; at Q4_K_M (the popular default) about 47 GB, including ~20% for the KV cache and runtime. Quantization is the main lever — see the per-quant table above.
Other combinations
Llama 3.3 70B on other GPUs: RTX 3090, RTX 4090, NVIDIA L4, RTX A6000, NVIDIA L40S, A100 40GB
Other models on the Apple M2 Ultra 192GB: Qwen2.5 72B, Mixtral 8x22B, Llama 3.1 405B
Related
- VRAM calculator — any model, quant and GPU.
- Running models locally — the hardware reality.
- All run-locally combinations →