Skip to main content
Pick the right hardware for local LLM inference. Every platform below is spec’d for memory, compute, bandwidth, and model capacity so you can compare at a glance and jump to the setup guide that matches.
Last verified: 2026-10-14. Prices and availability change; confirm with the manufacturer before buying.
If you already have hardware, start at the quick comparison table to see where your platform sits. If you are buying, use the decision guide to narrow down by budget and use case.

​
Hardware platforms

NVIDIA DGX Spark

Always-on inference server for large MoE modelsBest for teams running large MoE models 24/7 on Grace Blackwell silicon.See DGX Spark model compatibility for the full model list.

AMD Strix Halo

Compact workstation with up to 192 GB unified memoryAvailable with up to 192 GB unified memory on the AMD Ryzen AI Max+ Pro 495 developer platform. Found in Framework Desktop, mini-PCs, and laptops.
ROCm support is limited. Use the Vulkan backend with llama.cpp for best results on consumer hardware.

NVIDIA RTX 4090

Desktop workstation, single-user high-speed inferenceHigh memory bandwidth in a consumer GPU at a lower price than the 5090. Ideal for fast single-user inference.Pairs well with vLLM for high-throughput serving or Ollama for simple setup. See RTX workstation setup.

NVIDIA RTX A6000

Professional workstation, multi-user servingDouble the VRAM of a 4090 at lower peak bandwidth. Fits 70B models on 1 GPU.

NVIDIA RTX 5090

Consumer flagship with 32 GB GDDR7~39% faster than the RTX 4090 on 8B models (145 vs 104 tok/s). The extra 8 GB of VRAM over the 4090 runs 70B Q4 models that do not fit on 24 GB.Estimated tok/s: 7B ~162, 14B ~90, 32B dense ~45, 70B ~23. $1,999 MSRP.

NVIDIA RTX 5080

Mid-tier Blackwell with 16 GB GDDR7Best for 30B models at Q4. Gemma 4 E4B at Q8: ~141 tok/s. 14B models at Q4: ~83 tok/s. ~$1,000 MSRP.

NVIDIA RTX 5070 Ti / 5070 / 5060

Entry-level Blackwell consumer cardsAll RTX 50 series have native FP4 tensor cores. Software support: Ollama, llama.cpp, vLLM, SGLang, LM Studio.

NVIDIA RTX 4080 Super / 4070 Ti Super

Previous-gen Ada Lovelace mid-tierStill strong performers with full CUDA support. 14-29% slower than RTX 50 series at same VRAM tier.

NVIDIA RTX 3090 / 3080 / 3070 / 3060

Previous-gen Ampere — new life in 2026Why older cards have new life in 2026:
  1. Smaller, smarter models: Qwen3.6-35B-A3B (3B active), Phi-4-mini (3.8B), Gemma 4 E4B
  2. Better quantization: NVFP4, MXFP4, dynamic quants preserve quality at lower bits
  3. MoE architecture: 35B total but only 3B active = fits in 16GB
  4. CPU offloading: llama.cpp and KTransformers spill to system RAM when VRAM is tight
  5. Used market value: RTX 3090 at ~$800 is the best price-per-VRAM in 2026
The RTX 3090 (24GB) is the best value in 2026 for budget builders. It runs 70B Q4 models that newer 16GB cards cannot fit.

NVIDIA RTX PRO 6000 Blackwell

Professional workstation flagship — 96 GB GDDR73 variants:
  • Workstation (~$16,000): Full tower, active cooling, display outputs
  • Max-Q (~$8,500): Low-profile, 4x fit in one workstation (384 GB total), enables V4.1 Flash at high precision
  • Server (pricing varies): Rack-mount, datacenter cooling, no display outputs
See RTX PRO Blackwell variants for the full comparison.

NVIDIA RTX PRO 5000 / 4500 / 4000

Professional Blackwell mid-tierProfessional drivers, ECC memory support, and ISV certifications. Full CUDA support across all backends (Ollama, vLLM, llama.cpp, LM Studio).

NVIDIA RTX 6000D

China export-compliance variant — 84 GBCut-down RTX PRO 6000 for China market compliance. 84 GB instead of 96 GB. Performance is ~12% lower than the full 96 GB variant. Available through authorized China distributors.

CMP 170HX (unlocked)

Budget hacker tier — 64 GB HBM2e for $250-800What is it? NVIDIA CMP 170HX is a mining card based on the Ampere GA100 GPU. It ships artificially capped at 8 GB VRAM, but the full 64 GB HBM2e can be unlocked.Unlock methods:
  • Vix Loader (recommended): No-flashing unlock via Windows registry keys. Fully reversible. Requires driver 616.56+. See CMP 170HX unlock guide.
  • Linux kernel patching: For server deployments. Requires Proxmox VE or bare-metal Linux.
Performance:
  • Single card: 70B models at Q4/Q8, ~40-60 tok/s
  • 4x CMP 170HX: 816 tok/s aggregate on GLM-5.3-Flash
  • 8x CMP 170HX: ~30 tok/s on GLM-5.2 at 4-bit
Unsupported path: NVIDIA may patch the unlock in future drivers. No display outputs. Passive cooling requires directed airflow. Best for tinkerers and budget builders who accept the risk.

Tesla P40 / V100 (used)

Datacenter cast-offs — cheap VRAM per GBPros: Extremely cheap per GB ($4-12/GB vs $50-170/GB for pro cards). Full CUDA support. Cons: No display outputs. Passive cooling (requires server chassis or custom fans). Older architecture (Pascal/Volta) lacks FP8/FP4 tensor cores.Best for: Budget builders who need 24+ GB VRAM and accept the tinkering overhead.

Apple Mac Studio (M4 Ultra)

Large models on macOS, updated for 2025Runs 70B Q4 at ~18-24 tok/s. Found in Mac Studio and Mac Pro. A mid-point between M2 Ultra and M5 Ultra for large-model inference on macOS.

Apple Mac Studio (M5 Ultra)

Largest unified memory desktop — up to 512 GB50% more bandwidth than M3 Ultra. Runs DeepSeek V4-Flash-0731 at native precision (155 GB) on the 256 GB configuration. ~20-25% faster than M2 Ultra on LLM inference. Mac Studio M5 Ultra from $5,499.

Apple Mac Studio (M5 Max)

Mid-range Apple Silicon for 70B modelsRuns 70B Q4 models with room to spare. Mac Studio M5 Max from $2,499. A strong mid-point between M4 Max and M5 Ultra for macOS inference.

Apple Mac Studio (M2 Ultra)

Silent operation, large models on macOS192 GB unified memory runs 120B models silently. Superseded by M4 Ultra and M5 Ultra but still a strong option for large-model inference on macOS.Use LM Studio or Ollama with Metal acceleration. See Mac Studio setup.

AMD Ryzen AI Max+ Pro 495

Mini PC with up to 192 GB unified memoryAvailable in Framework Desktop, Minisforum MS-S1 MAX-P495, and other mini PCs. The 192 GB configuration runs MiniMax M2.7 (230B-284B MoE) at Q4 quantization. The 160 GB GPU-accessible portion is the key advantage over standard 128 GB systems.
ROCm support is limited. Use Vulkan backend with llama.cpp for best results on consumer hardware.

NVIDIA RTX Spark

Laptop form factor with unified memoryRTX Spark combines a 20-core Grace CPU with a Blackwell RTX GPU in a laptop form factor. Windows-native (unlike DGX Spark’s Linux). Runs 120B-parameter models according to NVIDIA. Available in 64 GB and 128 GB configurations.
RTX Spark drops the ConnectX-7 networking found in DGX Spark, so clustering is not supported. Best for single-device inference.

Multi-GPU (2x RTX 5090 / 2x RTX 4090 / 2x A6000)

Models too large for 1 GPUUse tensor parallelism across 2 GPUs when 1 GPU is not enough. See 1 vs 2 DGX Sparks for clustering patterns.

Consumer laptop (16 GB)

7B models, offline AI, travelRun 7B models on any laptop with no GPU required. Good for offline coding assistants and travel.Pair with Ollama or llama.cpp for the simplest local setup, then connect through Tokios to reach the model from your phone or another device.

Phone / edge device

On-device inference, maximum privacyRun small models directly on your phone. No network, no cloud, no latency.See From your phone for how to reach a home server model from iOS and Android chat apps.
DeepSeek-V4-Flash-0731 is a special case: its roughly 162–167 GB weight floor exceeds most single-GPU setups. The Mac Studio M5 Ultra (256 GB config) runs it at native precision. 2x 96 GB RTX PRO 6000 Blackwell cards also have enough aggregate VRAM, but 0731 needs community validation on that configuration. Compare the M5 Ultra, 2x RTX PRO 6000, 2x DGX Spark, and 2x Strix Halo paths before buying hardware.

​
Quick comparison

Token speeds are rough estimates for 7B Q4 inference and vary by backend, quantization, and context length. Benchmark your own setup with Test your deployment.

​
Runtime compatibility

Not every runtime works on every platform. Here is what each hardware tier supports today.
If your hardware has limited runtime support, start with Ollama or llama.cpp — they have the broadest platform coverage.

​
VRAM tier decision matrix

Not sure what fits your GPU? Start here. This table maps VRAM to the largest model class you can run at practical quantization levels.
MoE models punch above their weight. A 35B MoE model like Qwen3.6-35B-A3B has 35B total parameters but only 3B active per token. It fits in 16GB and runs at 120+ tok/s, but performs closer to a 35B dense model. Always check active parameter count, not just total.
Frontier models need multi-GPU or unified memory. DeepSeek V4.1 Flash (763B total params) requires at least 2x DGX Spark (256GB) or 2x RTX PRO 6000 (192GB) at mixed precision. See DeepSeek V4.1 Flash on 2 DGX Sparks for the full breakdown.

​
Power and form factor

Local inference hardware ranges from a phone in your pocket to a rack-mount server. Power draw and noise matter if the box lives on your desk or in a shared office. If the machine runs 24/7 in a shared space, prioritize low idle power and quiet cooling. Mac Studio and DGX Spark are the best always-on options for office environments.

​
Scaling beyond 1 machine

When 1 box is not enough, you have 3 options:
  1. Add a second GPU in the same machine using tensor parallelism (TP=2). Works well with 2x RTX 5090 (64 GB total), 2x RTX 4090, or 2x A6000. See Model sharding for the configuration.
  2. Cluster multiple nodes — for example, 2 DGX Sparks connected over the network can serve selected distributed models whose weights, cache, and runtime overhead fit below 256 GB. See 1 vs 2 DGX Sparks for the trade-offs.
  3. Offload to a remote server — run the model on a dedicated GPU server and reach it through Tokios from any client. See vLLM remote access and Ollama remote access.
For team serving, the key question is whether you need more throughput or more model capacity. Additional GPUs can improve either, but scaling depends on the model, parallelism strategy, and interconnect. Measure the full workload instead of assuming linear throughput.

​
Which hardware should you choose?

RTX 3060 (12 GB, ~$200-250 used) runs 13B models at Q4 with ~50 tok/s. **RTX 3070** (8 GB, ~$300-400 used) runs 7B models at ~61 tok/s. Both are entry points into local LLM with full CUDA support. For the absolute minimum, a consumer laptop with 16 GB RAM runs 7B models through Ollama or llama.cpp at 15-30 tok/s.
RTX 3090 (24 GB, ~$700-900 used) is the best value in 2026 for budget builders. It runs 70B Q4 models that newer 16GB cards cannot fit, with ~118 tok/s on 7B. **RTX 3080** (10 GB, ~$400-500 used) runs 13B models at ~80 tok/s. If you accept the tinkering overhead, a CMP 170HX (64 GB HBM2e after unlock, ~$250-800) or **Tesla P40** (24 GB, ~$150-250 used) offer extreme VRAM per dollar — see Budget hacker tier below.
CMP 170HX (64 GB HBM2e after unlock) is the most compelling budget story in local LLM. Originally an Ampere GA100 mining card capped at 8 GB, it unlocks to full 64 GB with 732-1,100 GB/s bandwidth. Single card runs 70B Q4-Q8 at ~40-60 tok/s. 4x CMP 170HX achieves 816 tok/s aggregate on GLM-5.3-Flash. Tesla V100 (32 GB HBM2, ~$400-600 used) runs 70B Q4 at ~70 tok/s. **Tesla P40** (24 GB GDDR5X, ~$150-250 used) is slower (~35 tok/s on 7B) but extremely cheap per GB.
Unsupported path: These cards require tinkering. No display outputs. Passive cooling needs directed airflow. CMP 170HX unlock may break with future NVIDIA drivers. Best for tinkerers who accept the risk. See CMP 170HX unlock guide for the Vix Loader method.
RTX 5090 (32 GB, $1,999 MSRP) is the sweet spot. 32 GB GDDR7 fits 70B Q4 models that the 4090's 24 GB cannot, with ~39% faster inference on small models (~162 tok/s on 7B). **RTX 4090** (24 GB, ~$1,600-2,000) remains strong if you find one at lower price — highest consumer memory bandwidth at 1008 GB/s. RTX 5080 (16 GB, ~$1,000) is the mid-tier choice for 30B models at ~100 tok/s.
Mac Studio M5 Ultra (256-512 GB unified, ~$5,499+) fits the largest models of any desktop — 120B+ at Q4, and DeepSeek V4-Flash at native precision on the 256 GB config. **Mac Studio M4 Ultra** (192 GB) and **M2 Ultra** (192 GB) also run 120B models silently. **DGX Spark** (128 GB, ~$3,000) runs 70B FP8 natively and 120B MoE models. Strix Halo (up to 192 GB unified, ~$1,500-5,000) is a compact alternative with Vulkan-based inference. **RTX PRO 6000** (96 GB, ~$8,500-16,000) fits 120B MoE at FP8 on a single card.
DeepSeek V4.1 Flash (763B total params) requires multi-GPU or large unified memory. Options:
  • 2x DGX Spark (256 GB total, ~$6,000): Mixed FP8+MXFP4 precision, 38-73 tok/s. Community-tested. See V4.1 Flash on 2 DGX Sparks.
  • Mac Studio M5 Ultra 512 GB (~$11,300+): V4.1 Flash fits at Q2_K-Q4 (~246-402 GiB), ~10-20 tok/s.
  • 4x RTX PRO 6000 Max-Q (384 GB total, ~$34,000): V4.1 Flash at high precision, ~135 tok/s per GPU.
  • 4x DGX Spark (512 GB total, ~$12,000): V4.1 Flash at NVFP4, 87-125 tok/s.
  • 8x CMP 170HX (512 GB HBM2e, ~$2,000-6,400): Budget path to 512 GB, ~30 tok/s on GLM-5.2.
GLM-5.2 (753B params) needs at least 4x DGX Spark at Q4 (~377 GB weight floor). See GLM-5.2 feasibility.
2x RTX PRO 6000 (192 GB total with NVLink, ~$32,000) serves V4.1 Flash at FP8 with ~140 tok/s per GPU. **2x RTX 5090** (64 GB total, ~$4,000) is a faster consumer alternative for 120B MoE models. 2x DGX Spark (256 GB total, ~$6,000) is a turnkey alternative with 128 GB unified memory per node and ConnectX-7 RoCE networking. For budget team serving, **4x CMP 170HX** (256 GB HBM2e total, ~$1,000-3,200) offers extreme value if you accept the unsupported path.
Mac Studio (any Apple Silicon) runs at 30-150W with near-silent cooling. It is the quietest option for always-on inference in an office or home environment. The M5 Ultra tops out at 512 GB unified memory for the largest models. DGX Spark (65-200W) is also quiet and designed for always-on server use.
Strix Halo with up to 192 GB unified memory on the Ryzen AI Max+ Pro 495 developer platform. Available in Framework Desktop, mini-PCs, and laptops. Use llama.cpp with the Vulkan backend — ROCm support is limited on consumer hardware. See the Strix Halo setup guide for details.
Phone / edge device for on-device inference with no network required. Run 3B-7B models through llama.cpp, MLC-LLM, or MediaPipe.

​
How memory maps to model size

The single biggest constraint for local LLM inference is memory. Here is how common model sizes map to the RAM or VRAM you need at different quantization levels.
Always leave 2-4 GB free for the KV cache and runtime overhead. A 70B Q4_K_M model that “fits” in 40 GB of VRAM still needs headroom for context — plan for 44-48 GB total. See KV cache and context for the math.
Frontier-scale models like Kimi K3 (2.8T parameters) need ~1.4 TB even at 4-bit quantization — no single desktop or prosumer setup fits them. Moonshot recommends a supernode with 64+ accelerators. See Kimi K3 hardware requirements for the full picture.
See Find a model that fits your hardware for a deeper look at how to match model size to your hardware, and Quantization concepts for what the quant levels mean.

​
Common hardware mistakes

  • Buying for TFLOPS, not memory. A 4090 or 5090 has more raw compute than an A6000, but the A6000’s 48 GB VRAM fits 70B models that the 4090’s 24 GB cannot. Even the 5090’s 32 GB is tight for 70B at higher quantizations. Memory determines what runs; TFLOPS determines how fast.
  • Ignoring KV cache overhead. A model that “fits” your VRAM at load time may still OOM during inference because the KV cache grows with context length. Budget 2-4 GB extra.
  • Assuming multi-GPU is free. Tensor parallelism across 2 GPUs requires fast interconnect (NVLink or PCIe 4.0 x16). Over PCIe, expect 10-20% throughput loss versus a single larger GPU.
  • Overlooking power and cooling. A 575W RTX 5090 or 450W RTX 4090 in a small office generates real heat and noise. If the box runs 24/7, factor in the electricity cost and acoustic impact.
  • Skipping the backend check. Not every runtime supports every GPU. Confirm your chosen backend works on your hardware before buying — see Runtime compatibility above.

​
Next steps

Choose a local model

Pick a model and backend based on your workflow, hardware, and latency needs.

Model fit

Calculate which model sizes and quantizations fit your GPU or unified memory.

Model sharding

Split models across multiple GPUs when 1 GPU is not enough.

DGX Spark model compatibility

See which models work on DGX Spark and at what quantization levels.

Strix Halo model compatibility

Check which models work on AMD Strix Halo with Vulkan and ROCm backends.

Test your deployment

Compare latency and throughput across deployments once traffic is flowing.

Supported backends

Configure Ollama, LM Studio, llama.cpp, vLLM, or SGLang as your upstream.

Quickstart

Go from zero to a working connector, deployment, and API key.