Skip to main content
Pick the right hardware for local LLM inference. Every platform below is spec’d for memory, compute, bandwidth, and model capacity so you can compare at a glance and jump to the setup guide that matches. If you already have hardware, start at the quick comparison table to see where your platform sits. If you are buying, use the decision guide to narrow down by budget and use case.

​
Hardware platforms

NVIDIA DGX Spark

Always-on inference server for large MoE modelsBest for teams running large MoE models 24/7 on Grace Blackwell silicon.See DGX Spark model compatibility for the full model list.

AMD Strix Halo (Radeon 8060S)

Compact workstation with 128 GB unified memory
ROCm support is limited. Use the Vulkan backend with llama.cpp for best results.

NVIDIA RTX 4090

Desktop workstation, single-user high-speed inferenceHighest memory bandwidth in a consumer GPU. Ideal for fast single-user inference.Pairs well with vLLM for high-throughput serving or Ollama for simple setup. See RTX workstation setup.

NVIDIA RTX A6000

Professional workstation, multi-user servingDouble the VRAM of a 4090 at lower peak bandwidth. Fits 70B models on 1 GPU.

Apple Mac Studio (M2 Ultra)

Silent operation, large models on macOSLargest unified memory on any desktop. Runs 120B models silently.Use LM Studio or Ollama with Metal acceleration. See Mac Studio setup.

Multi-GPU (2x RTX 4090 / 2x A6000)

Models too large for 1 GPUUse tensor parallelism across 2 GPUs when 1 GPU is not enough. See 1 vs 2 DGX Sparks for clustering patterns.

Consumer laptop (16 GB)

7B models, offline AI, travelRun 7B models on any laptop with no GPU required. Good for offline coding assistants and travel.Pair with Ollama or llama.cpp for the simplest local setup, then connect through Tokios to reach the model from your phone or another device.

Phone / edge device

On-device inference, maximum privacyRun small models directly on your phone. No network, no cloud, no latency.See From your phone for how to reach a home server model from iOS and Android chat apps.

​
Quick comparison

Token speeds are rough estimates for 7B Q4 inference and vary by backend, quantization, and context length. Benchmark your own setup with Benchmark a deployment.

​
Runtime compatibility

Not every runtime works on every platform. Here is what each hardware tier supports today.
If your hardware has limited runtime support, start with Ollama or llama.cpp — they have the broadest platform coverage.

​
Power and form factor

Local inference hardware ranges from a phone in your pocket to a rack-mount server. Power draw and noise matter if the box lives on your desk or in a shared office. If the machine runs 24/7 in a shared space, prioritize low idle power and quiet cooling. Mac Studio and DGX Spark are the best always-on options for office environments.

​
Scaling beyond 1 machine

When 1 box is not enough, you have 3 options:
  1. Add a second GPU in the same machine using tensor parallelism (TP=2). Works well with 2x RTX 4090 or 2x A6000. See Model sharding for the configuration.
  2. Cluster multiple nodes — for example, 2 DGX Sparks connected over the network can serve models up to ~240B parameters. See 1 vs 2 DGX Sparks for the math.
  3. Offload to a remote server — run the model on a dedicated GPU server and reach it through Tokios from any client. See vLLM remote access and Ollama remote access.
For team serving, the key question is whether you need more throughput (add GPUs) or more model capacity (add memory). Throughput scales linearly with GPU count; model capacity is constrained by total VRAM or unified memory.

​
Which hardware should you choose?

Use a consumer laptop with 16 GB RAM running 7B models through Ollama or llama.cpp. If you have a desktop with a PCIe slot, a used RTX 3090 (24 GB) offers strong value at $700-900 and fits 30B models at Q4.
RTX 4090 (24 GB). Highest consumer memory bandwidth at 1008 GB/s, fast token generation, and wide backend support across Ollama, vLLM, llama.cpp, and LM Studio. Fits 30B Q4 models comfortably.
Mac Studio (192 GB unified) fits the largest models of any desktop — 120B at Q4. DGX Spark (128 GB) runs 70B FP8 natively and 120B MoE models. Strix Halo (128 GB unified) is a compact alternative with Vulkan-based inference.
2x A6000 with NVLink gives 96 GB total VRAM and serves multiple users at once. DGX Spark is a turnkey alternative with 128 GB unified memory and no multi-GPU plumbing.
Mac Studio runs at 60-150W with near-silent cooling. It is the quietest option for always-on inference in an office or home environment.
Strix Halo with 128 GB unified memory. Use llama.cpp with the Vulkan backend — ROCm support is limited on this platform. See the Strix Halo setup guide for details.
Phone / edge device for on-device inference with no network required. Run 3B-7B models through llama.cpp, MLC-LLM, or MediaPipe.

​
How memory maps to model size

The single biggest constraint for local LLM inference is memory. Here is how common model sizes map to the RAM or VRAM you need at different quantization levels.
Always leave 2-4 GB free for the KV cache and runtime overhead. A 70B Q4_K_M model that “fits” in 40 GB of VRAM still needs headroom for context — plan for 44-48 GB total. See KV cache and context for the math.
Frontier-scale models like Kimi K3 (2.8T parameters) need ~1.4 TB even at 4-bit quantization — no single desktop or prosumer setup fits them. Moonshot recommends a supernode with 64+ accelerators. See Kimi K3 hardware requirements for the full picture.
See Find a model that fits your hardware for a deeper look at how to match model size to your hardware, and Quantization concepts for what the quant levels mean.

​
Common hardware mistakes

  • Buying for TFLOPS, not memory. A 4090 has more raw compute than an A6000, but the A6000’s 48 GB VRAM fits 70B models that the 4090’s 24 GB cannot. Memory determines what runs; TFLOPS determines how fast.
  • Ignoring KV cache overhead. A model that “fits” your VRAM at load time may still OOM during inference because the KV cache grows with context length. Budget 2-4 GB extra.
  • Assuming multi-GPU is free. Tensor parallelism across 2 GPUs requires fast interconnect (NVLink or PCIe 4.0 x16). Over PCIe, expect 10-20% throughput loss versus a single larger GPU.
  • Overlooking power and cooling. A 450W RTX 4090 in a small office generates real heat and noise. If the box runs 24/7, factor in the electricity cost and acoustic impact.
  • Skipping the backend check. Not every runtime supports every GPU. Confirm your chosen backend works on your hardware before buying — see Runtime compatibility above.

​
Next steps

Choose a local model

Pick a model and backend based on your workflow, hardware, and latency needs.

Model fit

Calculate which model sizes and quantizations fit your GPU or unified memory.

Model sharding

Split models across multiple GPUs when 1 GPU is not enough.

DGX Spark model compatibility

See which models work on DGX Spark and at what quantization levels.

Strix Halo model compatibility

Check which models work on AMD Strix Halo with Vulkan and ROCm backends.

Benchmark a deployment

Compare latency and throughput across deployments once traffic is flowing.

Supported backends

Configure Ollama, LM Studio, llama.cpp, vLLM, or SGLang as your upstream.

Quickstart

Go from zero to a working connector, deployment, and API key.