Skip to main content
A 70B model at FP16 needs ~140 GB of VRAM. No single consumer GPU holds that. Sharding is how inference engines split a model across multiple GPUs — or multiple machines — so you can serve models that physically cannot fit on 1 device, or serve more users than 1 device handles. This page explains the 4 sharding strategies, how each one splits the work, and which one fits your hardware. For a quick TP-vs-PP decision, see Tensor vs pipeline parallelism.

​
Why shard?

You reach for sharding when 1 of 4 things is true:
  1. The model is too large for 1 GPU. A 70B dense model at FP16 needs ~140 GB. Even at Q4 quantization (~35 GB), it fills a single RTX 4090 (24 GB). You need at least 2 GPUs to hold the weights.
  2. You need lower latency per token. When every GPU contributes to each token simultaneously, the wall-clock time per token drops.
  3. You need higher throughput. More GPUs means more concurrent users served in parallel.
  4. You need redundancy. Running 2 replicas behind a load balancer means 1 GPU can fail without downtime.
If the model already fits on 1 GPU and you have no throughput problem, stay on 1 GPU. Sharding adds complexity and communication overhead.

​
Where your GPU memory actually goes

Before you pick a sharding strategy, know what competes for VRAM. It is not just model weights: A 24 GB RTX 4090 running a 70B Q4 model has ~35 GB of weights — it does not fit on 1 GPU at all. With TP=2, each GPU holds ~17.5 GB of weights and has ~6.5 GB left for KV cache and overhead. That is enough for short contexts and a few concurrent users, but not much more.
Usable memory is less than total VRAM. On a DGX Spark (128 GB unified), expect ~110 GB usable for model + KV after the OS and runtime reserve their share. On discrete GPUs, vLLM’s --gpu-memory-utilization (default 0.90) controls this. Set it lower if you hit OOM.
For MoE models, the breakdown looks different — the expert planes dominate: This is why expert parallelism exists: instead of loading all 231 GB of experts onto every GPU, each GPU holds only its assigned experts. See Expert parallelism below.

​
2 decisions: make it fit, then make it fast

Sharding answers 2 questions in order: Make it fit — get the weights (and headroom for KV cache) inside your GPU memory budget:
  • Quantize first. Going from FP16 to Q4 cuts weight memory by ~4×. This is the cheapest win. See the quantization guide.
  • Shard across GPUs. When quantization alone is not enough, split the model with TP, PP, or EP.
  • Prune experts (MoE only). For MoE models with 256 experts where only 6 fire per token, REAP-style expert pruning cuts routed-expert weights roughly in half without touching active compute.
Make it fast — once it fits, reduce bytes per token and decode steps:
  • Pick the right parallelism. TP gives lowest per-token latency. PP tolerates slow interconnects. DP multiplies throughput.
  • Use MTP / speculative decoding. If the model ships multi-token prediction heads (Qwen3.6, DeepSeek-V3), enable them — they speed up decode without a quality loss. See model serving and scaling.
  • Tune KV cache and batch size. More KV cache headroom means more concurrent users. See DGX Spark memory tuning.

​
The 4 sharding strategies at a glance

​
Tensor parallelism: splitting layers across GPUs

Tensor parallelism (TP) slices every layer’s weight matrices across GPUs. Each GPU holds a fraction of every layer, and all GPUs work on the same token simultaneously.
After every layer, the GPUs run an AllReduce operation — they combine their partial results into the full output. This keeps per-token latency low because every GPU contributes, but it demands high bandwidth between GPUs. Memory: each GPU holds 1/N of the model weights, where N is the TP degree. A 70B model at Q4 (~35 GB) with TP=2 puts ~17.5 GB of weights per GPU. Latency: lowest per token. All GPUs compute on every token, so the wall-clock time is the compute time plus the AllReduce synchronization. Best for: 2–4 GPUs connected by NVLink or a fast PCIe bus. The 200Gb/s QSFP link between 2 DGX Sparks is ideal. vLLM flag: --tensor-parallel-size 2
PCIe delivers 3–5× less bandwidth than NVLink for AllReduce operations. Tensor parallelism over PCIe still works — 2× RTX 4090 at TP=2 is a common setup — but expect 20–30% lower decode throughput than the same model on NVLink-connected GPUs.

​
Example: 70B on 2× RTX 4090

Each GPU holds ~17.5 GB of Q4 weights. The remaining ~6.5 GB per GPU covers KV cache and CUDA graphs.

​
Pipeline parallelism: splitting layers sequentially

Pipeline parallelism (PP) assigns contiguous groups of layers to different GPUs. GPU 0 holds layers 0–19, GPU 1 holds layers 20–39. Tokens flow through the pipeline like an assembly line.
Each GPU processes its assigned layers and passes the activations to the next stage. The data that crosses GPUs is the activation tensor for a single layer boundary — much smaller than the full weight matrices that TP exchanges. Memory: each GPU holds its portion of layers. With PP=2, GPU 0 holds the first half of the model, GPU 1 holds the second half. Latency: additive. A token must pass through every stage sequentially. With PP=2, the latency is roughly the sum of both stages. Communication: point-to-point between adjacent stages. Only neighboring GPUs talk, so the bandwidth requirement is low. This makes PP suitable for slower interconnects — PCIe, Ethernet, or network links between machines. The bubble problem: when the batch size is small, later GPUs sit idle while earlier GPUs process their layers. With PP=2 and batch size 1, GPU 1 waits for GPU 0 to finish every single layer boundary. Larger batches amortize this — vLLM’s continuous batching helps fill the pipeline. vLLM flag: --pipeline-parallel-size 2

​
When PP beats TP

Use pipeline parallelism over tensor parallelism when:
  • GPUs are in different machines connected by a network (10GbE–100GbE).
  • GPUs are connected by PCIe and the model is small enough that PP’s bubble waste costs less than TP’s AllReduce bandwidth.
  • You chain 3+ nodes where TP across all of them would saturate the interconnect.

​
Data parallelism: running the same model on multiple GPUs

Data parallelism (DP) runs a full copy of the model on each GPU and distributes incoming requests across them. There is no inter-GPU communication during inference — each GPU independently handles its own batch of requests.
Memory: N copies of the full model. A 7B model at Q4 (~4 GB) with DP=4 uses 16 GB total across 4 GPUs. This is the expensive part — you trade memory for throughput. Latency: identical to single-GPU inference. Each GPU processes tokens independently. Throughput: scales linearly. 4 replicas serve ~4× the concurrent users. Best for: models that fit on 1 GPU (7B, 14B, sometimes 32B at aggressive quantization) when you need to serve many users simultaneously. vLLM setup: launch multiple vLLM instances and put a load balancer in front.
Then use nginx, HAProxy, or any round-robin balancer to distribute /v1/chat/completions requests across both ports.

​
Expert parallelism: for mixture-of-experts models

Expert parallelism (EP) applies only to mixture-of-experts (MoE) models. In an MoE model, each token activates only a small subset of the model’s “experts” — for example, Qwen3-235B has 256 routed experts but only 6 fire per token (~22B active parameters out of 235B total). EP puts different experts on different GPUs so you do not waste memory storing inactive experts everywhere.
Memory: each GPU holds only its assigned experts. The dense stack (attention, embeddings, shared layers — ~14 GB at 4-bit for Qwen3-235B) is replicated on every GPU. The expert planes (~231 GB at 2-bit) are sharded. This is far more memory-efficient than TP, which would replicate all 256 experts on every GPU even though most sit idle for each token. Communication: all-to-all between GPUs. When a token’s router selects an expert on a different GPU, the activation must be sent there. This creates variable traffic patterns — some tokens stay local, others cross GPUs. vLLM flag: --enable-expert-parallel combined with --tensor-parallel-size
DGX Spark’s 128 GB unified memory is well-suited for expert parallelism. A single Spark with ~110 GB usable can hold the dense stack plus a large share of experts locally, reducing the cross-GPU routing traffic that EP generates. For models like Qwen3-235B where even EP on 1 node is not enough, combine expert pruning with 2-bit expert quantization to bring the per-node footprint under budget. See the DGX Spark use case for hardware details.

​
Combining strategies: TP, PP, and DP

When you have many GPUs, you combine strategies hierarchically:
  • TP within a node (where the interconnect is fast).
  • PP across nodes (where the interconnect is slower).
  • DP for throughput (run multiple replicas of the combined TP+PP setup).

​
4 GPUs: TP=2 + PP=2

This creates 2 pipeline stages, each using 2-way tensor parallelism. GPUs 0 and 1 handle the first half of layers with TP, GPUs 2 and 3 handle the second half with TP.

​
8 GPUs: TP=4 + DP=2

Run 2 replicas of a 4-way tensor-parallel setup. Each replica uses 4 GPUs for TP, and a load balancer distributes requests between the 2 replicas. This gives you the low latency of TP=4 with 2× the throughput.

​
The rule of thumb

  1. TP first — within the fastest interconnect you have (NVLink, QSFP, PCIe).
  2. PP second — across slower links (Ethernet, network between machines).
  3. DP last — when you need more throughput and have spare GPUs for replicas.
vLLM requires TP × PP = total GPUs used for one replica. If you set --tensor-parallel-size 4 --pipeline-parallel-size 2, vLLM uses 8 GPUs for 1 replica.

​
Sharding decision matrix

70B, 2× RTX 4090 (PCIe)

PP. PCIe bandwidth is too low for efficient AllReduce. Pipeline parallelism avoids the bottleneck.

70B, 2× A100 (NVLink)

TP. NVLink delivers enough bandwidth for AllReduce. Lower latency per token.

235B MoE, DGX Spark

EP. Native MoE handling — each GPU holds only its experts, not the full 235B.

7B, high traffic

DP. Model fits on 1 GPU. Replicate for throughput.

70B, 4× A100 (NVLink)

TP=4 (or TP=2 + DP=2 if you need throughput over latency).

70B, 2 machines

PP=2. Network between machines is too slow for TP. Pipeline avoids the bandwidth wall.

​
Practical commands

​
Verify sharding is working

Run nvidia-smi on every GPU. All participating GPUs should show memory allocated for the model:
With TP=2, you should see roughly equal memory usage on both GPUs. With PP=2, the memory split may be uneven if the model’s early layers differ in size from later layers.

​
Common mistakes

Using TP over slow PCIe. AllReduce sends activation data after every layer. On PCIe 3.0, this chokes decode throughput. If your GPUs lack NVLink, test PP and compare. Using PP with batch size 1. The pipeline bubble wastes the most compute when only 1 token flows at a time. vLLM’s continuous batching mitigates this — make sure you run with enough concurrent requests to fill the pipeline. Forgetting that DP requires N× model memory. 4 replicas of a 70B Q4 model (~35 GB each) need 140 GB total. Plan memory before adding replicas. Not accounting for KV cache. Model weights are only part of GPU memory. The KV cache for active sequences takes the rest. When you shard weights across 2 GPUs, you also shard the KV cache — effectively doubling your concurrent user capacity. But if you plan memory for weights alone, you will OOM. Budget for weights + KV cache + ~3 GB overhead before you pick a sharding strategy. Mixing TP and PP in the wrong order. vLLM applies TP within each pipeline stage. Set --tensor-parallel-size to the GPU count within a single node and --pipeline-parallel-size to the node count. Reversing them puts PP within a node (wasting fast interconnect on point-to-point) and TP across nodes (saturating the network with AllReduce).

​
Next steps

Tensor vs pipeline parallelism

Detailed TP vs PP comparison with performance benchmarks.

One vs two DGX Sparks

When to add a second Spark and how to configure the 200Gb/s link.

DGX Spark memory tuning

KV cache math and memory parameters for sharded deployments.

Model serving and scaling

How inference engines manage batching, precision, and parallelism.