Tensor parallelism vs pipeline parallelism for local LLM inference

When your model does not fit on a single GPU — or you need more throughput than 1 GPU delivers — you split the workload across multiple GPUs. The 2 main strategies are tensor parallelism (TP) and pipeline parallelism (PP). They solve different problems, and picking the wrong one wastes hardware.

Quick verdict

Scenario	Strategy	Why
Single node, 2+ GPUs (DGX Spark pair, 2× RTX 4090)	TP	Lower latency, better GPU-to-GPU bandwidth usage
Multi-node (2+ machines over network)	PP	Tolerates network latency; each node holds a model shard
Large model + high concurrency	TP + PP hybrid	TP within a node, PP across nodes
Throughput scaling (many users, model fits on 1 GPU)	Data parallelism (DP)	Replicate the model, split the requests

Use tensor parallelism for single-node multi-GPU. Use pipeline parallelism for multi-node. This rule covers most setups.

Tensor parallelism explained

Tensor parallelism splits each layer’s weight matrices across GPUs. When a forward pass runs, every GPU computes a slice of the matrix multiplication simultaneously, then they synchronize via a high-bandwidth interconnect (NVLink, PCIe, or InfiniBand).

Layer N:
  GPU 0: weights[0:50%] × input  ──┐
                                    ├── all-reduce ──→ output
  GPU 1: weights[50:100%] × input ─┘

Key properties:

Every GPU holds a fraction of every layer
All GPUs are active on every token
Requires high-bandwidth interconnect (NVLink >> PCIe)
Adds synchronization overhead per layer, but keeps latency low per token
vLLM calls this --tensor-parallel-size

Best for: 2× DGX Spark connected via 200Gb/s QSFP, 2× RTX 4090 over PCIe, or any single-node setup where GPUs share a fast bus.

Pipeline parallelism explained

Pipeline parallelism assigns different layers to different GPUs. GPU 0 holds layers 0–15, GPU 1 holds layers 16–31. A forward pass flows like an assembly line: GPU 0 processes the input, passes activations to GPU 1, which continues.

GPU 0: [Layer 0–15]  ──activations──→  GPU 1: [Layer 16–31]  ──→ output

Key properties:

Each GPU holds a contiguous shard of layers
Only 1 GPU is active per token at a time (the others are idle or processing different tokens)
Tolerates slower interconnects — activations are smaller than full weight matrices
Adds pipeline bubbles (idle time) that reduce utilization
vLLM calls this --pipeline-parallel-size

Best for: multi-node clusters where nodes are connected over 10GbE–100GbE network, or when the model is too large for TP alone.

Decision matrix

Answer these questions to pick your strategy:

Does the model fit on 1 GPU?
- Yes → no parallelism needed. Use data parallelism if you need more throughput.
- No → continue.
Are all GPUs in the same machine?
- Yes → tensor parallelism (lower latency, better decode throughput).
- No → continue.
Are GPUs across 2+ machines?
- Yes → pipeline parallelism across nodes. If each node has 2+ GPUs, use TP within each node + PP across nodes.
Is it a very large model (405B+)?
- Yes → hybrid TP + PP. Example: TP=2 within each DGX Spark, PP=2 across 2 Sparks.

Hardware-specific recommendations

Hardware	GPUs	Interconnect	Recommended strategy
2× DGX Spark (GB10)	2 unified	200Gb/s QSFP	TP=2
2× RTX 4090	2 discrete	PCIe 4.0	TP=2
2× RTX 3090	2 discrete	PCIe 4.0	TP=2 (watch OOM — vLLM reserves more than Ollama)
2-node DGX Spark cluster	2 nodes	200Gb/s QSFP	PP=2 (or TP=2 + PP=2 for 405B)
3+ node cluster	3+ nodes	10–100GbE	PP = node count, TP within each node

Data parallelism and expert parallelism

Two other strategies matter for specific scenarios: Data parallelism (DP) — replicate the full model on each GPU and split incoming requests across them. Use this when the model already fits on 1 GPU and you want more concurrent users. vLLM supports this natively — just launch multiple instances behind a load balancer. Expert parallelism (EP) — specific to MoE models. Different GPUs hold different expert groups. When a token routes to an expert, the activation is sent to the GPU holding that expert. Useful for very large MoE models where TP alone wastes memory on inactive experts.

Strategy	Best for	Latency	Throughput	Hardware requirement
Tensor parallelism (TP)	Model too big for 1 GPU, single node	Low	High	Fast interconnect (NVLink, PCIe)
Pipeline parallelism (PP)	Multi-node, very large models	Higher (bubbles)	Medium	Network (10GbE+)
Data parallelism (DP)	Model fits, need more users	Same as single GPU	Scales linearly	Any — just more GPUs
Expert parallelism (EP)	Large MoE models	Medium	High	Fast interconnect

Configuration examples

vLLM with tensor parallelism

vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --tensor-parallel-size 2 \
  --gpu-memory-utilization 0.90

For 2× DGX Spark connected via 200Gb/s QSFP:

vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --tensor-parallel-size 2 \
  --distributed-executor-backend ray

vLLM with pipeline parallelism

vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --pipeline-parallel-size 2 \
  --tensor-parallel-size 1

Hybrid TP + PP (large model, multi-node)

For a 405B model across 2× DGX Spark (TP within each node, PP across nodes):

vllm serve meta-llama/Llama-3.1-405B-Instruct \
  --tensor-parallel-size 2 \
  --pipeline-parallel-size 2 \
  --distributed-executor-backend ray

SGLang with tensor parallelism

SGLang is competitive with vLLM on dual-GPU setups for specific workloads:

python -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-70B-Instruct \
  --tp 2 \
  --port 30000

NCCL environment variables

For multi-GPU communication, set these before launching:

export NCCL_DEBUG=INFO
export NCCL_IB_DISABLE=0          # Enable InfiniBand if available
export NCCL_SOCKET_IFNAME=eth0    # Pin the network interface
export NCCL_NET_GDR_LEVEL=2       # Enable GPUDirect RDMA

On DGX Spark clusters connected via 200Gb/s QSFP, NCCL automatically selects the optimal transport. You typically do not need to override NCCL_IB_DISABLE or NCCL_SOCKET_IFNAME unless you are debugging connectivity issues.

Performance comparison

Based on community benchmarks for 70B-class dense models:

Configuration	Decode tok/s (single user)	Concurrent users (5 tok/s each)	Notes
1× DGX Spark (no parallelism)	~10	2–3	Baseline
2× DGX Spark, TP=2 (200Gb/s QSFP)	~18	5–8	Near-linear scaling for decode
2× RTX 4090, TP=2 (PCIe)	~20	5–7	PCIe bandwidth limits prefill
2× DGX Spark, PP=2 (network)	~12	4–6	Pipeline bubbles reduce utilization

vLLM delivers 5.9× more tokens/s than Ollama at 50 concurrent users on the same hardware. If you serve models to a team, the inference engine matters as much as the parallelism strategy.

When not to use parallelism

Not every setup benefits from splitting across GPUs:

Model fits on 1 GPU with room to spare — parallelism adds overhead. Run on 1 GPU and use data parallelism for throughput.
Small models (7B–13B) on consumer GPUs — TP synchronization overhead can make 2× GPU slower than 1× GPU for small models. SGLang shows measurable degradation here.
Single-user inference — unless the model literally does not fit, 1 GPU is simpler and often faster.
Network slower than 10GbE between nodes — pipeline parallelism starves on slow links. Upgrade networking first.

Troubleshooting common issues

OOM on multi-GPU with vLLM

vLLM reserves memory for CUDA graphs and KV cache aggressively. On 2× RTX 3090 (48GB total), a 32B model may OOM even though Ollama handles it fine. Reduce --gpu-memory-utilization from the default 0.90:

vllm serve your-model \
  --tensor-parallel-size 2 \
  --gpu-memory-utilization 0.80

NCCL connection failures

If GPUs cannot communicate, check:

nvidia-smi topo -m — verify NVLink or PCIe topology
NCCL_DEBUG=INFO — confirm which transport NCCL selects
Firewall rules on port 29500 (default NCCL rendezvous port)

Mixed GPU types

vLLM supports heterogeneous GPU setups, but the slowest GPU determines throughput. For best results, use matched GPUs. If mixing, assign more layers to the faster GPU via pipeline parallelism with uneven stage allocation.

Expose multi-GPU deployments via Tokios

Whether you run TP, PP, or both, your vLLM or SGLang server exposes an OpenAI-compatible /v1 API on localhost. Run the Tokios connector on the same machine and your deployment becomes reachable from any device, agent, or teammate — without opening inbound ports.

# Start vLLM with tensor parallelism
vllm serve your-model --tensor-parallel-size 2

# Pair the Tokios connector
curl -fsSL https://tokios.com/downloads/install/linux | sh -s -- --code TKS-XXXX-XXXX

See the vLLM remote access guide for the full walkthrough.

Next steps

Model serving and scaling

Understand how inference engines manage memory, batching, and KV cache.

DGX Spark use case

Set up Tokios on a DGX Spark or a 2-Spark cluster.

vLLM remote access

Expose your vLLM server securely from anywhere.

Quantization guide

Shrink models to fit on fewer GPUs before reaching for parallelism.

Hardware & fit

Serve & connect your runtime

Connect a client or app

Compare

Tensor parallelism vs pipeline parallelism for local LLM inference

Quick verdict

Tensor parallelism explained

Pipeline parallelism explained

Decision matrix

Hardware-specific recommendations

Data parallelism and expert parallelism

Configuration examples

vLLM with tensor parallelism

vLLM with pipeline parallelism

Hybrid TP + PP (large model, multi-node)

SGLang with tensor parallelism

NCCL environment variables

Performance comparison

When not to use parallelism

Troubleshooting common issues

OOM on multi-GPU with vLLM

NCCL connection failures

Mixed GPU types

Expose multi-GPU deployments via Tokios

Next steps

Model serving and scaling

DGX Spark use case

vLLM remote access

Quantization guide

​Quick verdict

​Tensor parallelism explained

​Pipeline parallelism explained

​Decision matrix

​Hardware-specific recommendations

​Data parallelism and expert parallelism

​Configuration examples

​vLLM with tensor parallelism

​vLLM with pipeline parallelism

​Hybrid TP + PP (large model, multi-node)

​SGLang with tensor parallelism

​NCCL environment variables

​Performance comparison

​When not to use parallelism

​Troubleshooting common issues

​OOM on multi-GPU with vLLM

​NCCL connection failures

​Mixed GPU types

​Expose multi-GPU deployments via Tokios

​Next steps

Model serving and scaling

DGX Spark use case

vLLM remote access

Quantization guide

Quick verdict

Tensor parallelism explained

Pipeline parallelism explained

Decision matrix

Hardware-specific recommendations

Data parallelism and expert parallelism

Configuration examples

vLLM with tensor parallelism

vLLM with pipeline parallelism

Hybrid TP + PP (large model, multi-node)

SGLang with tensor parallelism

NCCL environment variables

Performance comparison

When not to use parallelism

Troubleshooting common issues

OOM on multi-GPU with vLLM

NCCL connection failures

Mixed GPU types

Expose multi-GPU deployments via Tokios

Next steps