> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tokios.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Tokios exposes private local models through https://api.tokios.com using an outbound-only connector tunnel.
> Use 'connector' for the local tokios-connector process.
> Use 'deployment' for the public model name in the API model field.
> Use 'upstream model id' for the model served by Ollama, llama.cpp, vLLM, LM Studio, or another local backend.
> API keys use the prefix sk-tok-. Connector tokens use the prefix ct-tok-.

# RTX 50 series consumer guide: 5090, 5080, 5070 for local LLM

> Complete guide to NVIDIA RTX 5090, 5080, 5070 Ti, 5070, 5060 Ti, and 5060 for local LLM inference. FP4 native support, model compatibility, and comparison to RTX 40 series.

NVIDIA's RTX 50 series (Blackwell architecture) brings native FP4 tensor cores and up to 32 GB GDDR7 to consumer GPUs. The RTX 5090 is the "best consumer GPU for LLMs in 2026" — but the entire lineup has something to offer at different price points. This guide breaks down which RTX 50 card fits your workload and budget.

> [!NOTE]
>
> Last verified: 2026-10-21. Prices and availability vary by region. Confirm with retailers before purchasing.

## RTX 50 series lineup

| Model | Memory | Bandwidth | TDP | Price (MSRP) | Best for |
| - | - | - | - | - | - |
| **RTX 5090** | 32 GB GDDR7 | 1,792 GB/s | 575W | \~\$1,999 | 70B Q4, 32B dense FP16 |
| **RTX 5080** | 16 GB GDDR7 | 960 GB/s | 360W | \~\$1,000 | 30B Q4, 14B Q8 |
| **RTX 5070 Ti** | 16 GB GDDR7 | 896 GB/s | 300W | \~\$750 | 30B Q4 (slower than 5080) |
| **RTX 5070** | 12 GB GDDR7 | 672 GB/s | 250W | \~\$550 | 14B Q4 |
| **RTX 5060 Ti** | 16 GB GDDR7 | 448 GB/s | 200W | \~\$450 | 30B Q4 (bandwidth-limited) |
| **RTX 5060** | 8 GB GDDR7 | 448 GB/s | 150W | \~\$350 | 7B Q4 |

> [!TIP]
>
> **All RTX 50 series cards have native FP4 tensor cores.** This is a Blackwell architecture feature that accelerates 4-bit quantization (NVFP4, MXFP4). For models that support FP4 inference, RTX 50 series delivers 20-40% better performance than RTX 40 series at the same VRAM tier.

## RTX 5090: The flagship

**Specs:**

* 32 GB GDDR7
* 1,792 GB/s bandwidth (highest consumer GPU)
* 575W TDP
* \~\$1,999 MSRP

**Performance:**

* 7B models: \~162 tok/s
* 14B models: \~90 tok/s
* 32B dense models: \~45 tok/s
* 70B models at Q4: \~23 tok/s
* \~39% faster than RTX 4090 on 8B models (145 vs 104 tok/s)

**What fits:**

* Qwen3 70B at Q4 (\~40 GB, tight fit with KV cache)
* Qwen3.5-122B-A10B at Q4 (\~70 GB, does NOT fit — need 2x 5090)
* Qwen3.6-35B-A3B at Q6\_K (\~28 GB, comfortable fit)
* Llama 3.1 32B at FP16 (\~64 GB, does NOT fit — use Q4 instead)
* Multiple 14B models simultaneously

**When to choose RTX 5090:**

* You need maximum single-GPU performance
* You're running 70B models at Q4 (32 GB is the minimum)
* You want future-proof FP4 tensor core support
* Budget is \~\$2,000

**When to avoid:**

* You only need 16 GB (get RTX 5080 instead, save \$1,000)
* You need 70B+ models at high precision (get 2x RTX 5090 or RTX PRO 6000)
* You're on a tight budget (get used RTX 3090 at \$700-900, same 24 GB VRAM)

> [!WARNING]
>
> **RTX 5090 draws 575W.** Ensure your PSU can handle it (850W+ recommended) and your case has adequate cooling. This is not a low-power card.

## RTX 5080: The mid-tier sweet spot

**Specs:**

* 16 GB GDDR7
* 960 GB/s bandwidth
* 360W TDP
* \~\$1,000 MSRP

**Performance:**

* 7B models: \~100 tok/s
* 14B models at Q8: \~70 tok/s
* 30B models at Q4: \~50 tok/s
* Gemma 4 E4B at Q8: \~141 tok/s (community benchmark)

**What fits:**

* Qwen3.6-35B-A3B at Q4 (\~20 GB, does NOT fit — need 24 GB)
* Qwen3 14B at Q8 (\~14 GB, comfortable fit)
* Gemma 4 E4B at Q8 (\~14 GB, comfortable fit)
* Llama 3.1 8B at FP16 (\~16 GB, tight fit)
* Multiple 7B models simultaneously

**When to choose RTX 5080:**

* You want 16 GB VRAM at a reasonable price
* You're running 14B-30B models at Q4
* You want FP4 tensor core support without paying 5090 prices
* Budget is \~\$1,000

**When to avoid:**

* You need 24+ GB VRAM (get RTX 5090 or used RTX 3090)
* You only need 12 GB (get RTX 5070, save \$450)
* You're on a tight budget (get used RTX 3060 12GB at \$200-250)

## RTX 5070 Ti / 5070: Entry-level Blackwell

### RTX 5070 Ti (16 GB, \~\$750)

**Specs:**

* 16 GB GDDR7
* 896 GB/s bandwidth (7% slower than 5080)
* 300W TDP
* \~\$750 MSRP

**Performance:**

* 7B models: \~90 tok/s
* 14B models at Q8: \~65 tok/s
* 30B models at Q4: \~45 tok/s

**When to choose RTX 5070 Ti:**

* You want 16 GB VRAM but can't afford RTX 5080
* You're okay with 7% lower bandwidth
* Budget is \~\$750

**When to avoid:**

* You can stretch to RTX 5080 (only \$250 more for 7% more bandwidth)
* You only need 12 GB (get RTX 5070, save \$200)

### RTX 5070 (12 GB, \~\$550)

**Specs:**

* 12 GB GDDR7
* 672 GB/s bandwidth
* 250W TDP
* \~\$550 MSRP

**Performance:**

* 7B models: \~75 tok/s
* 14B models at Q4: \~50 tok/s
* 30B models at Q4: does NOT fit (needs 16+ GB)

**What fits:**

* Qwen3 14B at Q4 (\~8 GB, comfortable fit)
* Llama 3.1 8B at Q8 (\~8 GB, comfortable fit)
* Gemma 3 12B at Q4 (\~7 GB, comfortable fit)
* Multiple 7B models simultaneously

**When to choose RTX 5070:**

* You want 12 GB VRAM at a good price
* You're running 7B-14B models at Q4
* Budget is \~\$550

**When to avoid:**

* You need 16+ GB VRAM (get RTX 5070 Ti or 5080)
* You only need 8 GB (get RTX 5060, save \$200)

## RTX 5060 Ti / 5060: Budget Blackwell

### RTX 5060 Ti (16 GB, \~\$450)

**Specs:**

* 16 GB GDDR7
* 448 GB/s bandwidth (53% slower than 5080)
* 200W TDP
* \~\$450 MSRP

**Performance:**

* 7B models: \~70 tok/s
* 14B models at Q4: \~45 tok/s
* 30B models at Q4: \~30 tok/s (bandwidth-limited)

**What fits:**

* Same as RTX 5080 (16 GB VRAM)
* But 53% slower due to lower bandwidth

**When to choose RTX 5060 Ti:**

* You need 16 GB VRAM on a tight budget
* You're okay with much slower inference (30% slower than RTX 4060 Ti 16GB)
* Budget is \~\$450

**When to avoid:**

* You can stretch to RTX 5070 (only \$100 more for 80% more bandwidth)
* You want fast inference (get RTX 5080 or used RTX 3090)

> [!TIP]
>
> **RTX 5060 Ti is the worst value in the RTX 50 lineup.** The 448 GB/s bandwidth is a severe bottleneck for LLM inference. Only choose this if you absolutely need 16 GB VRAM and can't afford RTX 5070.

### RTX 5060 (8 GB, \~\$350)

**Specs:**

* 8 GB GDDR7
* 448 GB/s bandwidth
* 150W TDP
* \~\$350 MSRP

**Performance:**

* 7B models: \~60 tok/s
* 14B models at Q4: does NOT fit (needs 8+ GB)

**What fits:**

* Qwen3 8B at Q4 (\~5 GB, comfortable fit)
* Llama 3.1 8B at Q4 (\~5 GB, comfortable fit)
* Phi-4-mini (3.8B) at Q8 (\~4 GB, comfortable fit)

**When to choose RTX 5060:**

* You're on a very tight budget
* You only need 7B models
* Budget is \~\$350

**When to avoid:**

* You can stretch to RTX 5070 (only \$200 more for 50% more VRAM)
* You need 14B+ models (get RTX 5070 or used RTX 3060 12GB)

## Comparison to RTX 40 series

Is the RTX 50 series worth the upgrade over RTX 40 series?

### RTX 5090 vs RTX 4090

| Feature | RTX 5090 | RTX 4090 | Difference |
| - | - | - | - |
| Memory | 32 GB GDDR7 | 24 GB GDDR6X | +8 GB (+33%) |
| Bandwidth | 1,792 GB/s | 1,008 GB/s | +78% |
| FP4 support | Yes (native) | No | New feature |
| TDP | 575W | 450W | +125W |
| Price | \~\$1,999 | \~\$1,600-2,000 (used) | Similar |
| 7B tok/s | \~162 | \~120 | +35% |
| 70B Q4 | Fits | Does NOT fit | Major advantage |

**Verdict:** RTX 5090 is 35% faster and fits 70B models that RTX 4090 cannot. Worth the upgrade if you need 32 GB VRAM or maximum performance.

### RTX 5080 vs RTX 4080 Super

| Feature | RTX 5080 | RTX 4080 Super | Difference |
| - | - | - | - |
| Memory | 16 GB GDDR7 | 16 GB GDDR6X | Same |
| Bandwidth | 960 GB/s | 736 GB/s | +30% |
| FP4 support | Yes (native) | No | New feature |
| TDP | 360W | 320W | +40W |
| Price | \~\$1,000 | \~\$800-900 (used) | +\$100-200 |
| 7B tok/s | \~100 | \~85 | +18% |

**Verdict:** RTX 5080 is 18% faster with 30% more bandwidth. Worth the upgrade if you can find RTX 4080 Super at similar prices.

### RTX 5070 vs RTX 4070 Super

| Feature | RTX 5070 | RTX 4070 Super | Difference |
| - | - | - | - |
| Memory | 12 GB GDDR7 | 12 GB GDDR6X | Same |
| Bandwidth | 672 GB/s | 504 GB/s | +33% |
| FP4 support | Yes (native) | No | New feature |
| TDP | 250W | 220W | +30W |
| Price | \~\$550 | \~\$500-600 (used) | Similar |
| 7B tok/s | \~75 | \~65 | +15% |

**Verdict:** RTX 5070 is 15% faster with 33% more bandwidth. Marginal upgrade — wait for sales or buy used RTX 4070 Super.

### RTX 5060 vs RTX 4060

| Feature | RTX 5060 | RTX 4060 | Difference |
| - | - | - | - |
| Memory | 8 GB GDDR7 | 8 GB GDDR6 | Same |
| Bandwidth | 448 GB/s | 272 GB/s | +65% |
| FP4 support | Yes (native) | No | New feature |
| TDP | 150W | 115W | +35W |
| Price | \~\$350 | \~\$300 (used) | +\$50 |
| 7B tok/s | \~60 | \~50 | +20% |

**Verdict:** RTX 5060 is 20% faster with 65% more bandwidth. Worth the upgrade if you're buying new.

> [!TIP]
>
> **Upgrade recommendation:**
>
> * **RTX 4090 → RTX 5090**: Yes, if you need 32 GB VRAM or 35% faster inference
> * **RTX 4080 Super → RTX 5080**: Maybe, if prices are similar
> * **RTX 4070 Super → RTX 5070**: No, wait for sales
> * **RTX 4060 → RTX 5060**: Yes, 20% faster for only \$50 more

## FP4 native support

All RTX 50 series cards have native FP4 tensor cores, which accelerate 4-bit quantization formats (NVFP4, MXFP4).

### What is FP4?

FP4 (4-bit floating point) is a quantization format that reduces model size by 75% compared to FP16, with minimal quality loss. RTX 50 series has dedicated hardware acceleration for FP4 inference.

### FP4 vs Q4

| Feature | FP4 (NVFP4/MXFP4) | Q4\_K\_M (GGUF) |
| - | - | - |
| Quantization | 4-bit floating point | 4-bit integer |
| Quality loss | Low (1-2%) | Low-medium (3-5%) |
| Hardware acceleration | Yes (RTX 50 series) | No (software) |
| Speed | 20-40% faster | Baseline |
| Model support | Limited (vLLM, SGLang) | Broad (llama.cpp, Ollama) |

### Which models support FP4?

**Supported:**

* DeepSeek V4.1 Flash (NVFP4 variant)
* Qwen3 series (NVFP4 quantization available)
* Most models with NVIDIA-optimized quantization

**Not supported:**

* GGUF quantization (Q4\_K\_M, Q5\_K\_M, etc.)
* AWQ quantization
* Most community GGUF builds

> [!TIP]
>
> **FP4 is not yet widely adopted.** Most models are distributed in GGUF format with Q4\_K\_M quantization. FP4 support is growing, but Q4\_K\_M remains the most compatible format. Use FP4 if your model supports it and you're running vLLM or SGLang on RTX 50 series.

## Software compatibility

All RTX 50 series cards work with all major inference backends:

| Backend | Support | Notes |
| - | - | - |
| **Ollama** | Yes | Simplest setup, single-user |
| **vLLM** | Yes | Production-grade, concurrent batching, FP4 support |
| **SGLang** | Yes | High-throughput, RadixAttention, FP4 support |
| **llama.cpp** | Yes (CUDA) | CPU+GPU offloading, GGUF quantization |
| **LM Studio** | Yes | Desktop GUI, easy model management |
| **TensorFold** | Yes | EXL3 quantization, highest throughput |

> [!TIP]
>
> **For FP4 acceleration, use vLLM or SGLang.** These backends have native FP4 support and can leverage RTX 50 series tensor cores. llama.cpp and Ollama use GGUF quantization, which does not benefit from FP4 hardware.

## Power and cooling considerations

RTX 50 series cards draw significant power. Plan your build accordingly.

| Model | TDP | Recommended PSU | Cooling requirements |
| - | - | - | - |
| RTX 5090 | 575W | 850W+ | Full tower, 3+ fans |
| RTX 5080 | 360W | 750W+ | Mid tower, 2+ fans |
| RTX 5070 Ti | 300W | 650W+ | Mid tower, 2+ fans |
| RTX 5070 | 250W | 600W+ | Mid tower, 1-2 fans |
| RTX 5060 Ti | 200W | 550W+ | Mid tower, 1 fan |
| RTX 5060 | 150W | 500W+ | Mini tower OK |

> [!WARNING]
>
> **RTX 5090 requires a 850W+ PSU and full tower case.** The card is 3-4 slots wide and generates significant heat. Do not attempt to run it in a small form factor case or with an undersized PSU.

## Price-performance analysis

Which RTX 50 card offers the best value for local LLM inference?

### Cost per GB of VRAM

| Model | VRAM | Price | \$/GB | Notes |
| - | - | - | - | - |
| RTX 5060 | 8 GB | \~\$350 | \$44/GB | Entry-level |
| RTX 5060 Ti | 16 GB | \~\$450 | \$28/GB | Worst value (bandwidth-limited) |
| RTX 5070 | 12 GB | \~\$550 | \$46/GB | Good for 14B models |
| RTX 5070 Ti | 16 GB | \~\$750 | \$47/GB | Same \$/GB as 5070 |
| RTX 5080 | 16 GB | \~\$1,000 | \$63/GB | Premium for bandwidth |
| RTX 5090 | 32 GB | \~\$1,999 | \$62/GB | Premium for capacity |

### Cost per tok/s (7B model)

| Model | tok/s (7B) | Price | \$/tok/s | Notes |
| - | - | - | - | - |
| RTX 5060 | \~60 | \~\$350 | \$5.84 | Budget |
| RTX 5060 Ti | \~70 | \~\$450 | \$6.43 | Bandwidth-limited |
| RTX 5070 | \~75 | \~\$550 | \$7.33 | Good value |
| RTX 5070 Ti | \~90 | \~\$750 | \$8.33 | Premium |
| RTX 5080 | \~100 | \~\$1,000 | \$10.00 | Premium |
| RTX 5090 | \~162 | \~\$1,999 | \$12.34 | Premium |

**Best value:** RTX 5070 at $7.33/tok/s. It offers 75 tok/s on 7B models for $550 — the sweet spot for price-performance.

**Best performance:** RTX 5090 at 162 tok/s, but costs \$12.34/tok/s. You pay a premium for maximum speed.

**Best VRAM value:** RTX 5060 Ti at \$28/GB, but it's bandwidth-limited and slow. Avoid unless you absolutely need 16 GB on a budget.

> [!TIP]
>
> **Recommendation by budget:**
>
> * \*\*$350-450**: RTX 5060 (8 GB) or used RTX 3060 12GB ($200-250)
> * **\$550-750**: RTX 5070 (12 GB) — best value
> * **\$1,000**: RTX 5080 (16 GB) — if you need 16 GB
> * **\$2,000**: RTX 5090 (32 GB) — if you need 70B models or maximum speed

## Multi-GPU configurations

RTX 50 series cards can be combined for more VRAM, but PCIe bandwidth is the bottleneck.

### 2x RTX 5090 (64 GB total)

**Best for:** 120B MoE models at Q4, 70B models at Q8

**Performance:**

* Qwen3.5-122B-A10B at Q4: \~40-50 tok/s (with tensor parallelism)
* GLM-4.5-Air at Q4: \~45 tok/s

**Requirements:**

* Workstation chassis with 2x full-length PCIe 5.0 x16 slots
* 2x 575W PCIe power connectors (1,150W total)
* Motherboard with PCIe bifurcation support (x16 → 2x x8)

**Cost:** \~$4,000 (2x $1,999)

### 2x RTX 5080 (32 GB total)

**Best for:** 70B models at Q4, multiple 30B models

**Performance:**

* Qwen3 70B at Q4: \~40-50 tok/s (with tensor parallelism)
* Multiple Qwen3.6-35B-A3B at Q4: \~50 tok/s each

**Requirements:**

* Workstation chassis with 2x full-length PCIe 5.0 x16 slots
* 2x 360W PCIe power connectors (720W total)

**Cost:** \~$2,000 (2x $1,000)

> [!WARNING]
>
> **PCIe bandwidth bottleneck:** 2x RTX 5090 over PCIe 5.0 x16 (128 GB/s per direction) is 10-20% slower than NVLink. For maximum multi-GPU performance, use RTX PRO 6000 with NVLink support.

## Next steps

> **[Hardware reference](/guides/hardware-reference.md)**
>
> Compare RTX 50 series against RTX 40/30 series, RTX PRO, and other platforms.

> **[RTX PRO Blackwell variants](/guides/rtx-pro-blackwell-variants.md)**
>
> Professional alternative: RTX PRO 4000/4500/5000/6000 with NVLink and ECC memory.

> **[Model fit](/guides/model-fit.md)**
>
> Calculate which model sizes and quantizations fit your RTX 50 series GPU.

> **[RTX workstation setup](/use-cases/rtx-workstation.md)**
>
> Set up your RTX 5090/5080/5070 as a private model endpoint with Tokios.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.