Run Llama locally and use it from anywhere

Meta’s Llama 3.1 and Llama 3.3 are open-weight, text-only chat models with 128K context and tool-use support — and they’re still the Llama models most people run on a single GPU. Llama 4 (Scout and Maverick) is Meta’s current generation: natively multimodal mixture-of-experts models with very long context, but large enough that they need multi-GPU or high-unified-memory hardware. This page leads with the single-GPU-friendly 3.1/3.3 tags, then covers Llama 4 below. Run the size that fits your hardware locally, then pair the Tokios connector so clients can call it through https://api.tokios.com instead of your machine’s localhost.

Run it locally with Ollama

Start with the 8B Llama 3.1 tag if you are new to local models:

ollama run llama3.1:8b

For larger machines, use one of the 70B tags:

ollama run llama3.3:70b

Only use llama3.1:405b if you have a multi-GPU or very large unified-memory setup. Leave Ollama on its default http://127.0.0.1:11434.

Variants and tags

Tags and quantizations change over time. Confirm the exact tag on Ollama’s Llama 3.1 page or Ollama’s Llama 3.3 page before you standardize on it.

Variant	Ollama tag	Hugging Face repo	Parameters	Download size	Context	Input	License
Llama 3.1 8B Instruct	`llama3.1:8b`	`meta-llama/Llama-3.1-8B-Instruct`	8B dense	4.9 GB	128K	Text	Llama 3.1 Community License
Llama 3.1 70B Instruct	`llama3.1:70b`	`meta-llama/Llama-3.1-70B-Instruct`	70B dense	43 GB	128K	Text	Llama 3.1 Community License
Llama 3.1 405B Instruct	`llama3.1:405b`	`meta-llama/Llama-3.1-405B-Instruct`	405B dense	243 GB	128K	Text	Llama 3.1 Community License
Llama 3.3 70B Instruct	`llama3.3:70b`	`meta-llama/Llama-3.3-70B-Instruct`	70B dense	43 GB	128K	Text	Llama 3.3 Community License

Sources: Ollama Llama 3.1 tags, Ollama Llama 3.3 tags, meta-llama/Llama-3.1-8B-Instruct, meta-llama/Llama-3.3-70B-Instruct, Llama 3.1 license, and Llama 3.3 license.

The Llama 3.1 and 3.3 Community Licenses are custom licenses, not Apache or MIT. They include an acceptable-use policy, attribution requirements when you redistribute Llama materials, and a separate Meta license requirement for products or services above 700 million monthly active users.

Quantization and formats

Ollama publishes Llama 3.1 and 3.3 as GGUF builds, with common quantizations such as Q4_K_M, Q5_K_M, Q8_0, and fp16 available across the tag lists. The default 8b, 70b, and 405b tags use practical local downloads; the fp16 tags are much larger.

Format	Where to use it	Notes
`GGUF`	Ollama, LM Studio, llama.cpp	Best beginner path and the broadest hardware support.
`fp16` / `bf16`	High-memory research or server use	Use when you need less quantization loss and have the memory.
`AWQ` / `GPTQ`	vLLM or another CUDA-focused runtime if you choose a compatible repo	Verify the exact quant repo before you serve it.
`MLX`	Apple Silicon if a compatible conversion is available	Use Mac-native tooling or LM Studio when offered.

See Quantization formats and the hardware they need for the format matrix, and what quantization is for the Q4_K_M naming basics.

Hardware fit

Comfort at roughly the Ollama default Q4_K_M class and a modest context length. The real limit is the actual model file plus KV cache for your context.

Target	8B	70B	405B
Laptop CPU or 8 GB GPU	Comfortable at short context	Won’t fit	Won’t fit
16 GB GPU	Comfortable	Won’t fit	Won’t fit
RTX 3090 / 4090, 24 GB	Comfortable	Won’t fit	Won’t fit
RTX 5090, 32 GB	Comfortable	Tight only with smaller 70B quants, not the default 43 GB tag	Won’t fit
RTX 6000 Ada / A6000, 48 GB	Comfortable	Tight; keep context modest	Won’t fit
Apple Silicon or Strix Halo, 96-128 GB unified	Comfortable	Comfortable by capacity	Won’t fit
Multi-GPU workstation or server	Comfortable	Comfortable	Requires multi-GPU or very large unified memory

If you are choosing between 70B and 405B, remember that both are dense models. You need to hold all weights in memory. See What size model fits your GPU? for the sizing method and memory and offloading for what happens when you spill to system memory.

Run it: beginner to advanced

Ollama (easiest)
LM Studio (GUI)
Unsloth Studio (no-code)
vLLM (advanced)
SGLang (advanced)

ollama run llama3.1:8b

For stronger quality on large-memory hardware:

ollama run llama3.3:70b

Ollama’s library pages mark these tags as tools capable. Raise the runtime context setting if you need more than the default context.

Search for Llama 3.1 or Llama 3.3 in LM Studio, download a GGUF quant that fits your machine, and start the local server from the Developer tab. LM Studio’s OpenAI-compatible API normally uses:

http://127.0.0.1:1234/v1

Use that URL as the Tokios upstream if LM Studio is the runtime.

Use Unsloth Studio when you want a GUI workflow around open models, LoRA training, or export. If your Studio build exposes an OpenAI-compatible inference endpoint, point Tokios at that /v1 URL. Otherwise export the model to GGUF and serve it through Ollama, LM Studio, or llama.cpp.

Hugging Face lists direct vLLM serve commands for the Llama instruct repos:

vllm serve meta-llama/Llama-3.1-8B-Instruct

For Llama 3.3:

vllm serve meta-llama/Llama-3.3-70B-Instruct

If your client uses OpenAI-style tool calls, enable vLLM’s Llama tool parser:

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --enable-auto-tool-choice \
  --tool-call-parser llama3_json

vLLM exposes an OpenAI-compatible API on http://127.0.0.1:8000/v1 by default.

Launch an OpenAI-compatible SGLang server with the Hugging Face model path:

python -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --host 127.0.0.1 \
  --port 8000

For Llama 3.3, use --model-path meta-llama/Llama-3.3-70B-Instruct. Point Tokios at http://127.0.0.1:8000/v1.

Capabilities

Tool use: Llama 3.1 and Llama 3.3 support tool use through chat templates. vLLM documents the llama3_json parser for Llama 3.1 tool calling; verify parser support for your exact Llama 3.3 serving path before you depend on it.
Context: Llama 3.1 8B, 70B, and 405B use 128K context. Llama 3.3 70B also uses 128K context.
Languages: Meta lists English, German, French, Italian, Portuguese, Hindi, Spanish, and Thai as supported languages on both model cards.
Vision: The Llama 3.1 and 3.3 models are text-only. For image input, use a dedicated vision model — or Llama 4, which is natively multimodal (see Beyond 3.3: Llama 4 below).
Reasoning: Llama 3.1 and 3.3 are general instruction-tuned models. They are not dedicated reasoning models with a separate thinking mode.

Tokios passes tool-call fields through to your upstream runtime. It does not add tool calling to a runtime or quantization that cannot parse the model’s tool format.

Beyond 3.3: Llama 4

Llama 4 is Meta’s current generation (launched April 2025, still current as of mid-2026 — there is no Llama 5 yet). Unlike the text-only 3.x models above, the Llama 4 models are natively multimodal (text + image) mixture-of-experts models. Both are open-weight, but they’re big — multi-GPU or high-unified-memory territory, not a single consumer card.

Model	Ollama tag	Parameters	Experts	Context	Modality	Download (`Q4`-class)
Llama 4 Scout	`llama4:scout` (alias `llama4:16x17b`)	109B total / 17B active	16	10M	Text + image	~67 GB
Llama 4 Maverick	`llama4:maverick` (alias `llama4:128x17b`)	400B total / 17B active	128	1M	Text + image	~245 GB

Sources: ollama.com/library/llama4, Meta Llama 4 announcement.

Llama 4 uses Meta’s custom Llama 4 Community License — like the 3.x licenses, it is not Apache or MIT, and it carries an acceptable-use policy plus a separate Meta license requirement for products above 700 million monthly active users. A third model, Llama 4 Behemoth (~2T parameters), was previewed as a teacher model but has not shipped publicly.

Scout’s ~67 GB and Maverick’s ~245 GB downloads put them well past a single 24–48 GB GPU — realistically multi-GPU servers or large unified-memory machines. If you want a Llama model that runs on one consumer GPU today, use the llama3.1:8b or llama3.3:70b tags above. When you do run Llama 4 on capable hardware, connecting it through Tokios is identical to the steps below — only the deployment’s upstream model id changes.

Connect it with Tokios

Keep your local runtime running

Start Llama 3.1 or Llama 3.3 in Ollama, LM Studio, vLLM, SGLang, or another OpenAI-compatible runtime. Keep the server bound to loopback.

Pair the Tokios connector

Sign in at tokios.com/console, open the Setup tab, and click Start pairing. Tokios shows a one-time claim code like TKS-XXXX-XXXX.Run the installer on the machine that runs the model:

Windows

irm https://tokios.com/downloads/install/windows | iex; tokios-install -Code 'TKS-XXXX-XXXX'

macOS

curl -fsSL https://tokios.com/downloads/install/macos | sh -s -- --code TKS-XXXX-XXXX

Linux

curl -fsSL https://tokios.com/downloads/install/linux | sh -s -- --code TKS-XXXX-XXXX

Back in the dashboard, approve the device on the Setup or Connectors tab. The connector token is delivered over the tunnel after approval.

Point the connector at the runtime's /v1 API

For Ollama, use:

{
  "Upstream": {
    "BaseUrl": "http://127.0.0.1:11434/v1"
  }
}

For LM Studio, use http://127.0.0.1:1234/v1. For vLLM or SGLang, use the port you launched, such as http://127.0.0.1:8000/v1.

In the Models tab, register the upstream model and choose a public name, for example llama3-tunnel. API clients send this deployment name, not llama3.1:8b or a Hugging Face repo id, in the model field.

Create an API key

In the Keys tab, click Create key. Copy the sk-tok-... API key when it appears.

Use it from anywhere

curl https://api.tokios.com/v1/chat/completions \
  -H "Authorization: Bearer sk-tok-YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3-tunnel",
    "stream": true,
    "messages": [{"role": "user", "content": "Draft a short checklist for testing a local model deployment."}]
  }'

export ANTHROPIC_BASE_URL=https://api.tokios.com
export ANTHROPIC_API_KEY=sk-tok-YOUR_KEY
claude --model llama3-tunnel

The model field is always your registered deployment name. Tokios forwards to your local runtime and does not change the upstream model id, quantization, or tool-calling behavior.

Caveats

Ollama’s default context may be lower than the model’s 128K maximum. Raise the runtime context setting for long-document work.
Tool calling needs both a tool-capable model and a runtime/client path that sends and parses tool calls correctly.
llama3.1:405b is not a single-consumer-GPU model. Use 8b or 70b unless you have multi-GPU or very large unified memory.
Check the Llama Community License before commercial redistribution or public product use.

Quickstart

Pair a connector and register your first deployment end to end.

GPU-to-model fit

Match Llama sizes to your GPU or unified memory.

Choose a model by task

Compare Llama 3 against other local models for chat, coding, and agent workloads.

Quantization and hardware

Check which formats run on which hardware.

Get Started

Models

Concepts

Reference

Run Llama locally and use it from anywhere

Run it locally with Ollama

Variants and tags

Quantization and formats

Hardware fit

Run it: beginner to advanced

Capabilities

Beyond 3.3: Llama 4

Connect it with Tokios

Use it from anywhere

Caveats

Quickstart

GPU-to-model fit

Choose a model by task

Quantization and hardware

​Run it locally with Ollama

​Variants and tags

​Quantization and formats

​Hardware fit

​Run it: beginner to advanced

​Capabilities

​Beyond 3.3: Llama 4

​Connect it with Tokios

​Use it from anywhere

​Caveats

Quickstart

GPU-to-model fit

Choose a model by task

Quantization and hardware

Run it locally with Ollama

Variants and tags

Quantization and formats

Hardware fit

Run it: beginner to advanced

Capabilities

Beyond 3.3: Llama 4

Connect it with Tokios

Use it from anywhere

Caveats