Run Kimi K3 locally: hardware and self-hosting reality

​

What is available?

​

Kimi K3 at a glance

​

Can you run Kimi K3 locally?

​

Kimi K3 hardware requirements

​

Serving K3 locally

vllm serve "moonshotai/Kimi-K3"

from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "moonshotai/Kimi-K3",
    trust_remote_code=True
)

​

Why the architecture matters for self-hosting

• Kimi Delta Attention (KDA) — a hybrid linear-attention design that makes million-token prompts cheaper to prefill. It needs matching kernels and cache semantics: Moonshot says conventional prefix caching does not map cleanly to KDA and that its vLLM prefill-cache contribution ships with the weights. A runtime that merely loads the weights is not automatically K3-compatible.
• Attention Residuals (AttnRes) — layers selectively retrieve useful representations from earlier depth instead of adding every layer’s output with fixed weight. Moonshot’s AttnRes paper reports more stable information flow in deeper models.
• Stable LatentMoE — 16 active + 2 shared experts route per token across 896 total. Sparse activation cuts per-token compute, but it does not shrink the stored checkpoint: every expert must stay resident across the serving cluster, which makes fast interconnects a first-order requirement.
• MoonViT-V2 — a 401M-parameter vision encoder that processes text, image, and video input natively.
• Native MXFP4 — quantization-aware training ran from the supervised fine-tuning stage onward, targeting MXFP4 weights and MXFP8 activations. This is the intended serving format, not a community quant — so the 4-bit figure above is a floor, not a compromise.

​

K3 API behavior to preserve

• K3 always reasons. Use the top-level reasoning_effort: "max"; do not send K2.x’s thinking parameter.
• Streaming responses separate reasoning into reasoning_content and the final answer into content.
• For multi-turn conversations and tool loops, replay the complete assistant message, including preserved reasoning and tool-call fields. Returning only content can destabilize later turns.
• max_completion_tokens defaults to 131,072 and can be set as high as 1,048,576.
• The launch API fixes sampling values. Moonshot recommends that you omit temperature, top_p, n, and penalty parameters rather than override them.
• Public image URLs are not accepted. Send base64 data or an ms://<file-id> reference.

​

Connect K3 with Tokios

irm https://tokios.com/downloads/install/windows | iex; tokios-install -Code 'TKS-XXXX-XXXX'

curl -fsSL https://tokios.com/downloads/install/macos | sh -s -- --code TKS-XXXX-XXXX

curl -fsSL https://tokios.com/downloads/install/linux | sh -s -- --code TKS-XXXX-XXXX

​

Call it from anywhere

curl https://api.tokios.com/v1/chat/completions \
  -H "Authorization: Bearer sk-tok-YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "kimi-k3-tunnel",
    "stream": true,
    "messages": [{"role": "user", "content": "Audit this repository and propose a phased refactor."}]
  }'

export ANTHROPIC_BASE_URL=https://api.tokios.com
export ANTHROPIC_API_KEY=sk-tok-YOUR_KEY
claude --model kimi-k3-tunnel

​

Launch caveats

• K3 benchmark results are Moonshot-reported. Independent reproduction requires the evaluation setup.
• Moonshot warns that losing preserved reasoning history, or switching another model’s existing session to K3, can make output unstable.
• K3 is trained for long, difficult tasks and can act too proactively when instructions are ambiguous. Put explicit boundaries in your system prompt or AGENTS.md.
• A 1M-token limit is a maximum, not a target. Long contexts increase prefill time, cache storage, and inference cost.
• The Kimi K3 License is custom — read the model card for terms before deploying in production.

​

FAQ