Use an MLX-LM Server from anywhere

By default, mlx_lm.server listens on http://127.0.0.1:8080/v1 — it isn’t reachable from another machine unless you bind it wider or forward the port. MLX-LM Server has no built-in authentication: anything that can reach port 8080 can call the API with no credential. Keep MLX-LM Server on localhost and run the Tokios connector next to it instead: the connector dials out to https://api.tokios.com, and your tools, teammates, or agents call one authenticated endpoint from anywhere.

Last verified: 2026-10-14. MLX-LM Server ships with the mlx-lm Python package from Apple’s ml-explore team. This guide covers the current release.

Apple Silicon native. MLX-LM Server is the reference inference server for Apple Silicon. It uses the MLX framework’s unified-memory architecture with zero-copy data paths — model weights live in the same memory pool as the CPU and GPU, so there is no separate VRAM to manage. On M5 chips, Neural Accelerators deliver up to 4x faster prompt processing compared to M4.

What is Apple Silicon native inference

MLX-LM Server is different from the other backends Tokios supports: it runs only on Apple Silicon Macs (M1 and later) and takes direct advantage of the unified-memory architecture that all Apple Silicon chips share.

Feature	MLX-LM Server	llama.cpp on Mac	vLLM
Hardware	Apple Silicon only	Cross-platform (CPU + GPU)	NVIDIA GPU (CUDA)
Memory model	Unified memory, zero-copy	Unified memory via Metal	Separate VRAM
Neural Accelerators (ANE)	Yes — automatic on M5	No	No
Installation	`pip install mlx-lm`	Build from source or prebuilt binary	`pip install vllm` + CUDA
Quantization	4-bit, 8-bit, BF16	GGUF (Q2_K through Q8_0)	AWQ, GPTQ, FP16, BF16
Prompt processing on M5	Up to 4x vs M4	Baseline	N/A

Because MLX uses unified memory, a Mac Studio with 192 GB of RAM can serve 70B models at full precision without the multi-GPU setup that other platforms require. There is no VRAM boundary — the model, the KV cache, and the working memory all share the same pool.

MLX-LM Server’s auth is off by default

mlx_lm.server starts an OpenAI-compatible HTTP server on port 8080. Run it and it accepts requests from anyone who can reach the port — no credential required:

mlx_lm.server --model mlx-community/Qwen-3.5-4B-8bit

MLX-LM Server does not ship an authentication layer. Its own documentation notes the server is not recommended for production use without a reverse proxy in front of it. That’s fine for local-only use, but it means exposing the port beyond loopback — via --host 0.0.0.0, port forwarding, or a tunnel — gives unauthenticated API access to anyone who can reach it.

This isn’t a knock on MLX-LM Server — it’s a lightweight, Apple-Silicon-native inference server, and leaving auth to an external proxy keeps the server simple. Tokios is purpose-built for a different problem: turning that local endpoint into an authenticated API you can call from anywhere, without opening an inbound port or hand-rolling TLS.

The outbound-only way

Install MLX-LM and start the server locally

MLX-LM Server requires an Apple Silicon Mac (M1 or later) with macOS 14.0 (Sonoma) or later. Install the package and start the server:

pip install mlx-lm
mlx_lm.server --model mlx-community/Qwen-3.5-4B-8bit

The server downloads the model from mlx-community on first use and listens on http://127.0.0.1:8080. Leave it on the default 127.0.0.1 — don’t pass --host 0.0.0.0.

Choosing a quantization. Models on mlx-community ship in 4-bit (default, smallest), 8-bit, and BF16 (full precision) variants. A 70B model at 4-bit needs roughly 35 GB of unified memory; at 8-bit roughly 70 GB; at BF16 roughly 140 GB. Pick the quantization that fits your Mac’s memory.

Pair the Tokios connector

Sign in at tokios.com/console, open the Setup tab, and click Start pairing. Tokios shows a one-time claim code like TKS-XXXX-XXXX, valid for about 15 minutes.On the Mac running MLX-LM Server, run the macOS install one-liner:

curl -fsSL https://tokios.com/downloads/install/macos | sh -s -- --code TKS-XXXX-XXXX

Already have tokios-connector installed? Pass the code directly instead:

tokios-connector TKS-XXXX-XXXX

Back in the console, approve the device on the Setup or Connectors tab. The connector’s token is delivered over the tunnel once approved — you never see it on screen or paste it into a file.

Point the connector at MLX-LM Server's /v1 API

MLX-LM Server already speaks the OpenAI Chat Completions surface Tokios expects. For the basic single-model path, the console’s Setup wizard uses http://127.0.0.1:8080/v1 as the upstream automatically. If you’re serving multiple models through a tokios-connector.json config, set the upstream BaseUrl explicitly:

{
  "Upstream": {
    "BaseUrl": "http://127.0.0.1:8080/v1"
  }
}

See Connector config for the full multi-model file format.

In the Models tab, register the MLX-LM model and choose a public name — for example, mlx-qwen-tunnel. That public name, not the HuggingFace model id, is what clients send in the model field.

Create an API key

In the Keys tab, click Create key. Optionally scope it to specific deployments with model-name patterns (comma-separated globs, * for all). Copy the sk-tok-... key now — it’s shown only once.

Use it from anywhere

Once you have a deployment name and an API key, any OpenAI- or Anthropic-compatible client can reach your MLX-LM model — from another machine, a teammate’s laptop, or a hosted agent.

curl https://api.tokios.com/v1/chat/completions \
  -H "Authorization: Bearer sk-tok-YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mlx-qwen-tunnel",
    "stream": true,
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

from openai import OpenAI

client = OpenAI(
    base_url="https://api.tokios.com/v1",
    api_key="sk-tok-YOUR_KEY",
)

response = client.chat.completions.create(
    model="mlx-qwen-tunnel",
    messages=[{"role": "user", "content": "Hello!"}],
    stream=True,
)

for chunk in response:
    print(chunk.choices[0].delta.content or "", end="")

export ANTHROPIC_BASE_URL=https://api.tokios.com
export ANTHROPIC_API_KEY=sk-tok-YOUR_KEY
claude --model mlx-qwen-tunnel

The model field is always your registered deployment name (mlx-qwen-tunnel above), never the HuggingFace model id. See Use Claude Code for persisting the environment variables across sessions.

Server configuration reference

MLX-LM Server exposes several flags for tuning inference. The most useful ones:

Flag	Default	Purpose
`--model`	(required)	HuggingFace model id or local path
`--host`	`127.0.0.1`	Bind address — keep on `127.0.0.1` for Tokios
`--port`	`8080`	Listen port
`--adapter-path`	—	Path to a LoRA adapter for serving fine-tuned models
`--max-tokens`	`512`	Default maximum tokens per generation
`--temp`	`0.0`	Sampling temperature
`--top-p`	`1.0`	Top-p (nucleus) sampling
`--top-k`	`0`	Top-k sampling (0 = disabled)
`--kv-bits`	—	KV cache quantization bits (reduces memory)
`--kv-group-size`	`64`	Group size for KV cache quantization
`--prefill-step-size`	`2048`	Prefill chunk size for long contexts
`--draft-model`	—	Model for speculative decoding
`--num-draft-tokens`	`3`	Draft tokens per speculative decoding step
`--prompt-cache-size`	`10`	Number of prompts to cache
`--decode-concurrency`	`32`	Maximum concurrent decode operations
`--prompt-concurrency`	`8`	Maximum concurrent prompt operations
`--trust-remote-code`	`false`	Trust remote code from HuggingFace repos

The server exposes these endpoints:

Endpoint	Purpose
`/v1/chat/completions`	Chat Completions (OpenAI-compatible)
`/v1/completions`	Text Completions (OpenAI-compatible)
`/v1/models`	List loaded models
`/health`	Health check

Speculative decoding can speed up generation by 2-3x for large models. Pass --draft-model with a smaller model of the same family — for example, use a 4B model as the draft for a 32B model. MLX-LM Server verifies draft tokens in parallel and only recomputes mismatches.

FAQ

What are the prerequisites for MLX-LM Server?

You need an Apple Silicon Mac (M1, M2, M3, M4, or M5) running macOS 14.0 (Sonoma) or later, Python 3.9 or later, and pip or conda. Install with pip install mlx-lm. The MLX framework uses Metal for GPU acceleration and unified memory for model weights — no separate GPU drivers or CUDA toolkit needed.

How do I install MLX-LM Server?

Install the Python package with pip install mlx-lm (or conda install -c conda-forge mlx-lm). Then start the server with mlx_lm.server --model <model-id>. The model is downloaded from HuggingFace on first use. Models from the mlx-community organization are pre-converted to MLX format and ready to use.

What port does MLX-LM Server use?

MLX-LM Server listens on port 8080 by default, bound to 127.0.0.1 (localhost only). Change the port with --port and the bind address with --host. For Tokios, keep the defaults — the connector reaches the server on http://127.0.0.1:8080/v1.

Does MLX-LM Server support streaming?

Yes. Set "stream": true in the request body and the server returns standard server-sent events (SSE). Through Tokios, streaming works end-to-end — set "stream": true and Tokios returns SSE in OpenAI or Anthropic format, depending on which API surface you called.

Does MLX-LM Server support tool calling?

Yes. MLX-LM Server supports structured tool calling. Use a model that supports chat templates with tool definitions — models from the mlx-community such as Qwen 3.5 and Llama 3 include the necessary chat templates. Pass tools in the OpenAI-compatible tools field of the request.

Can I use LoRA adapters with MLX-LM Server?

Yes. Pass --adapter-path pointing to a directory with LoRA adapter weights. Fine-tune adapters with mlx_lm.lora (included in the same package) and then serve the base model with the adapter overlaid. Register the adapter as its own Tokios deployment with its own public name.

How does MLX-LM Server compare to llama.cpp on Apple Silicon?

Both use unified memory on Apple Silicon. MLX-LM Server is the Apple-native path — it uses Metal GPU acceleration directly and, on M5 chips, takes advantage of Neural Accelerators for up to 4x faster prompt processing. llama.cpp is cross-platform and supports a wider range of quantization formats (GGUF variants). For pure Apple Silicon performance, MLX-LM Server is competitive with or slightly ahead of llama.cpp on M-series chips.

Can I serve more than one model?

Yes. Run multiple mlx_lm.server processes on different ports, or register additional deployments in the Models tab. Use a tokios-connector.json config file to define multiple upstreams and routes for a single connector. See Connector config for the full multi-model file format.

What models work with MLX-LM Server?

Any model on HuggingFace that has been converted to MLX format. The mlx-community organization maintains pre-converted versions of popular models — Llama, Qwen, Mistral, Gemma, Phi, and more — in 4-bit, 8-bit, and BF16 quantizations. You can also convert models yourself with mlx_lm.convert.

Your model never leaves your machine — your endpoint works everywhere

MLX-LM Server keeps doing exactly what it does today, on localhost, with no inbound ports and no exposed API. The connector only dials out. Everything past that — auth, routing, streaming — is one endpoint you can call from any machine, teammate, or agent you choose to give a key to.

Quickstart

Pair a connector and register your first model end to end.

API keys

Create, scope, and rotate sk-tok-... keys for teammates and agents.

Mac Studio setup

Turn a Mac Studio into a headless inference box with unified memory for 70B+ models.

Sources

ml-explore/mlx-lm — official repository for the MLX-LM Python package and server
MLX-LM Server documentation (SERVER.md) — server API reference and supported endpoints
MLX framework — the underlying array framework for Apple Silicon
mlx-framework.org — MLX framework overview and ecosystem links
mlx-community on HuggingFace — pre-converted MLX models ready for serving
Exploring LLMs with MLX and the Neural Accelerators in M5 — Apple Machine Learning Research on M5 ANE performance
Run local agentic AI on the Mac using MLX (WWDC26) — Apple session on MLX-LM Server, continuous batching, and distributed inference

Hardware & fit

Serve & connect your runtime

Connect a client or app

Compare

Use an MLX-LM Server from anywhere — safely

What is Apple Silicon native inference

MLX-LM Server’s auth is off by default

The outbound-only way

Use it from anywhere

Server configuration reference

FAQ

Your model never leaves your machine — your endpoint works everywhere

Quickstart

API keys

Mac Studio setup

Sources

​What is Apple Silicon native inference

​MLX-LM Server’s auth is off by default

​The outbound-only way

​Use it from anywhere

​Server configuration reference

​FAQ

​Your model never leaves your machine — your endpoint works everywhere

Quickstart

API keys

Mac Studio setup

​Sources

What is Apple Silicon native inference

MLX-LM Server’s auth is off by default

The outbound-only way

Use it from anywhere

Server configuration reference

FAQ

Your model never leaves your machine — your endpoint works everywhere

Sources