> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tokios.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Tokios exposes private local models through https://api.tokios.com using an outbound-only connector tunnel.
> Use 'connector' for the local tokios-connector process.
> Use 'deployment' for the public model name in the API model field.
> Use 'upstream model id' for the model served by Ollama, llama.cpp, vLLM, LM Studio, or another local backend.
> API keys use the prefix sk-tok-. Connector tokens use the prefix ct-tok-.

# Use an MLX-LM Server from anywhere — safely

> Reach your MLX-LM Server from anywhere on Apple Silicon — set up secure remote access with Tokios connector and API key auth. Start serving in 5 minutes.

By default, `mlx_lm.server` listens on `http://127.0.0.1:8080/v1` — it isn't reachable from another machine unless you bind it wider or forward the port. MLX-LM Server has no built-in authentication: anything that can reach port `8080` can call the API with no credential. Keep MLX-LM Server on localhost and run the Tokios connector next to it instead: the connector dials out to `https://api.tokios.com`, and your tools, teammates, or agents call one authenticated endpoint from anywhere.

> [!NOTE]
>
> Last verified: 2026-10-14. MLX-LM Server ships with the [`mlx-lm`](https://github.com/ml-explore/mlx-lm) Python package from Apple's ml-explore team. This guide covers the current release.

> [!TIP]
>
> **Apple Silicon native.** MLX-LM Server is the reference inference server for Apple Silicon. It uses the MLX framework's unified-memory architecture with zero-copy data paths — model weights live in the same memory pool as the CPU and GPU, so there is no separate VRAM to manage. On M5 chips, Neural Accelerators deliver up to 4x faster prompt processing compared to M4.

## What is Apple Silicon native inference

MLX-LM Server is different from the other backends Tokios supports: it runs only on Apple Silicon Macs (M1 and later) and takes direct advantage of the unified-memory architecture that all Apple Silicon chips share.

| Feature | MLX-LM Server | llama.cpp on Mac | vLLM |
| - | - | - | - |
| Hardware | Apple Silicon only | Cross-platform (CPU + GPU) | NVIDIA GPU (CUDA) |
| Memory model | Unified memory, zero-copy | Unified memory via Metal | Separate VRAM |
| Neural Accelerators (ANE) | Yes — automatic on M5 | No | No |
| Installation | `pip install mlx-lm` | Build from source or prebuilt binary | `pip install vllm` + CUDA |
| Quantization | 4-bit, 8-bit, BF16 | GGUF (Q2\_K through Q8\_0) | AWQ, GPTQ, FP16, BF16 |
| Prompt processing on M5 | Up to 4x vs M4 | Baseline | N/A |

Because MLX uses unified memory, a Mac Studio with 192 GB of RAM can serve 70B models at full precision without the multi-GPU setup that other platforms require. There is no VRAM boundary — the model, the KV cache, and the working memory all share the same pool.

## MLX-LM Server's auth is off by default

`mlx_lm.server` starts an OpenAI-compatible HTTP server on port `8080`. Run it and it accepts requests from anyone who can reach the port — no credential required:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
mlx_lm.server --model mlx-community/Qwen-3.5-4B-8bit
```

MLX-LM Server does not ship an authentication layer. Its [own documentation](https://github.com/ml-explore/mlx-lm/blob/main/mlx_lm/SERVER.md) notes the server is not recommended for production use without a reverse proxy in front of it. That's fine for local-only use, but it means exposing the port beyond loopback — via `--host 0.0.0.0`, port forwarding, or a tunnel — gives unauthenticated API access to anyone who can reach it.

> [!NOTE]
>
> This isn't a knock on MLX-LM Server — it's a lightweight, Apple-Silicon-native inference server, and leaving auth to an external proxy keeps the server simple. Tokios is purpose-built for a different problem: turning that local endpoint into an authenticated API you can call from anywhere, without opening an inbound port or hand-rolling TLS.

## The outbound-only way

**1. Install MLX-LM and start the server locally**

MLX-LM Server requires an Apple Silicon Mac (M1 or later) with macOS 14.0 (Sonoma) or later. Install the package and start the server:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
pip install mlx-lm
mlx_lm.server --model mlx-community/Qwen-3.5-4B-8bit
```

The server downloads the model from [mlx-community](https://huggingface.co/mlx-community) on first use and listens on `http://127.0.0.1:8080`. Leave it on the default `127.0.0.1` — don't pass `--host 0.0.0.0`.

> [!TIP]
>
> **Choosing a quantization.** Models on mlx-community ship in 4-bit (default, smallest), 8-bit, and BF16 (full precision) variants. A 70B model at 4-bit needs roughly 35 GB of unified memory; at 8-bit roughly 70 GB; at BF16 roughly 140 GB. Pick the quantization that fits your Mac's memory.

**2. Pair the Tokios connector**

Sign in at [tokios.com/console](https://tokios.com/console), open the **Setup** tab, and click **Start pairing**. Tokios shows a one-time **claim code** like `TKS-XXXX-XXXX`, valid for about 15 minutes.

On the Mac running MLX-LM Server, run the macOS install one-liner:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
curl -fsSL https://tokios.com/downloads/install/macos | sh -s -- --code TKS-XXXX-XXXX
```

Already have `tokios-connector` installed? Pass the code directly instead:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
tokios-connector TKS-XXXX-XXXX
```

Back in the console, **approve** the device on the **Setup** or **Connectors** tab. The connector's token is delivered over the tunnel once approved — you never see it on screen or paste it into a file.

**3. Point the connector at MLX-LM Server's /v1 API**

MLX-LM Server already speaks the OpenAI Chat Completions surface Tokios expects. For the basic single-model path, the console's Setup wizard uses `http://127.0.0.1:8080/v1` as the upstream automatically. If you're serving multiple models through a `tokios-connector.json` config, set the upstream `BaseUrl` explicitly:

```json theme={"theme":{"light":"github-light","dark":"css-variables"}}
{
  "Upstream": {
    "BaseUrl": "http://127.0.0.1:8080/v1"
  }
}
```

See [Connector config](/reference/connector-config.md) for the full multi-model file format.

**4. Register a deployment**

In the [Models](https://tokios.com/console/models) tab, register the MLX-LM model and choose a **public name** — for example, `mlx-qwen-tunnel`. That public name, not the HuggingFace model id, is what clients send in the `model` field.

**5. Create an API key**

In the [Keys](https://tokios.com/console/keys) tab, click **Create key**. Optionally scope it to specific deployments with model-name patterns (comma-separated globs, `*` for all). Copy the `sk-tok-...` key now — it's shown only once.

## Use it from anywhere

Once you have a deployment name and an API key, any OpenAI- or Anthropic-compatible client can reach your MLX-LM model — from another machine, a teammate's laptop, or a hosted agent.

```bash curl theme={"theme":{"light":"github-light","dark":"css-variables"}}
curl https://api.tokios.com/v1/chat/completions \
  -H "Authorization: Bearer sk-tok-YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mlx-qwen-tunnel",
    "stream": true,
    "messages": [{"role": "user", "content": "Hello!"}]
  }'
```

```python Python (OpenAI SDK) theme={"theme":{"light":"github-light","dark":"css-variables"}}
from openai import OpenAI

client = OpenAI(
    base_url="https://api.tokios.com/v1",
    api_key="sk-tok-YOUR_KEY",
)

response = client.chat.completions.create(
    model="mlx-qwen-tunnel",
    messages=[{"role": "user", "content": "Hello!"}],
    stream=True,
)

for chunk in response:
    print(chunk.choices[0].delta.content or "", end="")
```

```bash Claude Code theme={"theme":{"light":"github-light","dark":"css-variables"}}
export ANTHROPIC_BASE_URL=https://api.tokios.com
export ANTHROPIC_API_KEY=sk-tok-YOUR_KEY
claude --model mlx-qwen-tunnel
```

> [!TIP]
>
> The `model` field is always your registered **deployment** name (`mlx-qwen-tunnel` above), never the HuggingFace model id. See [Use Claude Code](/guides/claude-code.md) for persisting the environment variables across sessions.

## Server configuration reference

MLX-LM Server exposes several flags for tuning inference. The most useful ones:

| Flag | Default | Purpose |
| - | - | - |
| `--model` | (required) | HuggingFace model id or local path |
| `--host` | `127.0.0.1` | Bind address — keep on `127.0.0.1` for Tokios |
| `--port` | `8080` | Listen port |
| `--adapter-path` | — | Path to a LoRA adapter for serving fine-tuned models |
| `--max-tokens` | `512` | Default maximum tokens per generation |
| `--temp` | `0.0` | Sampling temperature |
| `--top-p` | `1.0` | Top-p (nucleus) sampling |
| `--top-k` | `0` | Top-k sampling (0 = disabled) |
| `--kv-bits` | — | KV cache quantization bits (reduces memory) |
| `--kv-group-size` | `64` | Group size for KV cache quantization |
| `--prefill-step-size` | `2048` | Prefill chunk size for long contexts |
| `--draft-model` | — | Model for speculative decoding |
| `--num-draft-tokens` | `3` | Draft tokens per speculative decoding step |
| `--prompt-cache-size` | `10` | Number of prompts to cache |
| `--decode-concurrency` | `32` | Maximum concurrent decode operations |
| `--prompt-concurrency` | `8` | Maximum concurrent prompt operations |
| `--trust-remote-code` | `false` | Trust remote code from HuggingFace repos |

The server exposes these endpoints:

| Endpoint | Purpose |
| - | - |
| `/v1/chat/completions` | Chat Completions (OpenAI-compatible) |
| `/v1/completions` | Text Completions (OpenAI-compatible) |
| `/v1/models` | List loaded models |
| `/health` | Health check |

> [!TIP]
>
> **Speculative decoding** can speed up generation by 2-3x for large models. Pass `--draft-model` with a smaller model of the same family — for example, use a 4B model as the draft for a 32B model. MLX-LM Server verifies draft tokens in parallel and only recomputes mismatches.

## FAQ

> **What are the prerequisites for MLX-LM Server?**
>
> You need an Apple Silicon Mac (M1, M2, M3, M4, or M5) running macOS 14.0 (Sonoma) or later, Python 3.9 or later, and `pip` or `conda`. Install with `pip install mlx-lm`. The MLX framework uses Metal for GPU acceleration and unified memory for model weights — no separate GPU drivers or CUDA toolkit needed.

> **How do I install MLX-LM Server?**
>
> Install the Python package with `pip install mlx-lm` (or `conda install -c conda-forge mlx-lm`). Then start the server with `mlx_lm.server --model <model-id>`. The model is downloaded from HuggingFace on first use. Models from the [mlx-community](https://huggingface.co/mlx-community) organization are pre-converted to MLX format and ready to use.

> **What port does MLX-LM Server use?**
>
> MLX-LM Server listens on port **8080** by default, bound to `127.0.0.1` (localhost only). Change the port with `--port` and the bind address with `--host`. For Tokios, keep the defaults — the connector reaches the server on `http://127.0.0.1:8080/v1`.

> **Does MLX-LM Server support streaming?**
>
> Yes. Set `"stream": true` in the request body and the server returns standard server-sent events (SSE). Through Tokios, streaming works end-to-end — set `"stream": true` and Tokios returns SSE in OpenAI or Anthropic format, depending on which API surface you called.

> **Does MLX-LM Server support tool calling?**
>
> Yes. MLX-LM Server supports structured tool calling. Use a model that supports chat templates with tool definitions — models from the [mlx-community](https://huggingface.co/mlx-community) such as Qwen 3.5 and Llama 3 include the necessary chat templates. Pass tools in the OpenAI-compatible `tools` field of the request.

> **Can I use LoRA adapters with MLX-LM Server?**
>
> Yes. Pass `--adapter-path` pointing to a directory with LoRA adapter weights. Fine-tune adapters with `mlx_lm.lora` (included in the same package) and then serve the base model with the adapter overlaid. Register the adapter as its own Tokios deployment with its own public name.

> **How does MLX-LM Server compare to llama.cpp on Apple Silicon?**
>
> Both use unified memory on Apple Silicon. MLX-LM Server is the Apple-native path — it uses Metal GPU acceleration directly and, on M5 chips, takes advantage of Neural Accelerators for up to 4x faster prompt processing. llama.cpp is cross-platform and supports a wider range of quantization formats (GGUF variants). For pure Apple Silicon performance, MLX-LM Server is competitive with or slightly ahead of llama.cpp on M-series chips.

> **Can I serve more than one model?**
>
> Yes. Run multiple `mlx_lm.server` processes on different ports, or register additional deployments in the [Models](https://tokios.com/console/models) tab. Use a `tokios-connector.json` config file to define multiple upstreams and routes for a single connector. See [Connector config](/reference/connector-config.md) for the full multi-model file format.

> **Can I share access with teammates without exposing MLX-LM Server directly?**
>
> Yes. Create a separate `sk-tok-...` key per teammate in the [Keys](https://tokios.com/console/keys) tab, and scope each key to only the deployments they need. Revoking a key stops it from authenticating immediately, so you can cut off access without touching MLX-LM Server or the connector.

> **What models work with MLX-LM Server?**
>
> Any model on HuggingFace that has been converted to MLX format. The [mlx-community](https://huggingface.co/mlx-community) organization maintains pre-converted versions of popular models — Llama, Qwen, Mistral, Gemma, Phi, and more — in 4-bit, 8-bit, and BF16 quantizations. You can also convert models yourself with `mlx_lm.convert`.

## Your model never leaves your machine — your endpoint works everywhere

MLX-LM Server keeps doing exactly what it does today, on localhost, with no inbound ports and no exposed API. The connector only dials out. Everything past that — auth, routing, streaming — is one endpoint you can call from any machine, teammate, or agent you choose to give a key to.

> **[Quickstart](/quickstart.md)**
>
> Pair a connector and register your first model end to end.

> **[API keys](/setup/api-keys.md)**
>
> Create, scope, and rotate `sk-tok-...` keys for teammates and agents.

> **[Mac Studio setup](/use-cases/mac-studio.md)**
>
> Turn a Mac Studio into a headless inference box with unified memory for 70B+ models.

## Sources

* [ml-explore/mlx-lm](https://github.com/ml-explore/mlx-lm) — official repository for the MLX-LM Python package and server
* [MLX-LM Server documentation (SERVER.md)](https://github.com/ml-explore/mlx-lm/blob/main/mlx_lm/SERVER.md) — server API reference and supported endpoints
* [MLX framework](https://github.com/ml-explore/mlx) — the underlying array framework for Apple Silicon
* [mlx-framework.org](https://mlx-framework.org/) — MLX framework overview and ecosystem links
* [mlx-community on HuggingFace](https://huggingface.co/mlx-community) — pre-converted MLX models ready for serving
* [Exploring LLMs with MLX and the Neural Accelerators in M5](https://machinelearning.apple.com/research/exploring-llms-mlx-m5) — Apple Machine Learning Research on M5 ANE performance
* [Run local agentic AI on the Mac using MLX (WWDC26)](https://developer.apple.com/videos/play/wwdc2026/232/) — Apple session on MLX-LM Server, continuous batching, and distributed inference


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.