Local models in LangChain, LlamaIndex, and Vercel AI SDK

All four frameworks let you set a custom OpenAI-compatible base URL for their OpenAI provider — that’s the integration point. Point it at https://api.tokios.com/v1, pass an sk-tok-… API key, and set model to your registered deployment name (for example, gemma-tunnel), not your local backend’s model id. If you’ve ever run a keyless local server, you’ve probably typed a throwaway string like "not-needed" just to satisfy a client that requires an API key field — with Tokios that field takes a real, revocable key instead.

Prerequisites

Install the connector

Pair tokios-connector with your local backend.

Register a deployment

Choose the public name your model field will use.

Create an API key

Generate an sk-tok-… key, scoped to the deployments you need.

Supported backends

Confirm Ollama, llama.cpp, vLLM, or LM Studio is wired up as the upstream.

Configure your framework

LangChain
LlamaIndex
Vercel AI SDK
Pydantic AI

LangChain’s ChatOpenAI class takes a base_url and api_key like the OpenAI SDK it wraps:

langchain_tokios.py

from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    base_url="https://api.tokios.com/v1",
    api_key="sk-tok-YOUR_KEY",
    model="gemma-tunnel",
)

print(llm.invoke("Summarize this code: print('hello')").content)

LlamaIndex talks to OpenAI-compatible servers through OpenAILike, which wraps the OpenAI client but defaults to settings tuned for a generic local server rather than a chat-capable model. Set is_chat_model and is_function_calling_model explicitly if your deployment supports chat and tool calls, and set context_window to match your model — otherwise LlamaIndex falls back to conservative defaults (is_chat_model=False and a small context_window) that quietly degrade behavior. The exact default values can vary between LlamaIndex versions, so confirm them against your installed version.

llamaindex_tokios.py

from llama_index.llms.openai_like import OpenAILike

llm = OpenAILike(
    api_base="https://api.tokios.com/v1",
    api_key="sk-tok-YOUR_KEY",
    model="gemma-tunnel",
    is_chat_model=True,
    is_function_calling_model=True,
    context_window=8192,
)

print(llm.complete("Summarize this code: print('hello')"))

The @ai-sdk/openai-compatible package’s createOpenAICompatible builds a provider from any OpenAI-compatible base URL, then you call it like the built-in OpenAI provider:

vercel-ai-tokios.ts

import { createOpenAICompatible } from '@ai-sdk/openai-compatible';
import { generateText } from 'ai';

const tokios = createOpenAICompatible({
  name: 'tokios',
  baseURL: 'https://api.tokios.com/v1',
  apiKey: process.env.TOKIOS_API_KEY,
});

const { text } = await generateText({
  model: tokios('gemma-tunnel'),
  prompt: "Summarize this code: print('hello')",
});

console.log(text);

Set TOKIOS_API_KEY in your environment rather than hardcoding the sk-tok-… key.

Pydantic AI’s OpenAIProvider accepts a custom base_url, and OpenAIChatModel takes that provider alongside your deployment name:

pydantic_ai_tokios.py

from pydantic_ai.models.openai import OpenAIChatModel
from pydantic_ai.providers.openai import OpenAIProvider

model = OpenAIChatModel(
    "gemma-tunnel",
    provider=OpenAIProvider(
        base_url="https://api.tokios.com/v1",
        api_key="sk-tok-YOUR_KEY",
    ),
)

Across all four frameworks, model is always the Tokios deployment name you registered in the Models tab — never the local model id your backend serves — and the base URL must include /v1. Omitting /v1 is the most common misconfiguration when adapting a snippet from a different OpenAI-compatible provider.

FAQ

Why does LlamaIndex need is_chat_model and is_function_calling_model set explicitly?

OpenAILike is a generic wrapper for OpenAI-compatible servers that don’t necessarily behave like OpenAI’s own chat models, so it doesn’t assume chat or function-calling support out of the box. Confirm the exact default values against your installed LlamaIndex version, and set both explicitly to match what your underlying deployment actually supports.

Do I still need a dummy API key value?

No. Frameworks built against local, keyless servers often expect a placeholder string because the field is required but unchecked. Tokios keys are real sk-tok-… credentials — created, scoped, and revoked from the Keys tab — so the API key field does actual authentication instead of satisfying a client-side check.

Does streaming work in these frameworks?

Yes. Tokios returns standard OpenAI-format SSE when a request sets "stream": true, so each framework’s native streaming support (for example, LangChain’s .stream() or the Vercel AI SDK’s streamText) works the same as it would against OpenAI directly.

Now call it from anywhere

Your backend still runs on localhost — the connector only dials out, and nothing you configured above changes that. What changes is who can reach it: once a teammate or a deployed agent has a scoped sk-tok-… key, the same LangChain, LlamaIndex, Vercel AI SDK, or Pydantic AI code that worked against your machine works from theirs.

Compatible clients

See the full list of OpenAI- and Anthropic-compatible clients Tokios works with.

API keys

Create a separate scoped key per teammate or agent.

Hardware & fit

Serve & connect your runtime

Connect a client or app

Compare

Local models in LangChain, LlamaIndex, and Vercel AI SDK

Prerequisites

Install the connector

Register a deployment

Create an API key

Supported backends

Configure your framework

FAQ

Now call it from anywhere

Compatible clients

API keys

​Prerequisites

Install the connector

Register a deployment

Create an API key

Supported backends

​Configure your framework

​FAQ

​Now call it from anywhere

Compatible clients

API keys

Prerequisites

Configure your framework

FAQ

Now call it from anywhere