Self-Hosted LLM Gateway for Model Routing and Cost Control
LLM gateway deployment for companies running AI on several models. One OpenAI-compatible endpoint in your own cloud account or data center routes each request to a self-hosted model on vLLM or Ollama, or to an API model such as OpenAI or Anthropic under your own keys, with per-team budgets, rate limits, PII redaction and logs on every call.
What an LLM Gateway Does for Your AI Traffic
An LLM gateway sits between your applications and the AI models they call. Apps, agents and internal tools send every request to one address.
One OpenAI-compatible endpoint
Apps call one URL in the OpenAI format, so adding or swapping a model is a gateway setting.
Model routing
The model router picks a model for each request by name, team or rule, such as a self-hosted model for internal documents.
Fallback and retries
When a model errors or reaches its rate limit, the gateway retries, then moves to the next model on your fallback list.
Keys per team
Each team, app or project gets its own virtual key, and provider keys stay inside the gateway.
Budgets and rate limits
Spend caps and token or request limits per key and per team, with the cost of each request recorded.
Request logs
Every call is logged with its key, team, model and cost, and exported to your observability tools.
What Runs Where in a Self-Hosted AI Gateway
The gateway, your self-hosted models and the logs run in your environment. Only requests you route to an external model API leave your network, after redaction, and their responses come back.
| Component | Where it runs | What crosses your network boundary |
|---|---|---|
| LLM gateway (LiteLLM, Portkey or Kong) | Containers in your cloud account or on your own servers | Receives requests from your apps on the internal network |
| Self-hosted models | vLLM or Ollama on GPU servers in the same network | Prompts and responses stay inside your network |
| API models (OpenAI, Anthropic, Gemini and others) | The provider's cloud, under your own API keys and contract | The redacted prompt goes out, and the response comes back |
| Provider API keys | Held by the gateway, while apps use virtual keys | Each key goes only to its own provider |
| Presidio redaction, logs and traces | Next to the gateway and in your observability stack | Stay inside your network |
vLLM ships an OpenAI-compatible API server and Ollama supports a subset of the OpenAI API, per each project's docs, so the gateway calls your own models the way it calls OpenAI. We set them up through our vLLM deployment or Ollama deployment services, inside a self-hosted AI deployment.
Model Routing and Fallback Across Self-Hosted and API Models
The LLM router decides which AI model answers each request and what happens when it is busy or errors. These settings follow LiteLLM's routing and reliability docs.
Model groups under one name
Several deployments of one model, such as two vLLM servers, sit behind one name. LiteLLM's default simple-shuffle strategy spreads requests across them.
Routing by team, task or data class
Internal documents go to a self-hosted model on vLLM or Ollama, and general drafting can go to an API model. Each team's key lists the models it may call.
Latency- and cost-based routing
The router can pick the deployment with the lowest recent latency or the lowest configured cost.
Retries, then fallback in order
After its retries, LiteLLM moves down your fallback list in order: a second self-hosted model first, then an API model where your AI policy allows it. Separate lists cover context-window and content-policy errors.
Cooldowns for failing deployments
A deployment that keeps failing rests in cooldown for the period you set while traffic moves to healthy ones.
Per-Team Keys, Budgets and Rate Limits for AI Spend
AI cost control starts with knowing who sent each request, so every team, app and agent gets its own virtual key.
Virtual keys per team
Each user, team or project gets a key that names the models it can call. Calls to vLLM and Ollama models use the same keys as API calls.
Team, member and key budgets
A shared spend cap per team, with optional caps per member and per key, reset on your schedule. LiteLLM records each request's cost in its spend logs.
Rate limits
Token, request and parallel-request limits per key or team keep capacity available for every team.
PII Redaction Before a Prompt Reaches an External AI Model
When a request is routed to an API model, the gateway runs your redaction rules first. LiteLLM's Presidio guardrail, in pre_call mode, masks or blocks personal data before the model call, and Presidio runs in containers inside your network:
Redaction runs in your network, before the API call.
The external AI model receives the masked prompt, and the original values stay in your systems. Portkey and Kong offer PII redaction in their enterprise editions.
AI Request Logs and Observability Export
Every model call leaves a record your platform and security teams can query.
One record per request
Each call is stored with its key, team, model and cost, feeding spend views and budget checks.
OpenTelemetry traces
LiteLLM exports OpenTelemetry traces with W3C traceparent propagation to your collector. Kong's open-source repository includes an OpenTelemetry plugin.
Langfuse for prompt-level traces
LiteLLM sends prompts, responses and metadata to Langfuse, which is MIT-licensed and can be self-hosted beside the gateway, per its repository.
Metrics, storage and SIEM
Request rates, errors and spend go to Prometheus or Datadog, and logs go to S3 or a SIEM such as Azure Sentinel under your retention policy.
Redacted logging
Personal data is masked before it reaches any log, using the same Presidio rules as live traffic.
Open-Source AI Gateways We Deploy and Configure
We deploy the enterprise AI gateway that fits your stack, and you own it. Each row comes from the project's own repository and docs.
| Gateway | Licence, as the project states it | Routing and fallback | Keys, budgets and limits | PII redaction | Logs and export |
|---|---|---|---|---|---|
| LiteLLM proxy | MIT outside the enterprise/ directory, which has its own licence | Weighted, latency and cost routing; ordered fallbacks, retries and cooldowns | Virtual keys; budgets per user, team and key; token, request and parallel limits | Presidio guardrail that masks or blocks before the model call | Langfuse, OpenTelemetry, Prometheus, Datadog, S3 |
| Portkey AI Gateway | MIT | Fallbacks on chosen errors, up to 5 retries, weighted load balancing, conditional routing | Secure key management in the enterprise version | 40+ guardrails; PII redaction in Portkey's enterprise version | Local logs in the Gateway Console; full tracing on Portkey's hosted app |
| Kong Gateway | Apache-2.0 | AI Proxy plugin for OpenAI, Anthropic, Gemini, Bedrock, Ollama, vLLM and more | Key authentication and rate-limiting plugins | AI Sanitizer, in Kong's AI Gateway Enterprise offering | OpenTelemetry and Prometheus plugins; AI Proxy adds model usage to logs |
Teams looking for a LiteLLM alternative can compare Portkey, for config-driven routing in a lightweight Node.js service, and Kong, for companies already running Kong Gateway for their APIs.
How the LLM Gateway Pairs With an MCP Gateway
AI agents send two kinds of traffic. Model calls go through the LLM gateway and tool calls go through the MCP gateway, each with its own keys, rules and logs.
LLM gateway: model traffic
Prompts and completions between your apps, agents and AI models, with routing, budgets, redaction and logs.
MCP gateway: tool traffic
Tool calls from Claude, ChatGPT and your agents to company systems, under SSO, scopes and an audit log.
Integrations that use both
Agents that call a model and a company system in one workflow, connected through both gateways by our AI integration services.
What Shapes an LLM Gateway Engagement
The scope follows your models, teams and data rules. Bring your model list and AI use policy, and we will map the deployment that fits.
Book a Strategy Session →Frequently Asked Questions on LLM Gateways
An LLM gateway, also called an AI gateway, is one OpenAI-compatible endpoint between your applications and the AI models they call. It routes each request, falls back when a model errors, holds provider keys, enforces per-team budgets and rate limits, and logs every call.
The best fit follows your stack. LiteLLM's docs list virtual keys, budgets, rate limits, fallbacks and Presidio redaction without an enterprise label, with admin UI SSO and audit logs under its commercial licence. Portkey's MIT-licensed gateway covers fallbacks, retries, load balancing and guardrails. Kong suits companies already running Kong Gateway.
Portkey's gateway and LiteLLM's code outside its enterprise directory are MIT-licensed and Kong Gateway is Apache-2.0, so the open-source editions carry no licence fee. Running cost is compute for the gateway, GPUs for any self-hosted models, and the API usage you pay each provider under your own keys. Our scope depends on your models, teams and redaction rules, and the strategy session gives you a scoped figure.
OpenRouter's documentation says it gives you access to hundreds of AI models through a single API endpoint, served from its hosted service. Teams searching for OpenRouter enterprise or OpenRouter self-hosted options often want that endpoint inside their own network. A self-hosted LLM gateway provides it, with your own keys, budgets, redaction and logs, and your vLLM or Ollama models behind it.
Related Self-Hosted AI Solutions and Services
Self-Hosted MCP Gateway for Claude and ChatGPT Connectors
Learn more →Self-Hosted AI Deployment: A Private AI Stack in Your Environment
Learn more →AI Integration Services for Claude, ChatGPT and Your Apps
Learn more →vLLM Deployment
Open-weight models on your GPUs, served through an OpenAI-compatible API.
See vLLM deployment →Ollama Deployment
A private, local LLM on your own servers, ready to sit behind the gateway.
See Ollama deployment →AI Strategy Session
A scoping call on your models, teams, budgets and data rules.
Book a session →Ready to Put One Gateway in Front of Your AI Models?
A strategy session to map your models, teams and data rules to a self-hosted LLM gateway.