When you connect OpenAI, Claude, DeepSeek, and Gemini at the same time, the hard part is usually not choosing a model. It is that every SDK has a different request format, key, and billing model. LiteLLM is an open-source project built to solve exactly that: it unifies multiple providers behind an OpenAI-compatible interface and provides a gateway you can self-host.

This article explains what LiteLLM is, the smallest working configuration, routing and key management, deployment options, and when to self-host versus use a managed gateway.

Note: LiteLLM is an open-source project of its respective owner. OpenAI, Anthropic, DeepSeek, and other names are trademarks of their respective owners. This article uses common versions as of October 2026; refer to the official documentation for exact parameters.

1. What LiteLLM actually is

LiteLLM comes in two forms:

  • Python SDK: call different providers' models through a unified completion function in your code.
  • Proxy gateway: a standalone service that exposes an OpenAI-compatible /v1/chat/completions endpoint, so any client supporting a custom base URL can connect.

It solves three things: protocol adaptation, routing, and accounting. It does not make a model good or bad. In other words, it is plumbing, not a model.

Core capabilities include:

Capability What it does
Unified protocol Multiple providers use the OpenAI format
Routing and load balancing One model name distributes across multiple deployments
Fallbacks Switch to a backup model when the primary fails
Virtual keys Issue separate keys and budgets to teams or projects
Usage and cost Record requests, tokens, and spend
Logging callbacks Integrate with Langfuse, OpenTelemetry, and similar systems

2. Minimum viable setup: start with pip

Create a virtual environment, then install the package with proxy support:

python3 -m venv .venv
source .venv/bin/activate
pip install "litellm[proxy]"

Write a minimal config.yaml:

model_list:
  - model_name: claude-sonnet
    litellm_params:
      model: anthropic/claude-sonnet-4-5
      api_key: os.environ/ANTHROPIC_API_KEY

  - model_name: deepseek-chat
    litellm_params:
      model: deepseek/deepseek-chat
      api_key: os.environ/DEEPSEEK_API_KEY

litellm_settings:
  drop_params: true

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY

Start it:

export ANTHROPIC_API_KEY="sk-ant-your-key"
export DEEPSEEK_API_KEY="sk-your-key"
export LITELLM_MASTER_KEY="sk-litellm-your-local-admin-key"
litellm --config config.yaml --port 4000

model_name is the name exposed to clients, while litellm_params.model is the real upstream model, usually in the form "provider/model-name". os.environ/XXX reads from an environment variable. Do not put keys directly in the YAML and commit it to a repository.

Once it is running, point the OpenAI SDK at the local gateway:

from openai import OpenAI

client = OpenAI(
    api_key="sk-litellm-your-local-admin-key",
    base_url="http://127.0.0.1:4000",
)
resp = client.chat.completions.create(
    model="claude-sonnet",
    messages=[{"role": "user", "content": "Reply with OK only"}],
)
print(resp.choices[0].message.content)

3. Model routing and fallbacks

Production rarely relies on a single upstream. A common configuration looks like this:

router_settings:
  routing_strategy: simple-shuffle
  num_retries: 2
  timeout: 600
  fallbacks:
    - claude-sonnet: ["deepseek-chat"]

What the parameters mean:

  • routing_strategy: how to choose among multiple deployments of the same model. simple-shuffle is a common starting point.
  • num_retries: how many times to retry after failure. Every retry incurs real cost.
  • timeout: timeout for a single request. Set it according to your slowest business path.
  • fallbacks: switch to a backup model when the primary is unavailable.

Fallbacks are not free. The backup model may differ in output style, context length, and tool-calling ability, and the business side must accept that. For scenarios with strict formatting requirements, it is better to fail explicitly than to degrade silently.

4. Virtual keys and budgets

Sharing one master key across a team is the beginning of an operational incident. The right approach is to issue a virtual key per project and limit the models and budget it can use:

curl -X POST "http://127.0.0.1:4000/key/generate" \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "models": ["claude-sonnet"],
    "max_budget": 20,
    "duration": "30d",
    "metadata": {"project": "demo"}
  }'

Give the returned key to the business team and keep the master key with operations only. The budget unit and currency follow LiteLLM's documentation; this example only demonstrates the field structure and does not represent any pricing.

5. Docker deployment and pre-launch checks

For a single-machine trial, run the container directly:

docker run -d --name litellm \
  -p 4000:4000 \
  -v "$(pwd)/config.yaml:/app/config.yaml" \
  -e ANTHROPIC_API_KEY \
  -e DEEPSEEK_API_KEY \
  -e LITELLM_MASTER_KEY \
  ghcr.io/berriai/litellm:main-latest \
  --config /app/config.yaml

Before going live, check at least the following:

  1. Do not expose the gateway directly to the public internet. Put Nginx or Caddy in front for TLS and access control.
  2. Use a strong random value for the master key and never commit it to the code repository.
  3. Connect PostgreSQL if you need usage persisted; otherwise some statistics are lost after a restart.
  4. Configure log rotation and disk alerts; request logs grow quickly.
  5. Keep a health check: curl http://127.0.0.1:4000/health.
  6. Read the changelog before upgrading, pin the image to a version tag, and do not stay on latest long term.

One more thing is easy to overlook: the gateway is a new single point of failure. If it goes down, every model goes down. At minimum, add process supervision, automatic restarts, and an independent alerting path.

6. Logs and cost: do not wait for the bill

One hidden value of a gateway is that it records calls. LiteLLM can record the model, tokens, latency, and cost of each request, and aggregate them by key, team, and tag. To do this completely, you need three steps:

  1. Configure a database so cost records persist instead of living only in memory.
  2. Tag requests, for example by putting the project name in metadata; otherwise you cannot split costs by project at month end.
  3. Connect an observability backend to centralize tracing and logs.

Example configuration:

litellm_settings:
  success_callback: ["langfuse"]
  failure_callback: ["langfuse"]

general_settings:
  database_url: os.environ/DATABASE_URL

Two cautions: first, the callback service itself must be highly available, because a blocked callback can slow down the main request; second, logs may contain raw user text, so define retention periods and access permissions in advance and do not keep production logs forever.

On cost, a gateway only makes spending visible; it does not automatically reduce spending. The real savings still come from choosing the right model, compressing context, and limiting retries.

7. Self-host or managed gateway

Dimension Self-hosted LiteLLM Managed AI gateway
Data path Fully under your control Passes through the provider
Initial cost Low; runs once installed Sign up and use
Long-term operations Upgrades, backups, and monitoring are yours Handled by the provider
Billing and invoices You connect to upstream yourself Unified RMB billing
Flexibility High; source and routing can be changed Limited by platform capabilities
Failure responsibility You handle it Per the terms of service

A practical rule:

  • Self-host if you have operations staff, compliance requirements, and a need for deeply customized routing.
  • Choose managed if your team is small, you only want unified keys and billing, and you do not want to fix containers at midnight.
  • You can also start managed to get the business working, then evaluate self-hosting once call volume stabilizes.

Teams in mainland China must also consider networking. A self-hosted gateway does not solve upstream reachability by itself; when upstream access is unstable, the gateway only amplifies the failures. In that case, "self-hosted plus a reachable domestic upstream" or "use a managed gateway directly" is more realistic.

If you do not need to modify source code and just want an OpenAI-compatible entry point, unified keys, and RMB billing, a managed gateway can usually be working the same day you register. LiteLLM is better suited to infrastructure that needs long-term maintenance. The two can also coexist: use LiteLLM internally for routing and a managed entry point externally.

8. FAQ

Is LiteLLM free?

The open-source part can be self-deployed, but upstream model calls still incur normal fees. Pricing for enterprise features and managed services is subject to the official source.

How is it different from One API and New API?

One API and New API focus more on multi-user quota distribution, while LiteLLM focuses more on multi-model routing and SDK unification. Both can provide an OpenAI-compatible entry point; choose based on whether you need billing distribution or routing capability.

Does it support Claude's native tool calling?

Support depends on the version and upstream adapter. For complex tool calls, caching, and similar features, verify with real requests rather than relying on documentation alone.

Can a self-hosted gateway save money?

Not necessarily. You save the platform service fee but pay for servers, operations, and downtime. At low call volume, managed is usually more economical.

What is the minimum for production?

TLS, a master key, virtual keys, budget limits, logging, database backups, and health checks. None of these can be omitted.

Next steps

If you want to validate multi-model calls first without maintaining containers yourself, register for the ZSQH AI Gateway and run your first request through the OpenAI-compatible endpoint. Models and prices are subject to model pricing. Once call volume and compliance requirements are clear, decide whether to self-host LiteLLM.

Related reading: