Engineering

Open-source AI gateways and inference servers you can host yourself

The open-source tools called “AI gateways” do two different jobs. Routers such as LiteLLM put one OpenAI-style endpoint in front of models that run elsewhere; inference servers such as vLLM, SGLang, Ollama and Xinference load the weights onto your own hardware and generate the tokens. Most production setups pair one of each, and all of them speak OpenAI's API format. Hosting them yourself gives you the whole data path, at the cost of GPUs, upgrades and on-call duty — and when jurisdiction is the only reason, an API whose operators are under EU control is often the lighter way to get there.

Published 6 Oct 2026 · checked on 20 Oct 2026

Two jobs: routing and inference

Start with the job you need done. A router holds no model: it receives your request, applies keys, budgets and retries, and forwards it to a provider or to your own engine. An inference engine loads the weights into memory and computes the answer. The tools below, as their repositories described them on 6 October 2026:

Tool	Job	Runs models itself	OpenAI-format API	Licence
LiteLLM	Router: proxy server and Python SDK	No — forwards to 100+ providers	Yes	MIT, except an enterprise folder
vLLM	Inference engine for production	Yes — GPUs, CPUs, other accelerators	Yes	Apache-2.0
SGLang	Inference engine for production	Yes	Yes	Apache-2.0
Ollama	Local runner	Yes — CPU or GPU	Yes, under /v1	MIT
TGI	Inference server, now archived	Yes	Yes	Apache-2.0
Xinference	Multi-model platform	Yes — GPU or CPU	Yes	Apache-2.0

A common production shape puts a router in front of one or more engines: LiteLLM handles keys and spend, vLLM or SGLang does the computing.

Why self-host, and what you take on

Control of the data path. Prompts and answers stay on machines you run, and no outside company holds them. On a US cloud that control is partial: the provider must disclose data in its “possession, custody, or control” wherever it is stored (18 U.S.C. § 2713).
Custom weights. Fine-tuned or niche models that no provider serves.
A fixed cost. Past a certain volume, renting GPUs costs less than paying per token — the threshold is worked out below.

What you take on: buying or renting GPUs, downloading and upgrading models, load balancing, monitoring, security patches, and the incident at 2 a.m. when a GPU runs out of memory. Without people who already do this work, it can outweigh the savings.

LiteLLM: one front door for many back ends

LiteLLM calls more than 100 LLM providers in OpenAI's format, as a Python library or as a proxy server; its README's examples run the proxy on port 4000. The proxy adds what a team needs around models: virtual keys, cost tracking and spend management per project or user, retries and fallbacks across deployments, load balancing, logging to observability tools, and caching. It is MIT-licensed, except an enterprise folder under a licence of its own.

Its limit is built in: it runs no model. Routing to a US provider still sends the prompt to that company; LiteLLM centralises the choice of back end, it does not change who processes the data. If jurisdiction is the point, put EU-controlled back ends behind it.

vLLM and SGLang: engines for production

vLLM is built around PagedAttention, its way of managing the attention cache, with continuous batching and prefix caching. At its launch in June 2023, its team measured up to 24 times the throughput of Hugging Face Transformers and 3.5 times that of TGI (launch post). It splits models across GPUs (tensor, pipeline and expert parallelism), serves an OpenAI-compatible API, and runs on NVIDIA, AMD and Intel GPUs and on CPUs, with plugins for other accelerators such as Google TPUs. Apache-2.0.

SGLang, also Apache-2.0, is the other engine that TGI's maintainers now point to, with OpenAI-compatible APIs for chat, vision and embeddings. Neither engine manages accounts, budgets or per-team keys: that is the router's job.

Ollama: models on a developer's machine

Ollama pulls and runs a model with one command on macOS, Windows or Linux, or in Docker. It answers on localhost:11434, with its own /api/chat endpoint and OpenAI-compatible ones under /v1 (OpenAI compatibility). A model too large for one GPU is spread across the available GPUs, and the number of parallel requests per model is a setting that defaults to one. MIT-licensed.

It is the right tool for prototypes and for trying models on your own prompts. A shared production service needs autoscaling and load balancing, which Ollama does not provide; teams usually move to vLLM, SGLang or a managed API, and keep their code, since the API format is the same.

TGI is archived; Xinference serves many model types

Hugging Face's Text Generation Inference long powered Hugging Face's own chat and inference services. Its README now says it is in maintenance mode and recommends vLLM, SGLang, llama.cpp or MLX instead, and its repository is archived, read-only, its last change dating from 21 March 2026. Do not start a new deployment on it.

Xinference, from the Xorbits team, serves language, embedding, image and audio models from one deployment, through an OpenAI-compatible API that includes function calling. It can spread inference across several machines, uses GPUs and CPUs, and can run vLLM underneath; it is Apache-2.0, with a commercial edition for production. It suits retrieval pipelines that need a chat model and an embedding model side by side.

What self-hosting costs

Start with memory. At 16-bit precision a model takes about two bytes per parameter: Llama 3.3 70B needs about 140 GB for its weights, so two 80 GB NVIDIA H100s; at 8-bit, about 70 GB, which fits one card with little room left for context. What open-weight AI means has the figures for larger models.

Then the rent. On 6 October 2026, OVHcloud listed an instance with one H100 at €2.80 an hour before VAT (prices): €2,452.80 for a 730-hour month, VAT included. At Evanesca's price for Llama 3.3 70B, €1.512 per million tokens, input or output, VAT included, that month buys about 1.6 billion tokens. Below that volume, the rented card costs more than the API; and since one card cannot hold the model at 16-bit, the threshold for two is about 3.2 billion tokens a month.

Then add what the GPU invoice does not show: engineering time, idle hours, and an upgrade each time a better model ships. Per-token pricing has none of these, and costs nothing when nobody uses it — Evanesca's prices, model by model, are on pricing, and self-hosting or paying per token works through the break-even in more detail.

When a managed EU API is the lighter route

Self-host when you need an air gap or a fully private network, when you run custom or fine-tuned weights, or when your volume is past the threshold above and you have a team to run it. Otherwise — no GPU team, volumes still unknown, or jurisdiction as the main reason — an API whose operators are under EU control gets you there with less work.

The two combine. LiteLLM can route to Evanesca's OpenAI-compatible API like any other provider: models served by European operators — Scaleway in France for most of them, with OVHcloud as a fallback —, no copy of prompts kept by Evanesca, and the operators' own rules: Scaleway may keep and read requests that cause errors or look abusive, for up to 14 days, and Mistral AI keeps its models' inputs and outputs 30 days. A minimal LiteLLM configuration:

config.yaml (LiteLLM)

model_list:
  - model_name: gemma-4-26b
    litellm_params:
      model: openai/gemma-4-26b
      api_base: https://evanesca.app/api/v1
      api_key: os.environ/EVANESCA_API_KEY

A prototype built on Ollama moves to the same API by changing the base URL. See the docs, integrations and the models. Evanesca's API is not open yet; the configuration works once it is.

Frequently asked questions

What is the difference between an AI gateway and an inference server?

A gateway, or router, such as LiteLLM runs no model: it forwards requests to providers or to your own engines and adds keys, budgets, retries and logs. An inference server such as vLLM, SGLang or Ollama loads the weights and computes the answers. Production setups often put a router in front of one or more engines.

Is LiteLLM free to self-host?

Yes: it is MIT-licensed, apart from an enterprise folder under a separate licence. You pay for the servers it runs on and for the providers it calls.

Can I use Ollama in production?

For light internal use, some teams do. It is built to run models on a developer's machine: it can spread a model across GPUs and serve parallel requests, but it has no autoscaling or load balancing. Shared production services usually run vLLM or SGLang, or call a managed API.

Is self-hosting in an EU cloud region enough for sovereignty?

Not if the cloud is run by a US company: US law can require it to disclose data it holds, wherever the servers are. Host on a provider with no US control, or use an API whose operators are under European control, such as the ones behind Evanesca: Scaleway in France for most of its models, with OVHcloud as a fallback.