Ivan Plankin

All projects

AI Gateway in Rust

A Rust gateway that lets an application use one familiar API while the system selects where the requested model can run. It keeps account, live quota, network path, and service version changes separate, so the application receives the requested model or a clear failure without losing session context.

What the application gets
One familiar OpenAI- or Anthropic-compatible entrypoint for the requested model, with no separate integration for every provider.
What changes behind it
The gateway selects an account after checking live quota, network path, and service version. An unavailable model is reported explicitly and is not silently replaced.
When no route remains
The application receives a clear failure. Session context stays separate from the failed condition, and an open stream finishes on the version that started it.

How a request crosses the gateway

No prompts, accounts, credentials, private traffic, or proprietary source are shown.

  1. Application APIThe application sends an OpenAI- or Anthropic-compatible request, and a provider-specific integration is not needed.
  2. Provider adapterModels, messages, tools, and streaming events are mapped to the selected provider's protocol.
  3. Where it can runModel support, account access, quota, and the assigned network path are checked together.
  4. Session continuationSession context stays intact while quota, network, and service-version failures follow separate recovery paths.

The application gets one entrypoint

The application uses it like an ordinary API: send a familiar request and receive the requested model's response through the same interface. Inside, the gateway chooses the provider and account, translates the request, and keeps the context when conditions change.

Provider-specific formats, account access, and live quota stay inside the Rust gateway. Application integrations do not need to know about them.

The path of one request

  • One application entrypoint: The application sends a familiar OpenAI or Anthropic request without rebuilding its integration for every provider.
  • Request translation: Messages, tools, thinking fields, and streaming events are mapped to the provider's actual protocol, since APIs do not all behave the same way.
  • Where the request can run: The requested model, account entitlement, live quota, and assigned network path are checked before work starts. If the model is unavailable, the gateway says so and does not silently substitute another one.
  • The session continues: Session context stays attached across requests and restarts. Tool history stays with the same continuation, so the conversation picks up where it left off.
  • Long response and version change: Elapsed time alone cannot end an active generation. A new service version must complete a real model response before new traffic moves to it, while existing connections finish where they started.

What the application gets

One entrypoint
OpenAI and Anthropic APIs
Execution choice
model, access, and live quota
Continuation
persistent session affinity

Why a reverse proxy is not enough for this job

Problem
Blind forwarding or round-robin selection can send a request to an account that cannot run the model, discard a bound session, and hide whether quota, account state, the provider protocol, or the network actually failed.
Solution
The gateway checks where the request can run, translates it for that provider, and keeps session context separate from the failure. Recovery changes only the failed part and keeps the whole integration working.
Why not the other option
A separate thin proxy for every provider is simpler at first, but duplicates client integration and leaves account, quota, and continuation state split across services.

What remains visible for each request

Selected route
The requested and effective model, provider adapter, account capacity, and the reason a route may be unavailable.
Request history
First output, duration, final completion, and recovery reason in the sanitized view, without exposing private prompt content.
Failure class
Quota, account, network, provider rejection, and a long model wait remain different events with different next actions.
Working version
A candidate must finish a real response on the requested model before new work moves to it; older streams keep running.

Implementation

Gateway and adapters
Rust, Axum, Tokio, OpenAI and Anthropic APIs
State and diagnostics
PostgreSQL, SQLx, Prometheus, Grafana
Interface and delivery
Leptos, NixOS, systemd, nginx