Call the LiteLLM gateway
Read Credit wallet and LiteLLM gateway first for the model. In short: the user's access token for the gateway audience is the API key, the gateway debits the wallet, and your application never handles a debit.
Prerequisites
- LiteLLM support is on for the application in auth-service, and the plan of the user grants access to models (see the auth-service guide LiteLLM gateway and credit wallet in an application).
gateway_urlis set in theAuthClientconfiguration, and anAiClientexists:AiClient(auth=auth, grants=grants).- The user signed in after
gateway_urlwas set, so their grant covers the gateway and wallet audiences.
With the SDK's own calls
UserAi.gateway.post_json(path, body) posts JSON and returns the parsed answer as object; you validate it. A refusal raises an AuthAiError.
from typing import Annotated
from fastapi import Depends, FastAPI
from fastapi.responses import JSONResponse
from pydantic import BaseModel, Field
from auth_ai import AiClient, AuthAiError, is_insufficient_credit
from auth_ai.starlette import require_auth
from auth_ai.web import AuthHandlers, Session
class ChatBody(BaseModel):
model: str = Field(min_length=1)
message: str = Field(min_length=1, max_length=2000)
class Message(BaseModel):
content: str | None
class Choice(BaseModel):
message: Message
class ChatAnswer(BaseModel):
choices: list[Choice]
def add_chat_route(app: FastAPI, handlers: AuthHandlers, ai: AiClient) -> None:
current_session = require_auth(handlers)
@app.post("/api/chat", response_model=None)
async def chat(
body: ChatBody, session: Annotated[Session, Depends(current_session)]
) -> dict[str, str] | JSONResponse:
user = ai.for_user(session.user_id)
try:
answer = await user.gateway.post_json(
"/v1/chat/completions",
{
"model": body.model,
"messages": [{"role": "user", "content": body.message}],
"metadata": {"session_id": session.id},
},
)
except AuthAiError as err:
if is_insufficient_credit(err):
return JSONResponse(
{"error": err.to_public_json(), "topupUrl": user.wallet.topup_url()},
status_code=402,
)
if err.code == "GW_002":
headers: dict[str, str] = {}
retry_after = err.details.get("retryAfterSeconds") if err.details else None
if isinstance(retry_after, int | float):
headers["retry-after"] = str(round(retry_after))
return JSONResponse(
{"error": err.to_public_json()}, status_code=429, headers=headers
)
return JSONResponse({"error": err.to_public_json()}, status_code=502)
choices = ChatAnswer.model_validate(answer).choices
return {"reply": (choices[0].message.content or "") if choices else ""}
The gateway maps to the SDK's errors by status: 402 is GW_001 (empty wallet), 429 is GW_002 (rate limit, with details["retryAfterSeconds"] when the gateway sends Retry-After), 401 and 403 are GW_003, 503 is GW_004. Return the public JSON to your users and keep err.message for logs.
You do not have to catch these errors yourself: an AuthAiError that escapes a route is answered by the handler install_auth registers, with the public envelope and the status of the error table (402 for GW_001, 429 for GW_002…). Catch them when you want to add something, such as the topupUrl above.
With an OpenAI-compatible SDK
gateway.create_http_client() returns an httpx2.AsyncClient that signs each request with a fresh user token, replacing whatever Authorization header the SDK set, and refuses any origin other than the gateway's so the token cannot leak. Give it to the openai package together with base_url and a placeholder key. The caller owns the client and closes it:
from openai import APIStatusError, AsyncOpenAI
from auth_ai import AiClient, is_insufficient_credit_status
async def say_hello(ai: AiClient, user_id: str, session_id: str) -> str | None:
gateway = ai.for_user(user_id).gateway
async with gateway.create_http_client() as http_client:
client = AsyncOpenAI(
base_url=gateway.openai_base_url, # <gateway_url>/v1
api_key="unused", # replaced by the signing client
http_client=http_client,
)
try:
completion = await client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": "Hello"}],
metadata={"session_id": session_id},
)
except APIStatusError as err:
if is_insufficient_credit_status(err):
return None # 402: the wallet is empty or negative. Offer a top-up.
raise
return completion.choices[0].message.content
base_url and http_client options of the openai package, and was run against a scripted gateway. Model names depend on your gateway: list them with gateway.list_models().openai package 3.x is built on httpx2 and annotates http_client as an httpx2.AsyncClient, which is the type the SDK returns: the snippet above type-checks as written.A 401 from the gateway drops the cached token, so retrying the call mints a new one. Tokens live one hour and are never stored.
is_insufficient_credit_status(err) tests for status_code or status equal to 402 on any error object, which is how OpenAI-compatible SDKs expose the HTTP status. For AuthAiErrors use is_insufficient_credit(err).
gateway.get_access_token() returns a fresh token as a string, for clients that cannot take an httpx2 client: mint it per call, since it expires after an hour.
Attribute spend to a session
Put metadata={"session_id": "<id>"} in the request body. The wallet then keeps the spend of that session, and you can read it back (for the calling application only):
from auth_ai import UserAi, format_euros
async def print_usage(user: UserAi) -> None:
usage = await user.wallet.get_session_usage("chat-7f3a", days=7)
print(usage.calls, format_euros(usage.total))
for entry in usage.by_model:
print(entry.model, entry.count, entry.total_tokens)
usage.total is an int of micro-euros; by_model lists model, amount, count and token counts per model (model is mcp for tool calls and other when the gateway reported none). Billing is asynchronous: a call may take a few seconds to appear, so poll if you need the figure right after the call. The example app's self-test does exactly that.
Models, agents and MCP servers
from auth_ai import GatewayClient
async def discover(gateway: GatewayClient) -> str:
models = await gateway.list_models() # /v1/models, only what the user's plan may call
agents = await gateway.list_agents() # /v1/agents
servers = await gateway.list_mcp_servers() # /v1/mcp/server
print(len(models), len(agents), len(servers))
url = gateway.mcp_url() # <gateway_url>/mcp
one_server = gateway.mcp_url("my-server") # <gateway_url>/my-server/mcp
print(url, one_server)
return await gateway.get_access_token() # for an MCP client's Authorization: Bearer header
mcp_url(name) rejects names outside [A-Za-z0-9][A-Za-z0-9_.-]{0,127} with CFG_001. Mint the token per connection: it expires after an hour.
Show the balance first
Check wallet.get_account() before an expensive action and use can_spend, never the raw balance: it is true for an unlimited account or a strictly positive balance. A call that finishes after the balance reached zero is still billed, so a balance can go negative until the next top-up.
Use the TypeScript client
Pair the Python backend with the browser client and the React and Vue bindings of @circle-rd/auth-ai, which are documented in the TypeScript SDK site.
Billing and consumption
From your backend, read a user's subscription, metered usage and invoices, and report consumption with idempotency keys.