CIRCLE
AI and billing

Call the LiteLLM gateway

Call LLMs and MCP tools through the gateway with the user's own token, handle empty wallets and rate limits, and attribute spend to a session.

Read Credit wallet and LiteLLM gateway first for the model. In short: the user's access token for the gateway audience is the API key, the gateway debits the wallet, and your application never handles a debit.

Prerequisites

  • LiteLLM support is on for the application in auth-service, and the plan of the user grants access to models (see the auth-service guide LiteLLM gateway and credit wallet in an application).
  • gateway_url is set in the AuthClient configuration, and an AiClient exists: AiClient(auth=auth, grants=grants).
  • The user signed in after gateway_url was set, so their grant covers the gateway and wallet audiences.

With the SDK's own calls

UserAi.gateway.post_json(path, body) posts JSON and returns the parsed answer as object; you validate it. A refusal raises an AuthAiError.

from typing import Annotated

from fastapi import Depends, FastAPI
from fastapi.responses import JSONResponse
from pydantic import BaseModel, Field

from auth_ai import AiClient, AuthAiError, is_insufficient_credit
from auth_ai.starlette import require_auth
from auth_ai.web import AuthHandlers, Session


class ChatBody(BaseModel):
    model: str = Field(min_length=1)
    message: str = Field(min_length=1, max_length=2000)


class Message(BaseModel):
    content: str | None


class Choice(BaseModel):
    message: Message


class ChatAnswer(BaseModel):
    choices: list[Choice]


def add_chat_route(app: FastAPI, handlers: AuthHandlers, ai: AiClient) -> None:
    current_session = require_auth(handlers)

    @app.post("/api/chat", response_model=None)
    async def chat(
        body: ChatBody, session: Annotated[Session, Depends(current_session)]
    ) -> dict[str, str] | JSONResponse:
        user = ai.for_user(session.user_id)
        try:
            answer = await user.gateway.post_json(
                "/v1/chat/completions",
                {
                    "model": body.model,
                    "messages": [{"role": "user", "content": body.message}],
                    "metadata": {"session_id": session.id},
                },
            )
        except AuthAiError as err:
            if is_insufficient_credit(err):
                return JSONResponse(
                    {"error": err.to_public_json(), "topupUrl": user.wallet.topup_url()},
                    status_code=402,
                )
            if err.code == "GW_002":
                headers: dict[str, str] = {}
                retry_after = err.details.get("retryAfterSeconds") if err.details else None
                if isinstance(retry_after, int | float):
                    headers["retry-after"] = str(round(retry_after))
                return JSONResponse(
                    {"error": err.to_public_json()}, status_code=429, headers=headers
                )
            return JSONResponse({"error": err.to_public_json()}, status_code=502)
        choices = ChatAnswer.model_validate(answer).choices
        return {"reply": (choices[0].message.content or "") if choices else ""}

The gateway maps to the SDK's errors by status: 402 is GW_001 (empty wallet), 429 is GW_002 (rate limit, with details["retryAfterSeconds"] when the gateway sends Retry-After), 401 and 403 are GW_003, 503 is GW_004. Return the public JSON to your users and keep err.message for logs.

You do not have to catch these errors yourself: an AuthAiError that escapes a route is answered by the handler install_auth registers, with the public envelope and the status of the error table (402 for GW_001, 429 for GW_002…). Catch them when you want to add something, such as the topupUrl above.

With an OpenAI-compatible SDK

gateway.create_http_client() returns an httpx2.AsyncClient that signs each request with a fresh user token, replacing whatever Authorization header the SDK set, and refuses any origin other than the gateway's so the token cannot leak. Give it to the openai package together with base_url and a placeholder key. The caller owns the client and closes it:

from openai import APIStatusError, AsyncOpenAI

from auth_ai import AiClient, is_insufficient_credit_status


async def say_hello(ai: AiClient, user_id: str, session_id: str) -> str | None:
    gateway = ai.for_user(user_id).gateway

    async with gateway.create_http_client() as http_client:
        client = AsyncOpenAI(
            base_url=gateway.openai_base_url,  # <gateway_url>/v1
            api_key="unused",  # replaced by the signing client
            http_client=http_client,
        )
        try:
            completion = await client.chat.completions.create(
                model="gpt-4o-mini",
                messages=[{"role": "user", "content": "Hello"}],
                metadata={"session_id": session_id},
            )
        except APIStatusError as err:
            if is_insufficient_credit_status(err):
                return None  # 402: the wallet is empty or negative. Offer a top-up.
            raise
        return completion.choices[0].message.content
This snippet relies on the documented SDK API and on the base_url and http_client options of the openai package, and was run against a scripted gateway. Model names depend on your gateway: list them with gateway.list_models().
The openai package 3.x is built on httpx2 and annotates http_client as an httpx2.AsyncClient, which is the type the SDK returns: the snippet above type-checks as written.

A 401 from the gateway drops the cached token, so retrying the call mints a new one. Tokens live one hour and are never stored.

is_insufficient_credit_status(err) tests for status_code or status equal to 402 on any error object, which is how OpenAI-compatible SDKs expose the HTTP status. For AuthAiErrors use is_insufficient_credit(err).

gateway.get_access_token() returns a fresh token as a string, for clients that cannot take an httpx2 client: mint it per call, since it expires after an hour.

Attribute spend to a session

Put metadata={"session_id": "<id>"} in the request body. The wallet then keeps the spend of that session, and you can read it back (for the calling application only):

from auth_ai import UserAi, format_euros


async def print_usage(user: UserAi) -> None:
    usage = await user.wallet.get_session_usage("chat-7f3a", days=7)
    print(usage.calls, format_euros(usage.total))
    for entry in usage.by_model:
        print(entry.model, entry.count, entry.total_tokens)

usage.total is an int of micro-euros; by_model lists model, amount, count and token counts per model (model is mcp for tool calls and other when the gateway reported none). Billing is asynchronous: a call may take a few seconds to appear, so poll if you need the figure right after the call. The example app's self-test does exactly that.

Models, agents and MCP servers

from auth_ai import GatewayClient


async def discover(gateway: GatewayClient) -> str:
    models = await gateway.list_models()  # /v1/models, only what the user's plan may call
    agents = await gateway.list_agents()  # /v1/agents
    servers = await gateway.list_mcp_servers()  # /v1/mcp/server
    print(len(models), len(agents), len(servers))

    url = gateway.mcp_url()  # <gateway_url>/mcp
    one_server = gateway.mcp_url("my-server")  # <gateway_url>/my-server/mcp
    print(url, one_server)
    return await gateway.get_access_token()  # for an MCP client's Authorization: Bearer header

mcp_url(name) rejects names outside [A-Za-z0-9][A-Za-z0-9_.-]{0,127} with CFG_001. Mint the token per connection: it expires after an hour.

Show the balance first

Check wallet.get_account() before an expensive action and use can_spend, never the raw balance: it is true for an unlimited account or a strictly positive balance. A call that finishes after the balance reached zero is still billed, so a balance can go negative until the next top-up.

Copyright © 2026