LiteLLM gateway & shared credit wallet — programme plan

Living document: update the status columns, the decision log and the change log in the same commit as the work they describe. Status legend: ✅ done · 🟡 in progress · ⬜ planned · 🚫 dropped · ❓ blocked on a decision.

1. Goal

All CIRCLE applications reach LLMs and MCP servers through a single LiteLLM proxy, authenticated with auth-service access tokens. Users buy prepaid credits (bought with Stripe when it is configured, or recorded by a superadmin otherwise) that are shared by every application and consumed per LLM token and/or per MCP tool call.

2. Target architecture

Apps ──OAuth (resource=<litellm url>)──► auth-service (IdP + wallet + Stripe)
  │                                          ▲   │ JWKS
  │ Bearer JWT                               │   ▼
  └──────────────► LiteLLM proxy ── hooks ───┘ (balance check, idempotent debits)
                        │
                        └──► LLM providers / MCP servers

• auth-service is the source of truth for money (ledger, Stripe, the USD→EUR rate and the margin; model and tool prices stay in LiteLLM, D36).
• LiteLLM is the metering and enforcement point: it validates the JWT, computes the raw cost, and calls auth-service through a custom callback (option B): async_pre_call_hook (balance check, short cache) and async_log_success_event (debit with an idempotency key generated for each debit, D32).
• Credit unit is monetary (integer micro-units), never tokens, so tokens and MCP calls share one balance. Raw quantities are stored in ledger metadata.
• LiteLLM is registered in auth-service as a protected resource (RFC 8707 audience). Public (PKCE) and private (confidential) applications are both supported; the caller is identified by azp, the user by sub.

3. Programme overview

A lot is done only when pnpm build:server, pnpm lint, pnpm format:check, pnpm test and pnpm test:integration pass (plus the lot-specific checks below).

4. Lot details

Lot 1 — LiteLLM stack (local first, then dev server)

Exit criteria: a user token issued by auth-service is accepted by LiteLLM and a wrong-audience, tampered or client_credentials token is rejected, on local and dev. The spike script (/authorize PKCE flow → LiteLLM) becomes the reusable smoke test (1.7).

Lot 2 — Wallet data model and core service

Lot 3 — Wallet APIs

Lot 4 — Stripe top-ups

Lot 5 — LiteLLM callback (option B)

Lot 6 — Pricing

Lot 7 — Frontend

Lot 8 — Hardening and documentation

5. Decision log

6. Open questions

6b. Spike findings (lot 1.1, local, 2026-10-09)

Setup: throwaway Postgres, auth-service on the host, ghcr.io/berriai/litellm:v1.104.2 (latest and main-stable both resolve to it today), mocked model.

• Native JWT auth is Enterprise-only → D10. custom_auth works without a licence.
• Access tokens: a JWT is issued only when resource=<litellm url> is sent, otherwise the token is opaque. Algorithm EdDSA (Ed25519), kid set, iss = BETTER_AUTH_URL (no path), aud = [<litellm url>, <userinfo url>], sub = user id, azp = application slug, sid present on user tokens.
• client_credentials tokens: aud is a single string and sub == azp == client_id (no user, no sid) → rejected by comparing sub and azp.
• Resource access: a client must be linked to the resource, otherwise /authorize returns invalid_target (D12). replaceClientResource deletes every link of a client on create/update and only re-links the client's own URL, so a manual link is wiped by the next application update.
• Validated with custom_auth.py: public (PKCE) and private application user tokens → 200; master key → 200; M2M, wrong audience, tampered signature, garbage and missing token → 401. Spend logs record user = AuthService sub.
• Observations: rejected requests are also written to the spend logs as failure rows (noise for reconciliation, filter on status); the image ships pyjwt and cryptography (EdDSA supported); custom_auth_run_common_checks must be set explicitly if DB budgets/model allowlists should apply.
• Role checks bypassed (found on 2026-10-11): with custom_auth and no role, LiteLLM skipped its route authorisation: a plain user token got 200 on /key/generate, /user/new and /team/new (only /model/new and /config/update were refused by their own handlers). Fixed by D21.
• Not yet verified: key rotation behaviour of the refetch logic, MCP hook coverage (5.5), behaviour behind Traefik/TLS on the dev server.
• Full stack check (compose, production-mode image): pnpm llm:setup then pnpm llm:smoke pass 11/11 on a throwaway project. /authorize answers with a JSON { redirect, url } body instead of a 302 in production mode (the smoke client handles both). docker compose interpolates services of inactive profiles, so required LiteLLM variables are enforced by custom_auth.py at start-up instead of ${VAR:?}.

6c. Side fixes found while building lot 3

Both bugs pre-dated this programme and broke the contract of the new API, so they were fixed in the same change (each covered by tests):

• Error contract. The global error handler was registered after the route plugins, so it never applied to them: a thrown ApiError came back as Fastify's default { statusCode, code, error, message } (no details) and a ZodError as a 500. The handler now lives in src/error-handler.ts, is registered before the routes and is shared with the unit-test app. Framework 4xx errors (rate limit, malformed JSON) keep their default serialisation. One existing test had encoded the old shape and was updated.
• Opaque access tokens. BetterAuth stores access tokens as SHA-256 hashes but verifyBearerAccessToken looked them up in clear, so every opaque token (a client_credentials request without resource) was refused, including on /api/consumption. The lookup now hashes the token and refuses revoked rows.

7. Risks

• auth-service on the critical path of every LLM call: cached balance, explicit failure policy, wallet code isolated in services/wallet/ so it can be extracted.
• custom_auth is our code on the hot path: it relies on the UserAPIKeyAuth contract of LiteLLM, which can change between versions (pinned image + smoke test). JWKS keys are cached and refetched at most once per cooldown when an unknown kid appears, so a rotation is picked up fast.
• LiteLLM upgrades can change hook behaviour: pin the image, re-run the smoke test and the end-to-end test on every bump.
• Overspend window between the pre-check and the post-call debit: usage is always recorded and may take the balance below zero (D8, D18), so the overspend becomes a debt owed by the user; it is bounded by the per-user ceilings (8.2, D33). A per-request cost cap is not possible before the call.
• Double charge or lost debit: idempotency keys, outbox and reconciliation.

8. Change log

| 2026-10-11 | Lot 6 closed: exchange rate and margin editable in the admin interface (D36); wallet_pricing_settings inlined in the 0000 baseline (pre-production exception: recreate the auth database) | | 2026-10-11 | LiteLLM trusted_proxy_ranges configured (D37); stale admin UI session after a database reset diagnosed (not a deployment fault) | | 2026-10-11 | Admin UI sign-in through auth-service delivered (D38): litellm-ui application, role mapping, PKCE, auto-redirect; litellm-setup provisions it | | 2026-10-12 | Master-key login of the LiteLLM interface closed while SSO is on; isSystem applications (D39); SCIM assessed and deferred (D40) | | 2026-10-27 | Migration to one gateway team per (application × plan) with a per-plan Access Group (D42): plan-aware team provisioning and reconciliation, plan resolution endpoint on the wallet:read scope in custom_auth.py, JSON models/access-groups export and import referencing LLM Credentials by name (export redacts every known provider-secret field and the output is gitignored). Verified the Access Group mechanism against LiteLLM v1.106.2 | | 2026-10-27 | Wallet consumption aggregated by day (by application) and by model (D43): byModel and daily in the stats, GET /api/user/wallet/usage per chat session, sessionId forwarded by the gateway, per-dimension chart palettes and a per-day model detail; the raw ledger is now an administrator, on-demand view |