LLM gateway

What you'll build

Step-by-step build

1. 1

   Define one entry contract

   Expose a single OpenAI-compatible chat endpoint so any existing client library can point at your gateway with just a base-URL change. Nail this request/response shape first; everything else plugs in behind it. This one contract is what makes the gateway a drop-in.

2. 2

   Add provider adapters

   Write a thin adapter per provider that translates your common request into that provider's API and normalizes the response back. Start with two providers so you have something real to route between and fall back to. Keep credentials in environment variables, never in code.

3. 3

   Build the router

   Add rules that choose a provider based on the requested model, a cost preference, or a simple task hint. Make the routing table config, not hard-coded branches, so you can change policy without a redeploy. Log which provider each request was sent to.

4. 4

   Add retries and fallback

   On a timeout, 429, or 5xx, retry the same provider with exponential backoff, then fall back to the backup provider if it still fails. Cap total attempts so a request can never hang forever. This failover behaviour is the core value of the gateway.

5. 5

   Cache responses

   Hash the full request (model plus messages plus parameters) and store the response in Redis with a TTL. On a cache hit, return instantly and skip the provider call entirely. Make caching opt-out per request so non-deterministic calls can bypass it.

6. 6

   Track cost and tokens

   For every call, record the provider, model, prompt and completion tokens, computed cost, latency, and outcome to Postgres or Atlas. Do this even on cache hits and failures so the log is complete. This table is what makes spend visible.

7. 7

   Add keys and rate limits

   Issue per-client API keys and enforce request or token quotas using Redis counters. Return clear 429s with a retry-after header when a client is over limit. This keeps one noisy consumer from draining a shared budget.

8. 8

   Deploy and document

   Ship the gateway, point a real app at it by changing only the base URL, and confirm failover by killing the primary provider. Write a README with the endpoint, the routing config format, and a screenshot of the cost log. Show the failover working, because that is the headline feature.

Done when

• ✓An existing OpenAI client works against your gateway with only a base-URL change.
• ✓When you disable the primary provider, requests still succeed via the fallback.
• ✓A repeated identical request is served from cache without hitting any provider.
• ✓The cost log has a row for every call with tokens, cost, latency, and outcome.

Ship it

Run the gateway as a NestJS or FastAPI service on the AWS free tier, with Redis for cache and rate limits and Postgres or Atlas for the call log. Point at least one real app at it by changing only the base URL. Put the endpoint, the routing config format, and a screenshot of the cost log in the README, and describe how failover was tested.

What it proves: You can build the reliability layer every production LLM app needs: routing, retries, fallback, caching, and cost control behind one endpoint. That is core AI infrastructure work.

Paste this into Cursor, Claude, or ChatGPT and build it step by step.

You are my senior AI engineer pair. Help me build "LLM gateway" step by step. Stack: NestJS or FastAPI for the proxy, Groq free tier as the primary provider and a second free provider as the fallback, Redis for the response cache and rate-limit counters, and Postgres or MongoDB Atlas (free M0) for the durable call log.

Requirements:
1. One OpenAI-compatible endpoint so any client works with just a base-URL change.
2. A thin adapter per provider that normalizes requests and responses.
3. A config-driven router that picks a provider by model, cost, or task hint.
4. Retry with exponential backoff, then fallback to a backup provider (fallback = automatically switch providers when the primary keeps failing).
5. A request-hash response cache in Redis and a per-call log of tokens, cost, latency, and outcome, plus per-key rate limiting.

Work in this order: the endpoint contract, provider adapters, router, retries and fallback, caching, cost logging, then rate limits. STOP after each step so I can test. Do not write the whole app at once.