Work / AI products
Multi-provider LLM gateway.
A provider-agnostic gateway routing requests across Gemini, Claude, and OpenAI with rate limiting and fallbacks, so switching models never needed a code change.
- Stack
- TypeScript, Node.js, Redis
- Scale
- Every LLM call on the platform
- Role
- Designed and built
Flow
- Service call
- Unified request
- Policy: model & budget
- Rate limiter
- Provider adapter
- Normalized response
- Metrics
How it works
Application code speaks one interface: a task type, a prompt, and options. The gateway maps that to a provider and model based on policy, so a feature never hard-codes which vendor it uses.
Per-tenant and per-provider rate limits live in Redis. When a provider is throttled or down, requests fail over to the next configured model, and the caller sees the same response shape either way.
Every call is logged with tokens, latency, and cost, which is what made it possible to move traffic between providers on evidence rather than habit.
Hard parts
- Normalizing streaming, tool calls, and error semantics across three vendors that disagree on all of them.
- Keeping fallbacks safe for features with strict output formats.
Outcome
Model changes became configuration. Provider incidents stopped being product incidents.