Rate limits

The OpenAI-compatible API is rate limited per client IP at 60 requests per minute on a rolling window.

Exceeding it returns:

HTTP/1.1 429 Too Many Requests
Retry-After: 12

{ "error": { "message": "Rate limit exceeded", "type": "rate_limit_error" } }

Handling 429#

1. Read Retry-After — it is in seconds.
2. Wait at least that long.
3. Retry with exponential backoff and jitter if it happens again.

Never retry a 429 immediately in a tight loop. It extends the window and makes the stall longer for every request behind it.

Staying under the limit#

• Queue, don't fan out. A worker pool with a fixed concurrency is far more predictable than firing every request at once.
• Batch on your side. Combining ten short prompts into one request is usually cheaper as well as fewer requests.
• Cache. Identical inputs producing identical outputs should not hit the API twice — the key/value store is a reasonable cache for per-user results.
• Stream long generations. Streaming doesn't reduce request count, but it does stop clients from re-issuing a request they think has hung.

Note on shared IPs#

The limit is keyed on the source IP. A server-side integration behind one egress IP shares one budget across all of its users, so a busy backend should implement its own queue rather than relying on the ceiling being per-user.

Separately from rate limits, a request is rejected if the paying account's credit balance can't cover it. See Errors.