Rate limits

The OpenAI-compatible API is rate limited per client IP at 60 requests per minute on a rolling window.

Exceeding it returns:

HTTP/1.1 429 Too Many Requests
Retry-After: 12

{ "error": { "message": "Rate limit exceeded", "type": "rate_limit_error" } }

Handling 429#

1. Read Retry-After — it is in seconds.
2. Wait at least that long.
3. Retry with exponential backoff and jitter if it happens again.

Never retry a 429 immediately in a tight loop. It extends the window and makes the stall longer for every request behind it.

Staying under the limit#

• Queue, don't fan out. A worker pool with a fixed concurrency is far more predictable than firing every request at once.
• Batch on your side. Combining ten short prompts into one request is usually cheaper as well as fewer requests.
• Cache. Identical inputs producing identical outputs should not hit the API twice — the key/value store is a reasonable cache for per-user results.
• Stream long generations. Streaming doesn't reduce request count, but it does stop clients from re-issuing a request they think has hung.

Note on shared IPs#

The limit is keyed on the source IP. A server-side integration behind one egress IP shares one budget across all of its users, so a busy backend should implement its own queue rather than relying on the ceiling being per-user.

Separately from rate limits, a request is rejected if the paying account's credit balance can't cover it. See Errors.

Error-reporting quotas#

POST /errors/report has separate quotas so a broken client cannot flood the diagnostic inbox. The default ceilings are 10 reports per minute and 100 per day for one user and app, 200 per day for one user across apps, 300 per minute and 10,000 per day for one app, and a 16 KB report payload limit. The endpoint returns 429 with Retry-After when a limit is reached. Reports are also deduplicated by (app, user, event_id) and retained for seven days.