Model busy; raise max_instances to let it scale further. LoRA adapters run on a shared base model and count toward that model’s limit. The one exception: if we have set a limit for a specific model on your account, that model is rate limited, private or not.
Default limit: 200 concurrent requests per model
On public models, every account has a default limit of 200 concurrent requests per model. If you query two different models simultaneously, you can handle 400 total concurrent requests (200 per model). This limit is sufficient for most production applications, including services with hundreds of thousands of daily active users.Understanding concurrent vs. requests per minute
The rate limit is on concurrent requests, not per-minute volume. Throughput depends on how long each request takes:
As requests complete, new ones can immediately take their place.
Batch jobs
For large batch jobs (e.g., embedding a knowledge base), use a token bucket algorithm to stay under 200 concurrent requests. You’ll still complete the work in a reasonable time.Rate limit errors
You’ll receive HTTP 429 with aRate limited message when the limit is exceeded.
Actions to take:
- Retry after a short delay
- Slow down your request rate
- Apply for a limit increase
You may occasionally receive 429 errors when a public model becomes very busy, even if you’re under the limit. Auto-scaling will kick in shortly. Retry after a brief wait.