Limits and budgets
The stacked layers of rate limits, jobs at once, storage and budgets, how they combine, and what callers see when one is reached.
Open Model Gateway enforces limits in layers. Every layer that applies is checked before a request is sent, and the strictest one decides. Platform Admins set the platform layers in Admin › Settings › Defaults & limits and on each Team's or Project's page.
The layers
| Layer | Set by | Applies to |
|---|---|---|
| Installation ceiling | Platform Admins | All workspaces together: one shared ceiling. |
| Personal / Team / Project default | Platform Admins | Each workspace of that type, separately. |
| Platform override | Platform Admins, on a Team's or Project's page | One workspace, instead of its type default. |
| Workspace | The workspace's owners and admins | One Team or Project, tighten only. |
| Key | Whoever creates or manages the key | One key and its rotations, tighten only. |
Type defaults apply to each workspace: a Team default of 2,000 requests per minute gives every Team without an override 2,000 requests per minute, not 2,000 shared between them. The installation ceiling is the one that is shared.
An override replaces the type default completely, including any field you leave blank. To go back, reset the override.
What each layer can limit
| Limit | Installation | Type default / override | Workspace | Key |
|---|---|---|---|---|
| Requests per minute | ✓ | ✓ | ✓ | ✓ |
| Tokens per minute | ✓ | ✓ | ✓ | ✓ |
| Requests at once | ✓ | ✓ | ✓ | ✓ |
| Jobs at once | ✓ | ✓ (2 by default) | ✓ | ✓ |
| Storage | ✓ (1 GiB by default) | ✓ | ||
| Budgets: day, week, month, lifetime | ✓ | ✓ | ✓ | ✓ |
- Per-minute limits use fixed UTC minutes.
- Requests at once counts requests in flight.
- Jobs at once counts active batch and video jobs. Jobs don't count toward per-minute limits, and a job holds a "requests at once" slot only while it is being created.
- Storage is the most a workspace may keep in the file store, checked while files upload.
- Budgets: each layer can have up to one budget per period. Days and months are UTC calendar days and months, weeks are ISO weeks from Monday 00:00 UTC, and lifetime is everything since the scope was created. Every budget at every layer applies, so a $10 daily budget under a $200 monthly one stops at whichever runs out first.
How layers combine
- Every applicable limit applies. A request must fit all of them.
- Blank means inherit. A layer without a limit doesn't remove its parent's.
- Lower layers only tighten. A workspace or key limit can't be above its parent's for the same thing (a key's daily budget is compared with daily budgets), and once saved it can be lowered but not raised or removed. Different periods are independent.
- Child limits don't reserve anything. A Team limited to 500 requests per minute doesn't set aside 500 of the installation's capacity.
- Raising a limit never resets spend. Past requests stay in the window they were admitted in.
- Personal workspaces take their limits from the Personal default, an override and the installation. Their owners can still limit each key.
How budgets are checked
A budget's "used" amount is the settled cost of requests admitted in its window, plus holds for requests still in progress or whose cost is unknown. Before each request the gateway adds the request's own hold, the most it could cost, and refuses it if any budget would go over.
- Requests that can't be bounded (no price, or a meter with no maximum) can't run under a budget at all.
- Unknown, unbounded costs block budgets. If requests in a budget's current window have unknown costs with no upper bound, new budgeted requests are refused with
unresolved_usageuntil those are resolved. - Budget checks read maintained totals, so their cost doesn't grow with history.
What callers see
| Code (HTTP 429) | Cause | Retry? |
|---|---|---|
rate_limit_error | A per-minute or at-once limit, or the gateway's own capacity. | Yes, later |
job_limit_exceeded | A "jobs at once" limit. | Yes, when a job ends |
token_reservation_exceeds_limit | The model's input plus output ceiling alone is larger than a tokens-per-minute limit. | No: change the ceiling or the limit |
budget_exceeded | A budget would be exceeded in its current window. | No (x-should-retry: false) |
unresolved_usage | Unknown, unbounded costs block budgeted requests. | No (x-should-retry: false) |
The message names the scope (API key, workspace or installation) but never an amount or how much is left. When several layers refuse at once, the narrowest scope is reported. The installation's ceiling always reports budget_exceeded, so it never reveals another workspace's activity.
Installation budgets
Overview and Admin's Usage & costs show each installation budget with its window, settled amount, amount on hold and whether it is exhausted. Use them as a backstop on total spend, on top of workspace budgets.
Alerts
Limits block; alerts warn. Add budget threshold alerts so people hear before a budget runs out.