> ## Documentation Index > Fetch the complete documentation index at: https://routerdocs.hivenet.com/llms.txt > Use this file to discover all available pages before exploring further. # Admission control > Configure request caps, pool pressure shedding, token-weighted occupancy, and serverless per-key limits. Hivenet Router applies admission checks before an LLM request enters the routing queue or reaches an inference backend. The checks reject inputs that cannot fit, shed work when the serving pool is under pressure, and limit token-weighted work already in flight. Admission control applies to: * `POST /v1/chat/completions` * `POST /v1/messages` `POST /v1/messages/count_tokens` is deliberately exempt. It runs no model and consumes no KV cache, so it skips the input and image caps, pressure shed, occupancy budgets, per-key token rates, and daily token budget. It remains protected by the request-per-minute limiter. Embedding and reranking requests are also exempt because these gates model KV-cache-bound generation. Their request-rate, body-size, and agent-concurrency protections still apply. ## Gate order After authentication and the request-per-minute check, the LLM handler evaluates: 1. **B1: request caps** — input-token and image-count limits 2. **B3: pressure shed** — live pressure across healthy replicas 3. **B2: pool occupancy** — token-weighted occupancy and `max_inflight` 4. **B4: serverless key limits** — per-key occupancy share, output-rate state, and input tokens per minute 5. **Daily token admission** — the existing tenant token budget If a later check rejects the request, every occupancy reservation already taken is released. ## Configure a policy Admission fields live at the top level of a global or per-model policy: ```yaml theme={null} models: - Qwen/Qwen3.6-27B-A3B mode: reserved max_input_tokens: 131072 images_max: 8 admit_budget_tokens: 262144 max_inflight: 16 routing_policy: match: engine: vllm strategy: least-loaded ``` All four numeric limits are optional. `0` or an omitted field disables that check. Negative values make policy validation fail. `admit_budget_tokens` and `max_inflight` are **per-replica** limits that the router scales to the number of healthy replicas serving the model. | Field | Current behavior | | --------------------- | -------------------------------------------------------------------- | | `max_input_tokens` | Rejects a request when the learned prompt estimate exceeds the limit | | `images_max` | Rejects a request carrying more images than the limit | | `admit_budget_tokens` | Per-replica token-weighted occupancy capacity | | `max_inflight` | Per-replica in-flight request-count backstop | ## One input estimate The router computes one prompt estimate and uses it consistently for B1, occupancy admission, serverless input tokens per minute, daily input admission, and the later exact-usage correction. The estimate includes: * text in OpenAI- and Anthropic-style message content * the Anthropic top-level `system` prompt, as a string or text-block array * raw `tools` JSON, because tool schemas become part of the model prompt * a small per-message overhead, including for textless messages The estimator starts at one token per 3.2 bytes and learns a separate tokens-per-byte ratio for each model from exact backend usage. New observations use an EWMA weight of `0.2` and are clamped to `0.05` through `1.0` tokens per byte so one abnormal report cannot poison later estimates. The learned state resets when the router restarts. OpenAI usage fields (`prompt_tokens`, `completion_tokens`) and Anthropic fields (`input_tokens`, `output_tokens`) are normalized to the same internal totals. Streaming Anthropic `message_start` and `message_delta` usage is also recognized. Image content is not byte-estimable. The router counts OpenAI `image_url` parts and Anthropic `image` blocks separately through `images_max`. Image-bearing requests still receive an exact occupancy true-up, but they are not used to train the text ratio. ## B1: request caps The router checks the learned input estimate and image count independently. A breach returns HTTP `400` with `input_too_long`: ```json theme={null} { "error": { "code": "input_too_long", "message": "input is 140000 tokens, over the model limit of 131072", "source": "router" } } ``` Reducing only `max_tokens` does not fix this response. Reduce the input text, tool definitions, or number of images. Hivenet Router does not impose its own output-token cap. ## B2: pool occupancy For each model, the router maintains a token-weighted occupancy total and an in-flight request count. The effective pool limits are: ```text theme={null} token budget = floor(HIVENET_ROUTER_ADMIT_FRACTION × admit_budget_tokens × healthy_replicas) request backstop = max_inflight × healthy_replicas ``` The healthy-replica count is recomputed for each request and floored at `1`. This keeps the gate active when no healthy replica is currently reported, while routing can still return the more specific availability error. `HIVENET_ROUTER_ADMIT_FRACTION` defaults to `0.90` and accepts values greater than `0` and at most `1`. Invalid values leave the default unchanged. A request's initial token footprint is: ```text theme={null} estimated input tokens + declared maximum output tokens ``` `max_completion_tokens` is preferred, with `max_tokens` as the fallback. A negative output value is treated as undeclared for occupancy accounting rather than reducing the footprint. If output is undeclared, the reservation starts with the input estimate and grows as output streams. When the backend reports exact input usage, the router replaces the estimated input portion with the exact count. Success, error, timeout, and client disconnect all release the reservation. When a request does not fit, it waits for up to `HIVENET_ROUTER_ADMIT_PARK_TIMEOUT`. The default is `250ms`; `0` rejects immediately. If capacity remains unavailable, the router returns HTTP `429`, sets `Retry-After: 1`, and uses `concurrency_limit_exceeded`: ```json theme={null} { "error": { "code": "concurrency_limit_exceeded", "message": "server at capacity for this model, please retry", "source": "router" } } ``` Retry this error with bounded exponential backoff and jitter. ## B3: front-door shedding Use `shed_if` to reject new work before queueing when the healthy serving pool is already under pressure: ```yaml theme={null} shed_if: kv_cache_utilization: gt: 0.90 waiting_requests: gt: 20 ``` The router averages each available signal across healthy agents serving the requested model. If any configured threshold fails, it returns HTTP `429`, sets `Retry-After: 1`, and uses `concurrency_limit_exceeded` with the message `model is under high load, please retry`. Missing engine signals pass the gate. A single hot replica should normally be handled by per-agent `exclude_if`; `shed_if` is for pressure across the pool. ## B4: serverless key limits `mode` accepts `reserved` and `serverless`; omitting it selects `reserved`. B1, B2, and B3 run in either mode. Serverless mode additionally enables these key fields: ```yaml theme={null} api: mode: api-key keys: - key_hash: "" key_preview: "sk-hivenet-...abcd" metadata: name: "Shared tenant" owner: "tenant-a" max_occupancy_share: 0.25 quota: input_tokens_per_minute: 262144 output_tokens_per_minute: 32768 ``` * `max_occupancy_share` limits the key's in-flight footprint to: ```text theme={null} max_occupancy_share × admit_budget_tokens × healthy_replicas ``` This fairness limit intentionally does **not** apply `HIVENET_ROUTER_ADMIT_FRACTION`. It rejects immediately with `429 rate_limit_exceeded`; the global B2 budget remains the box-safety limit. * `input_tokens_per_minute` is a continuously refilling one-minute bucket charged from the shared input estimate before the daily budget. * `output_tokens_per_minute` is charged after a local response. If recent output drains the bucket, the next request is rejected. One response can drain at most one minute of capacity. The key limits are fairness and anti-abuse controls and may be deliberately oversubscribed. Their configured shares do not need to add up to `1`. The input and output buckets are independent for each key-and-model combination. A zero value disables the corresponding cap. B4 rejections return `429 rate_limit_exceeded` with `Retry-After: 1`. The ordinary request-per-minute limiter uses the same error code but does not set `Retry-After`. Dynamic keys have a stable key ID, so the three serverless controls are isolated per key. Static-key authentication does not currently populate a key ID; the handler falls back to `anonymous`. As a result, static keys share these B4 token buckets and occupancy state for a model in the audited source. Do not describe them as isolated per-static-key controls until the runtime supplies a distinct static key ID. ## Validation and derived defaults The router validates the cross-config serverless invariants at startup, during auth and policy reloads, and for dynamic `PUT /admin/api-keys/{id}` and `POST /admin/api-keys/replace` writes: * `max_occupancy_share` must be `0` (unset) or greater than `0` and at most `1`. * input and output token-per-minute values cannot be negative. * a nonzero `input_tokens_per_minute` must be at least `max_input_tokens` for every reachable serverless model. Invalid dynamic writes return HTTP `400`; an invalid reload leaves the previous valid configuration active. The repository includes `auth.DeriveKeyDefaults` to calculate suggested B4 values from certified model limits. The router does not load a model's `router_limits.yaml` or call this helper while loading `auth.yaml`. Operators or provisioning tooling must write the resulting values into the key configuration. ## Provider fallback accounting Provider fallback consumes no local KV cache or decode throughput. When routing leaves the local pool, the router releases the global and per-key occupancy reservations immediately. Request-rate, input-token-per-minute, and daily input charges taken before routing remain in place. Provider output does not charge the local OTPM bucket or train the local model's estimator. The provider path also does not perform the normal post-response daily output deduction. Provider-reported usage is still recorded in Prometheus tenant counters and the audit log under the originally requested model. ## Idle-state cleanup Every five minutes, the router removes full rate buckets, daily buckets from a past UTC day, and occupancy state that has held no requests for at least 15 minutes. This bounds memory when dynamic keys are created frequently. Cumulative metrics and audit history are unaffected. ## Related pages Review every routing and admission field. Configure RPM, daily token, and serverless per-key limits. Handle admission, routing, and backend failures. Review admission environment variables and defaults. Monitor occupancy, budgets, concurrency, and rejections by gate.