Admission control

Hivenet Router applies admission checks before an LLM request enters the routing queue or reaches an inference backend. The checks reject inputs that cannot fit, shed work when the serving pool is under pressure, and limit token-weighted work already in flight. Admission control applies to:

POST /v1/chat/completions
POST /v1/messages

POST /v1/messages/count_tokens is deliberately exempt. It runs no model and consumes no KV cache, so it skips the input and image caps, pressure shed, occupancy budgets, per-key token rates, and daily token budget. It remains protected by the request-per-minute limiter. Embedding and reranking requests are also exempt because these gates model KV-cache-bound generation. Their request-rate, body-size, and agent-concurrency protections still apply.

Gate order

After authentication and the request-per-minute check, the LLM handler evaluates:

B1: request caps — input-token and image-count limits
B3: pressure shed — live pressure across healthy replicas
B2: pool occupancy — token-weighted occupancy and max_inflight
B4: serverless key limits — per-key occupancy share, output-rate state, and input tokens per minute
Daily token admission — the existing tenant token budget

If a later check rejects the request, every occupancy reservation already taken is released.

Configure a policy

Admission fields live at the top level of a global or per-model policy:

models:
  - Qwen/Qwen3.6-27B-A3B

mode: reserved
max_input_tokens: 131072
images_max: 8
admit_budget_tokens: 262144
max_inflight: 16

routing_policy:
  match:
    engine: vllm
  strategy: least-loaded

All four numeric limits are optional. 0 or an omitted field disables that check. Negative values make policy validation fail. admit_budget_tokens and max_inflight are per-replica limits that the router scales to the number of healthy replicas serving the model.

Field	Current behavior
`max_input_tokens`	Rejects a request when the learned prompt estimate exceeds the limit
`images_max`	Rejects a request carrying more images than the limit
`admit_budget_tokens`	Per-replica token-weighted occupancy capacity
`max_inflight`	Per-replica in-flight request-count backstop

One input estimate

The router computes one prompt estimate and uses it consistently for B1, occupancy admission, serverless input tokens per minute, daily input admission, and the later exact-usage correction. The estimate includes:

text in OpenAI- and Anthropic-style message content
the Anthropic top-level system prompt, as a string or text-block array
raw tools JSON, because tool schemas become part of the model prompt
a small per-message overhead, including for textless messages

The estimator starts at one token per 3.2 bytes and learns a separate tokens-per-byte ratio for each model from exact backend usage. New observations use an EWMA weight of 0.2 and are clamped to 0.05 through 1.0 tokens per byte so one abnormal report cannot poison later estimates. The learned state resets when the router restarts. OpenAI usage fields (prompt_tokens, completion_tokens) and Anthropic fields (input_tokens, output_tokens) are normalized to the same internal totals. Streaming Anthropic message_start and message_delta usage is also recognized. Image content is not byte-estimable. The router counts OpenAI image_url parts and Anthropic image blocks separately through images_max. Image-bearing requests still receive an exact occupancy true-up, but they are not used to train the text ratio.

B1: request caps

The router checks the learned input estimate and image count independently. A breach returns HTTP 400 with input_too_long:

{
  "error": {
    "code": "input_too_long",
    "message": "input is 140000 tokens, over the model limit of 131072",
    "source": "router"
  }
}

Reducing only max_tokens does not fix this response. Reduce the input text, tool definitions, or number of images. Hivenet Router does not impose its own output-token cap.

B2: pool occupancy

For each model, the router maintains a token-weighted occupancy total and an in-flight request count. The effective pool limits are:

token budget = floor(HIVENET_ROUTER_ADMIT_FRACTION
                     × admit_budget_tokens
                     × healthy_replicas)

request backstop = max_inflight × healthy_replicas

The healthy-replica count is recomputed for each request and floored at 1. This keeps the gate active when no healthy replica is currently reported, while routing can still return the more specific availability error. HIVENET_ROUTER_ADMIT_FRACTION defaults to 0.90 and accepts values greater than 0 and at most 1. Invalid values leave the default unchanged. A request’s initial token footprint is:

estimated input tokens + declared maximum output tokens

max_completion_tokens is preferred, with max_tokens as the fallback. A negative output value is treated as undeclared for occupancy accounting rather than reducing the footprint. If output is undeclared, the reservation starts with the input estimate and grows as output streams. When the backend reports exact input usage, the router replaces the estimated input portion with the exact count. Success, error, timeout, and client disconnect all release the reservation. When a request does not fit, it waits for up to HIVENET_ROUTER_ADMIT_PARK_TIMEOUT. The default is 250ms; 0 rejects immediately. If capacity remains unavailable, the router returns HTTP 429, sets Retry-After: 1, and uses concurrency_limit_exceeded:

{
  "error": {
    "code": "concurrency_limit_exceeded",
    "message": "server at capacity for this model, please retry",
    "source": "router"
  }
}

Retry this error with bounded exponential backoff and jitter.

B3: front-door shedding

Use shed_if to reject new work before queueing when the healthy serving pool is already under pressure:

shed_if:
  kv_cache_utilization:
    gt: 0.90
  waiting_requests:
    gt: 20

The router averages each available signal across healthy agents serving the requested model. If any configured threshold fails, it returns HTTP 429, sets Retry-After: 1, and uses concurrency_limit_exceeded with the message model is under high load, please retry. Missing engine signals pass the gate. A single hot replica should normally be handled by per-agent exclude_if; shed_if is for pressure across the pool.

B4: serverless key limits

mode accepts reserved and serverless; omitting it selects reserved. B1, B2, and B3 run in either mode. Serverless mode additionally enables these key fields:

api:
  mode: api-key
  keys:
    - key_hash: "<sha256>"
      key_preview: "sk-hivenet-...abcd"
      metadata:
        name: "Shared tenant"
        owner: "tenant-a"
      max_occupancy_share: 0.25
      quota:
        input_tokens_per_minute: 262144
        output_tokens_per_minute: 32768

max_occupancy_share limits the key’s in-flight footprint to:
max_occupancy_share × admit_budget_tokens × healthy_replicas
This fairness limit intentionally does not apply HIVENET_ROUTER_ADMIT_FRACTION. It rejects immediately with 429 rate_limit_exceeded; the global B2 budget remains the box-safety limit.
input_tokens_per_minute is a continuously refilling one-minute bucket charged from the shared input estimate before the daily budget.
output_tokens_per_minute is charged after a local response. If recent output drains the bucket, the next request is rejected. One response can drain at most one minute of capacity.

The key limits are fairness and anti-abuse controls and may be deliberately oversubscribed. Their configured shares do not need to add up to 1. The input and output buckets are independent for each key-and-model combination. A zero value disables the corresponding cap. B4 rejections return 429 rate_limit_exceeded with Retry-After: 1. The ordinary request-per-minute limiter uses the same error code but does not set Retry-After.

Dynamic keys have a stable key ID, so the three serverless controls are isolated per key. Static-key authentication does not currently populate a key ID; the handler falls back to anonymous. As a result, static keys share these B4 token buckets and occupancy state for a model in the audited source. Do not describe them as isolated per-static-key controls until the runtime supplies a distinct static key ID.

Validation and derived defaults

The router validates the cross-config serverless invariants at startup, during auth and policy reloads, and for dynamic PUT /admin/api-keys/{id} and POST /admin/api-keys/replace writes:

max_occupancy_share must be 0 (unset) or greater than 0 and at most 1.
input and output token-per-minute values cannot be negative.
a nonzero input_tokens_per_minute must be at least max_input_tokens for every reachable serverless model.

Invalid dynamic writes return HTTP 400; an invalid reload leaves the previous valid configuration active. The repository includes auth.DeriveKeyDefaults to calculate suggested B4 values from certified model limits. The router does not load a model’s router_limits.yaml or call this helper while loading auth.yaml. Operators or provisioning tooling must write the resulting values into the key configuration.

Provider fallback accounting

Provider fallback consumes no local KV cache or decode throughput. When routing leaves the local pool, the router releases the global and per-key occupancy reservations immediately. Request-rate, input-token-per-minute, and daily input charges taken before routing remain in place. Provider output does not charge the local OTPM bucket or train the local model’s estimator. The provider path also does not perform the normal post-response daily output deduction. Provider-reported usage is still recorded in Prometheus tenant counters and the audit log under the originally requested model.

Idle-state cleanup

Every five minutes, the router removes full rate buckets, daily buckets from a past UTC day, and occupancy state that has held no requests for at least 15 minutes. This bounds memory when dynamic keys are created frequently. Cumulative metrics and audit history are unaffected.

Policy YAML reference

Review every routing and admission field.

auth.yaml reference

Configure RPM, daily token, and serverless per-key limits.

Error codes

Handle admission, routing, and backend failures.

Configuration reference

Review admission environment variables and defaults.