Hold at zero

Hold at zero lets a model scale down to zero replicas without failing the requests that arrive while it is asleep. When a request asks for a model that the sibling map places on this site with scale_from_zero: true, and no healthy agent serves it here, Hivenet Router holds the request instead of failing it. It counts the request in the hivenet_pending_demand metric, keeps the connection alive, and serves the request as soon as an agent for the model registers. Hivenet Router does not start or stop models. An external autoscaler, such as KEDA, reads hivenet_pending_demand and scales the model from zero. The agent registers once its backend answers its health check, and the router releases the held requests. Hold at zero needs cross-site redirect: the sibling map says which models may be held on this site. With no --site-id, or for a model the map does not place here with scale_from_zero: true, nothing is held and the router behaves as before.

The sibling map is held in memory. A restarted router holds nothing until the map writer pushes the map again, and requests for a model at zero fail as before in the meantime.

How a request is handled

After the API key is authenticated, the model alias is resolved and the model is checked against the key’s allowlist:

Served here. If a healthy agent on this site serves the model, the request is served here as usual.
Redirect first. If the sibling map lists another site in the request’s scope that serves the model, the router answers 307 to it, following the rules in Cross-site redirect. The one exception: while the map’s hints are fresh, every sibling in the caller’s home set (the two sites a key is redirected to for a model) is cold and the model may be held here (step 3), the request is not redirected, because a cold sibling is no faster than waking the model here. A warm sibling outside the home set does not prevent this. The request then continues to step 3, as it does when no sibling is in scope or the request has already been redirected twice.
Held here? If the map places the model on this site, for the endpoint’s kind, with scale_from_zero: true, the request is held. Otherwise it fails as before, normally with 404 model_not_found.
Quota. The key’s quota is checked before the hold, so a key over its limits is refused at once. See Quota while held.
Hold. The request waits, counted once in hivenet_pending_demand{model}. After --wake-keepalive, the client receives keepalive bytes.
Wake. The autoscaler starts the model. When an agent for the model registers, every held request is released and continues as a normal request: admission, routing and the backend call. It stays counted in hivenet_pending_demand until it gets a slot on an agent, and keeps receiving keepalives until its response starts. See After the wake.
Timeout. If the model does not wake within the hold budget, the request gets 503 model_waking with Retry-After.

Every inference endpoint can be held, by the placement kind that matches it: chat for POST /v1/chat/completions and POST /v1/messages; embedding for POST /v1/embeddings; rerank for POST /v1/rerank. POST /v1/messages/count_tokens is never held: a client sends it just before the real request, which is held and wakes the model. Holding both would count each prompt twice in the pending demand. A request whose key may not use the model is never held: it gets 403 model_forbidden at once.

What the client sees

A held request looks like a slow first response. The client needs no changes, but its request timeout must be longer than the hold budget plus the generation time.

Outcome	Before keepalives start	After keepalives start
The model wakes and the request succeeds	The normal response	`200`, keepalive bytes, then the normal response
The hold budget runs out	`503 model_waking` with `Retry-After`	`200`, keepalive bytes, then the error in the body
A hold limit is reached	`503 model_waking` with `Retry-After`, at once	Not applicable: the request is refused before it is held
An error after the release, such as a routing failure	The normal error status	`200`, keepalive bytes, then the error in the body

Keepalive

A held connection carries no data, so proxies and load balancers with an idle timeout may close it, and client SDKs may then retry. After --wake-keepalive (default 15s), and then at the same interval, the router sends keepalive bytes. They go on after the wake while the released request waits for capacity, with no longer gap at the release, and stop when the response starts:

an SSE comment, : keep-alive, to a streaming request ("stream": true);
a single space to any other request. Leading whitespace is valid before a JSON body.

The first keepalive sends the 200 status and the response headers, so the status can no longer change. An error after it is written in the body:

for a streaming request, as a final SSE event. OpenAI-compatible endpoints get data: {"error":{...}}. /v1/messages gets an Anthropic error event whose message starts with the error code.
for any other request, as the usual JSON error envelope after the spaces.

The audit log and the HTTP metrics record the status the request ended with, such as 503, not the 200 sent on the wire.

A non-streaming client that receives 200 parses the body as a success. With keepalives on, a hold that expires reaches such a client as a 200 whose body is an error envelope. Set --wake-keepalive 0 to keep plain status codes when no proxy between your clients and the router closes idle connections within the hold budget.

`model_waking`

HTTP 503
Retry-After: 30

{
  "error": {
    "code": "model_waking",
    "message": "model llama-3-70b is waking up; retry in 30 seconds",
    "source": "router",
    "retry_after": 30
  }
}

The code is the same in every case; the message says which one it is:

Case	Message
The hold budget runs out	`model <model> is waking up; retry in 30 seconds`
A hold limit is reached	`model <model> is waking up and too many requests are already waiting for it; retry in 30 seconds`
A request released by a wake finds no room in the admission gate before its hold budget ends	`model <model> woke up but is still at capacity; retry in 30 seconds`

Match on code, not on the message. retry_after repeats the Retry-After header, in seconds. When you retry, send your own ID in X-Client-Request-ID (or X-Request-ID) and resend the same ID, so the retry is recognized as the same request. The ID must be a UUID, in any letter case: the router ignores any other value, and the retry then counts as a new request. A client that disconnects while held is recorded with status 499 in the audit log.

Hold limits

Limit	Default	Setting
How long a request is held	The placement’s `hold_budget`, else `5m`	`placements[].hold_budget` in the sibling map, `--wake-hold-budget`
Requests held per model	`100`	`--wake-hold-max`
Requests held per API key and model; requests without an API key (no authentication) have only the other limits	`20`	`--wake-hold-max-per-key`
Requests held in total, across all models	`1000`	`--wake-hold-max-total`

Each held request keeps its request body in memory, so --wake-hold-max-total bounds that memory. A request beyond any limit gets 503 model_waking at once and is not counted in the pending demand. Set the hold budget per model from its measured cold start: the time from the autoscaler’s decision to the backend’s first successful health check, plus a margin. A budget shorter than the model’s cold start fails every wake.

Quota while held

The per-model requests-per-minute quota scales with the number of healthy replicas, and is skipped when a model has none. For a request that may be held here (its model is placed here, for the endpoint’s kind, with scale_from_zero: true), the quota counts at least one replica, so a held request is charged like any other and a key cannot hold requests past its quota. A request refused at once because a hold limit is reached is charged too.

After the wake

A wake releases every held request together, often more than the woken replicas can serve at once. Released requests line up for the woken capacity instead of being refused, and receive keepalives while they wait:

Admission gate. The global admission budget (max_inflight, admit_budget_tokens) lets a released request wait for room until its hold budget ends, even when HIVENET_ROUTER_ADMIT_PARK_TIMEOUT is 0. While it waits, the budget is recomputed every second, so replicas that join after the wake are used. If there is still no room when the hold budget ends, the request gets 503 model_waking, not 429. A request larger than the whole budget of the current pool gets the usual 429 concurrency_limit_exceeded at once; it can fit later if more replicas join. The per-key max_occupancy_share does not wait: a key over its share gets 429 rate_limit_exceeded at once, as usual.
Capacity queue. The capacity queue accepts released requests beyond --queue-depth; the hold limits already bound how many there are. The wait there is bounded by --request-timeout (default 60s), not by the hold budget. When it runs out, the request moves to the next fallback step, or gets 504 request_timeout when there is none. With --queue-depth 0 there is no queue: a released request that finds no free slot moves to the next fallback step or fails at once.

Size --request-timeout for the wake burst: with one replica of 4 slots and 100 released requests, many wait longer than 60s. A new or recovered agent wakes every queued request together. A request that finds no free slot goes back in line at its original place, ahead of later arrivals.

Pending demand and the autoscaler

hivenet_pending_demand{model} is the number of distinct requests waiting for a model: the requests held now, plus the requests released by a wake that have not yet got a slot on an agent. It does not drop to 0 as soon as the first replica is up, so it also shows how many requests the woken capacity still has to take. A request leaves it when it gets a slot, so a request being generated is never counted. A client that retries the same request, with the same API key and the same client request ID (X-Client-Request-ID or X-Request-ID), counts once. The client ID must be a UUID: any other value is ignored, and each retry then counts on its own. A request without a client ID counts on its own, and requests from different keys always count separately, even with the same client ID. The metric has a series, 0 when nothing is held, for every model that the map places on this site with scale_from_zero: true. It has no series for the model while the router has no map (after a restart, until the map writer pushes it again), or when the map no longer places the model here with scale_from_zero: true. Write autoscaler queries so that a missing series counts as 0, as in the example below. Each router counts only the requests it holds itself. With several routers per site, sum the metric across them.

Waking the model

Held requests are released when an agent for the model registers, so the agent must register only when its backend can serve. The Hivenet Router agent checks its backend’s health endpoint before it registers: run it next to the backend, for example as a sidecar container, and start them together. Some backends report healthy before they can answer quickly: Ollama loads the model on its first request, and SGLang compiles kernels on it, which can take a minute or more. For these two engines, when the agent serves a chat model (--capability llm), it sends a one-token chat completion before it registers, and registers only once it succeeds, so the held requests do not pay for the load. When an agent’s connection to the router drops, the agent leaves the routing pool at once. A model that was just scaled down is therefore held again immediately, rather than receiving requests for the missed-heartbeat timeout.

Example: KEDA and a LeaderWorkerSet

This example scales a model served by a LeaderWorkerSet (LWS) between 0 and 1 replica. The model wakes when a request is held, stays awake while it serves traffic, and goes back to 0 after 30 minutes without requests:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: llama-3-70b
spec:
  scaleTargetRef:
    apiVersion: leaderworkerset.x-k8s.io/v1
    kind: LeaderWorkerSet
    name: llama-3-70b
  minReplicaCount: 1
  maxReplicaCount: 1
  idleReplicaCount: 0
  pollingInterval: 5
  cooldownPeriod: 1800
  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://prometheus.monitoring.svc:9090
      query: >-
        clamp_max(
          (sum(hivenet_pending_demand{model="llama-3-70b"}) or vector(0))
          + (sum(increase(hivenet_routing_requests_routed_total{model="llama-3-70b"}[2m])) or vector(0)),
          1)
      threshold: "1"
      activationThreshold: "0.5"

Wake. hivenet_pending_demand rises above 0 when a request is held, so the value passes activationThreshold and KEDA scales the LWS from 0 to 1.
Stay awake. increase(...[2m]) is above 0 for 2 minutes after each routed request, so the trigger stays active while the model serves traffic.
Idle. cooldownPeriod is the idle window. KEDA starts it when the trigger becomes inactive, about 2 minutes after the last request, and scales the LWS back to idleReplicaCount: 0 at its end. Keep the increase window short: a long window adds its own length to the cooldown, so [30m] with a 30-minute cooldown keeps an idle model up for about an hour.
On or off only. clamp_max(..., 1) makes the trigger say only whether the model should run. Without it, KEDA also reads the value as a replica count: with threshold: "1", 30 requests in the window ask for 30 replicas. That does not matter with maxReplicaCount: 1. For a model allowed more replicas, add a separate trigger for the replica count, for example hivenet_pending_demand divided by the requests one replica serves at once: it still counts the released requests waiting for capacity right after a wake.
No data. hivenet_pending_demand has no series while the router has no map, and hivenet_routing_requests_routed_total has none before the model’s first request. Each or vector(0) keeps one missing term from emptying the whole query: an empty result would scale a model that serves traffic down to 0.
New series. The counter has tenant_id, engine and region labels, and increase does not count the first sample of a series. The first request of a new label combination, or the first after a router restart, does not keep the trigger active, so the cooldown may start up to 2 minutes earlier than after the last request.

Use increase, not rate: a per-second rate of a lightly used model stays below any useful threshold. Give the backend container a readiness probe on its health endpoint, so Kubernetes does not report the pod ready before the model can serve.

Configuration

The flags are listed in the configuration reference: --wake-hold-budget, the three hold limits in Hold limits, and --wake-keepalive (default 15s, 0 disables keepalives). The router refuses to start when a value is invalid: a budget or limit that is not greater than 0, or a negative keepalive.

The `hold_budget` placement field

A placement in the sibling map may set its own hold budget:

{"model": "llama-3-70b", "kind": "chat", "site": "site-fr-1", "scale_from_zero": true, "hold_budget": "240s"}

hold_budget is optional: a Go duration greater than 0 and at most 1h. Any other value rejects the whole map push with 400. It applies only to placements with scale_from_zero: true.

Metrics

The hold-at-zero metrics are hivenet_pending_demand (see Pending demand), hivenet_wake_holds_total{outcome} and hivenet_wake_hold_seconds. A rising outcome="expired" count means the model takes longer to wake than its hold budget. A rising outcome="full" count means the hold limits are too low for the traffic that arrives while the model is asleep.

Limitations

Hold at zero needs cross-site redirect and the sibling map: there is no other way to declare which models may be held.
A model whose wake takes longer than its hold budget is still held for the whole budget before model_waking. The router does not know a model’s expected wake time.
There is no endpoint to wake a model ahead of traffic. Scale it up with your autoscaler instead.
Each router holds its own requests. A client retry that reaches another router of the same site is counted there too.
Once keepalives have started, a later error is reported in the body of a 200 response.

Getting started

Deploy Hivenet Router

Use the API

Routing and policies

Security and auth

Observability

Integrations

Reference

Project

How a request is handled

What the client sees

Keepalive

`model_waking`

Hold limits

Quota while held

After the wake

Pending demand and the autoscaler

Waking the model

Example: KEDA and a LeaderWorkerSet

Configuration

The `hold_budget` placement field

Metrics

Limitations

​How a request is handled

​What the client sees

​Keepalive

​model_waking

​Hold limits

​Quota while held

​After the wake

​Pending demand and the autoscaler

​Waking the model

​Example: KEDA and a LeaderWorkerSet

​Configuration

​The hold_budget placement field

​Metrics

​Limitations

How a request is handled

What the client sees

Keepalive

`model_waking`

Hold limits

Quota while held

After the wake

Pending demand and the autoscaler

Waking the model

Example: KEDA and a LeaderWorkerSet

Configuration

The `hold_budget` placement field

Metrics

Limitations