> ## Documentation Index
> Fetch the complete documentation index at: https://routerdocs.hivenet.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Hold at zero

> Hold requests for a model that is scaled to zero on this site until an agent for it is healthy, and publish the demand an autoscaler uses to wake it.

Hold at zero lets a model scale down to zero replicas without failing the requests that arrive while it is asleep. When a request asks for a model that the sibling map places on this site with `scale_from_zero: true`, and no healthy agent serves it here, Hivenet Router holds the request instead of failing it. It counts the request in the `hivenet_pending_demand` metric, keeps the connection alive, and serves the request as soon as an agent for the model registers.

Hivenet Router does not start or stop models. An external autoscaler, such as [KEDA](https://keda.sh), reads `hivenet_pending_demand` and scales the model from zero. The agent registers once its backend answers its health check, and the router releases the held requests.

Hold at zero needs [cross-site redirect](https://routerdocs.hivenet.com/routing/cross-site-redirect.md): the sibling map says which models may be held on this site. With no `--site-id`, or for a model the map does not place here with `scale_from_zero: true`, nothing is held and the router behaves as before.

> [!NOTE]
>
> The sibling map is held in memory. A restarted router holds nothing until the map writer pushes the map again, and requests for a model at zero fail as before in the meantime.

## How a request is handled

After the API key is authenticated, the model alias is resolved and the model is checked against the key’s allowlist:

1. **Served here.** If a healthy agent on this site serves the model, the request is served here as usual.
2. **Redirect first.** If the sibling map lists another site in the request’s scope that serves the model, the router answers `307` to it, following the rules in [Cross-site redirect](https://routerdocs.hivenet.com/routing/cross-site-redirect.md#choosing-the-site). The one exception: while the map’s hints are fresh, every sibling in the caller’s [home set](https://routerdocs.hivenet.com/routing/cross-site-redirect.md#choosing-the-site) (the two sites a key is redirected to for a model) is cold and the model may be held here (step 3), the request is not redirected, because a cold sibling is no faster than waking the model here. A warm sibling outside the home set does not prevent this. The request then continues to step 3, as it does when no sibling is in scope or the request has already been redirected twice.
3. **Held here?** If the map places the model on this site, for the endpoint’s kind, with `scale_from_zero: true`, the request is held. Otherwise it fails as before, normally with `404 model_not_found`.
4. **Quota.** The key’s quota is checked before the hold, so a key over its limits is refused at once. See [Quota while held](#quota-while-held).
5. **Hold.** The request waits, counted once in `hivenet_pending_demand{model}`. After `--wake-keepalive`, the client receives [keepalive bytes](#keepalive).
6. **Wake.** The autoscaler starts the model. When an agent for the model registers, every held request is released and continues as a normal request: admission, routing and the backend call. It stays counted in `hivenet_pending_demand` until it gets a slot on an agent, and keeps receiving keepalives until its response starts. See [After the wake](#after-the-wake).
7. **Timeout.** If the model does not wake within the hold budget, the request gets `503 model_waking` with `Retry-After`.

Every inference endpoint can be held, by the placement kind that matches it: `chat` for `POST /v1/chat/completions` and `POST /v1/messages`; `embedding` for `POST /v1/embeddings`; `rerank` for `POST /v1/rerank`. `POST /v1/messages/count_tokens` is never held: a client sends it just before the real request, which is held and wakes the model. Holding both would count each prompt twice in the pending demand.

A request whose key may not use the model is never held: it gets `403 model_forbidden` at once.

## What the client sees

A held request looks like a slow first response. The client needs no changes, but its request timeout must be longer than the hold budget plus the generation time.

| Outcome | Before keepalives start | After keepalives start |
| - | - | - |
| The model wakes and the request succeeds | The normal response | `200`, keepalive bytes, then the normal response |
| The hold budget runs out | `503 model_waking` with `Retry-After` | `200`, keepalive bytes, then the error in the body |
| A hold limit is reached | `503 model_waking` with `Retry-After`, at once | Not applicable: the request is refused before it is held |
| An error after the release, such as a routing failure | The normal error status | `200`, keepalive bytes, then the error in the body |

### Keepalive

A held connection carries no data, so proxies and load balancers with an idle timeout may close it, and client SDKs may then retry. After `--wake-keepalive` (default `15s`), and then at the same interval, the router sends keepalive bytes. They go on after the wake while the released request waits for capacity, with no longer gap at the release, and stop when the response starts:

* an SSE comment, `: keep-alive`, to a streaming request (`"stream": true`);
* a single space to any other request. Leading whitespace is valid before a JSON body.

The first keepalive sends the `200` status and the response headers, so the status can no longer change. An error after it is written in the body:

* for a streaming request, as a final SSE event. OpenAI-compatible endpoints get `data: {"error":{...}}`. `/v1/messages` gets an Anthropic `error` event whose message starts with the error code.
* for any other request, as the usual JSON error envelope after the spaces.

The audit log and the HTTP metrics record the status the request ended with, such as `503`, not the `200` sent on the wire.

> [!WARNING]
>
> A non-streaming client that receives `200` parses the body as a success. With keepalives on, a hold that expires reaches such a client as a `200` whose body is an error envelope. Set `--wake-keepalive 0` to keep plain status codes when no proxy between your clients and the router closes idle connections within the hold budget.

### `model_waking`

```text theme={null}
HTTP 503
Retry-After: 30
```

```json theme={null}
{
  "error": {
    "code": "model_waking",
    "message": "model llama-3-70b is waking up; retry in 30 seconds",
    "source": "router",
    "retry_after": 30
  }
}
```

The code is the same in every case; the message says which one it is:

| Case | Message |
| - | - |
| The hold budget runs out | `model <model> is waking up; retry in 30 seconds` |
| A [hold limit](#hold-limits) is reached | `model <model> is waking up and too many requests are already waiting for it; retry in 30 seconds` |
| A request released by a wake finds no room in the admission gate before its hold budget ends | `model <model> woke up but is still at capacity; retry in 30 seconds` |

Match on `code`, not on the message. `retry_after` repeats the `Retry-After` header, in seconds.

When you retry, send your own ID in `X-Client-Request-ID` (or `X-Request-ID`) and resend the same ID, so the retry is recognized as the same request. The ID must be a UUID, in any letter case: the router ignores any other value, and the retry then counts as a new request.

A client that disconnects while held is recorded with status `499` in the audit log.

## Hold limits

| Limit | Default | Setting |
| - | - | - |
| How long a request is held | The placement’s `hold_budget`, else `5m` | `placements[].hold_budget` in the sibling map, `--wake-hold-budget` |
| Requests held per model | `100` | `--wake-hold-max` |
| Requests held per API key and model; requests without an API key (no authentication) have only the other limits | `20` | `--wake-hold-max-per-key` |
| Requests held in total, across all models | `1000` | `--wake-hold-max-total` |

Each held request keeps its request body in memory, so `--wake-hold-max-total` bounds that memory. A request beyond any limit gets `503 model_waking` at once and is not counted in the pending demand.

Set the hold budget per model from its measured cold start: the time from the autoscaler’s decision to the backend’s first successful health check, plus a margin. A budget shorter than the model’s cold start fails every wake.

### Quota while held

The per-model requests-per-minute quota scales with the number of healthy replicas, and is skipped when a model has none. For a request that may be held here (its model is placed here, for the endpoint’s kind, with `scale_from_zero: true`), the quota counts at least one replica, so a held request is charged like any other and a key cannot hold requests past its quota. A request refused at once because a hold limit is reached is charged too.

## After the wake

A wake releases every held request together, often more than the woken replicas can serve at once. Released requests line up for the woken capacity instead of being refused, and receive keepalives while they wait:

* **Admission gate.** The global [admission budget](https://routerdocs.hivenet.com/routing/admission-control.md#b2-pool-occupancy) (`max_inflight`, `admit_budget_tokens`) lets a released request wait for room until its hold budget ends, even when `HIVENET_ROUTER_ADMIT_PARK_TIMEOUT` is `0`. While it waits, the budget is recomputed every second, so replicas that join after the wake are used. If there is still no room when the hold budget ends, the request gets `503 model_waking`, not `429`. A request larger than the whole budget of the current pool gets the usual `429 concurrency_limit_exceeded` at once; it can fit later if more replicas join. The per-key [`max_occupancy_share`](https://routerdocs.hivenet.com/routing/admission-control#b4-serverless-key-limits) does not wait: a key over its share gets `429 rate_limit_exceeded` at once, as usual.
* **Capacity queue.** The [capacity queue](https://routerdocs.hivenet.com/routing/fallback-chains.md#capacity-and-the-wait-queue) accepts released requests beyond `--queue-depth`; the hold limits already bound how many there are. The wait there is bounded by `--request-timeout` (default `60s`), not by the hold budget. When it runs out, the request moves to the next fallback step, or gets `504 request_timeout` when there is none. With `--queue-depth 0` there is no queue: a released request that finds no free slot moves to the next fallback step or fails at once.

Size `--request-timeout` for the wake burst: with one replica of 4 slots and 100 released requests, many wait longer than `60s`.

A new or recovered agent wakes every queued request together. A request that finds no free slot goes back in line at its original place, ahead of later arrivals.

## Pending demand and the autoscaler

`hivenet_pending_demand{model}` is the number of distinct requests waiting for a model: the requests held now, plus the requests released by a wake that have not yet got a slot on an agent. It does not drop to `0` as soon as the first replica is up, so it also shows how many requests the woken capacity still has to take. A request leaves it when it gets a slot, so a request being generated is never counted.

A client that retries the same request, with the same API key and the same [client request ID](https://routerdocs.hivenet.com/observability/audit-logging.md#request-ids) (`X-Client-Request-ID` or `X-Request-ID`), counts once. The client ID must be a UUID: any other value is ignored, and each retry then counts on its own. A request without a client ID counts on its own, and requests from different keys always count separately, even with the same client ID.

The metric has a series, `0` when nothing is held, for every model that the map places on this site with `scale_from_zero: true`. It has no series for the model while the router has no map (after a restart, until the map writer pushes it again), or when the map no longer places the model here with `scale_from_zero: true`. Write autoscaler queries so that a missing series counts as `0`, as in the example below.

Each router counts only the requests it holds itself. With several routers per site, sum the metric across them.

### Waking the model

Held requests are released when an agent for the model registers, so the agent must register only when its backend can serve. The Hivenet Router agent checks its backend’s health endpoint before it registers: run it next to the backend, for example as a sidecar container, and start them together.

Some backends report healthy before they can answer quickly: Ollama loads the model on its first request, and SGLang compiles kernels on it, which can take a minute or more. For these two engines, when the agent serves a chat model (`--capability llm`), it sends a one-token chat completion before it registers, and registers only once it succeeds, so the held requests do not pay for the load.

When an agent’s connection to the router drops, the agent leaves the routing pool at once. A model that was just scaled down is therefore held again immediately, rather than receiving requests for the missed-heartbeat timeout.

### Example: KEDA and a LeaderWorkerSet

This example scales a model served by a [LeaderWorkerSet](https://lws.sigs.k8s.io) (LWS) between 0 and 1 replica. The model wakes when a request is held, stays awake while it serves traffic, and goes back to 0 after 30 minutes without requests:

```yaml theme={null}
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: llama-3-70b
spec:
  scaleTargetRef:
    apiVersion: leaderworkerset.x-k8s.io/v1
    kind: LeaderWorkerSet
    name: llama-3-70b
  minReplicaCount: 1
  maxReplicaCount: 1
  idleReplicaCount: 0
  pollingInterval: 5
  cooldownPeriod: 1800
  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://prometheus.monitoring.svc:9090
      query: >-
        clamp_max(
          (sum(hivenet_pending_demand{model="llama-3-70b"}) or vector(0))
          + (sum(increase(hivenet_routing_requests_routed_total{model="llama-3-70b"}[2m])) or vector(0)),
          1)
      threshold: "1"
      activationThreshold: "0.5"
```

* **Wake.** `hivenet_pending_demand` rises above `0` when a request is held, so the value passes `activationThreshold` and KEDA scales the LWS from `0` to `1`.
* **Stay awake.** `increase(...[2m])` is above `0` for 2 minutes after each routed request, so the trigger stays active while the model serves traffic.
* **Idle.** `cooldownPeriod` is the idle window. KEDA starts it when the trigger becomes inactive, about 2 minutes after the last request, and scales the LWS back to `idleReplicaCount: 0` at its end. Keep the `increase` window short: a long window adds its own length to the cooldown, so `[30m]` with a 30-minute cooldown keeps an idle model up for about an hour.
* **On or off only.** `clamp_max(..., 1)` makes the trigger say only whether the model should run. Without it, KEDA also reads the value as a replica count: with `threshold: "1"`, 30 requests in the window ask for 30 replicas. That does not matter with `maxReplicaCount: 1`. For a model allowed more replicas, add a separate trigger for the replica count, for example `hivenet_pending_demand` divided by the requests one replica serves at once: it still counts the released requests waiting for capacity right after a wake.
* **No data.** `hivenet_pending_demand` has no series while the router has no map, and `hivenet_routing_requests_routed_total` has none before the model’s first request. Each `or vector(0)` keeps one missing term from emptying the whole query: an empty result would scale a model that serves traffic down to `0`.
* **New series.** The counter has `tenant_id`, `engine` and `region` labels, and `increase` does not count the first sample of a series. The first request of a new label combination, or the first after a router restart, does not keep the trigger active, so the cooldown may start up to 2 minutes earlier than after the last request.

Use `increase`, not `rate`: a per-second rate of a lightly used model stays below any useful threshold. Give the backend container a readiness probe on its health endpoint, so Kubernetes does not report the pod ready before the model can serve.

## Configuration

The flags are listed in the [configuration reference](https://routerdocs.hivenet.com/reference/configuration-reference.md#hold-at-zero): `--wake-hold-budget`, the three hold limits in [Hold limits](#hold-limits), and `--wake-keepalive` (default `15s`, `0` disables keepalives). The router refuses to start when a value is invalid: a budget or limit that is not greater than `0`, or a negative keepalive.

### The `hold_budget` placement field

A placement in the [sibling map](https://routerdocs.hivenet.com/routing/cross-site-redirect.md#pushing-the-sibling-map) may set its own hold budget:

```json theme={null}
{"model": "llama-3-70b", "kind": "chat", "site": "site-fr-1", "scale_from_zero": true, "hold_budget": "240s"}
```

`hold_budget` is optional: a Go duration greater than `0` and at most `1h`. Any other value rejects the whole map push with `400`. It applies only to placements with `scale_from_zero: true`.

## Metrics

The [hold-at-zero metrics](https://routerdocs.hivenet.com/observability/prometheus-metrics.md#hold-at-zero-metrics) are `hivenet_pending_demand` (see [Pending demand](#pending-demand-and-the-autoscaler)), `hivenet_wake_holds_total{outcome}` and `hivenet_wake_hold_seconds`. A rising `outcome="expired"` count means the model takes longer to wake than its hold budget. A rising `outcome="full"` count means the hold limits are too low for the traffic that arrives while the model is asleep.

## Limitations

* Hold at zero needs cross-site redirect and the sibling map: there is no other way to declare which models may be held.
* A model whose wake takes longer than its hold budget is still held for the whole budget before `model_waking`. The router does not know a model’s expected wake time.
* There is no endpoint to wake a model ahead of traffic. Scale it up with your autoscaler instead.
* Each router holds its own requests. A client retry that reaches another router of the same site is counted there too.
* Once keepalives have started, a later error is reported in the body of a `200` response.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.