Run several routers

You can run several routers behind one load balancer, so the site keeps serving when one router stops. Each router counts the requests and tokens it admits. Without a shared store every router enforces every limit on its own, so a key spread over three routers gets up to three times its limit. With --sync-store=valkey the routers of one site share their usage through a Valkey store and enforce each limit together.

What is shared

Limit	Shared	How it holds
Requests per minute	Yes	Within one sync interval of usage
Input and output tokens per minute	Yes	Within one sync interval of usage
Tokens per day	Yes	Within one sync interval of usage; today’s total survives restarts
Occupancy budgets (B2, B4)	Yes	Exact: the routers together never hold more than the budget (see Occupancy credits)
Agents	Yes	Every agent connects to every router (see Connect agents to every router)
Redirect tokens	Yes	A cross-site redirect token is accepted once by the whole site, not once per router
Dynamic API keys	No	Push key changes to every router

Every router keeps deciding in memory, exactly as a single router does. The only time a request waits on the store is the first request of the day for a key with a daily limit, which reads that key’s total once: at most 200 ms, and requests for the same key share the read.

How it works

Every --sync-interval (default 1s) each router:

publishes the usage it admitted since the last round as one entry on a stream in Valkey (nothing when it admitted nothing);
reads the entries the other routers published and subtracts them from its own buckets;
writes its own running total of today’s tokens for each daily bucket it used (a bucket’s total is the sum over routers, so a write lost in a failover is healed by the next one);
records a heartbeat, which tells every router how many routers are live.

Between two rounds a router cannot see the others’ latest usage. To keep the routers together within a limit, each router admits at most its share of what was left at the last round: the remainder divided by the number of live routers. The first request of each round always passes if the limit has room, so a request larger than the share is not starved. A router that starts or restarts replays the last minute of usage and reads today’s totals, so it does not hand out a fresh budget. Usage already in the totals it reads is not counted again. The store load depends on request rate, not on the number of keys: each round carries only the buckets that changed. At 1,000 requests per second per router, each request on its own key, a round is about 36 KB.

Occupancy credits

The occupancy budgets must never be exceeded, even for a moment, so they are not synced like the rate limits. For each model the store holds a pool: how much of the budget each router holds right now. Each router keeps a wallet of credits in memory and admits against it, so a request never waits on the store.

When its wallet runs short, a router asks the pool for more. The pool grants only from the part of the budget no router holds, so the routers together never hold more than the budget. With Sentinel, a replica acknowledges each grant, so a failover cannot hand the same capacity out twice. When no replica acknowledges in time, the grant is used but the router stays at its share of the budget until a replica acknowledges again, so admission never stops because replicas lag.
A wallet no request has used for a minute hands its credits back and is dropped.
Each round a router keeps what it uses plus a small headroom and hands back the rest. A router that found its wallet empty flags it, and the others then keep no headroom, so the budget follows the traffic.
Streamed output that grows a request past its credits is recorded as held on the next round.

The store sees a refill when a wallet runs short and one write per model per round, not one call per request. A router that stops gives its credits back. Each router publishes, with its heartbeat, the agents it serves; the agents report which routers they are connected to, and stop the requests of a router they lose, so the engines free that work at once.

Situation	When its credits come back
Clean shutdown	At once, once it can no longer reach the agents. It admits nothing more after that
Every agent it served reports no connection to it (for example, it crashed)	As soon as it misses its heartbeats (about three sync intervals)
It restarted	As soon as its new run is live: one router ID runs one process at a time
It lost the store, and at least one of its agents is still connected to it	Never while that lasts: it may still be admitting within the credits it holds (at most its share). They come back once every agent it served has left it, or when it reconnects and rewrites what it holds
One of its agents reports to no other router (a network split, or the agent stopped with it)	Never while that lasts: the others cannot tell whether that agent still serves it. They come back when it restarts, or when the agent reports again

A router that was given up by the others drops the credits it believed it held and asks the pool again. A router cut off from the store admits nothing more once an agent connects to it again: that agent may have reported it gone in the meantime, and the others may have taken its credits back. It admits again once the store answers.

A router that fails together with one of its agents, for example when a whole node goes down, keeps its credits until it restarts under the same --router-id: the site runs with that much less budget meanwhile. It is never over budget.

When the store is unreachable

Situation	Behavior
Store unreachable for three sync intervals	Each router enforces its share of every limit (limit ÷ routers last seen, never fewer than `--sync-min-routers`), so the routers together stay within the limit. `hivenet_sync_up` drops to `0`
One router cut off from the store while the others still reach it	The cut-off router enforces its share of every limit on its own. The others count it as silent (it stopped heartbeating but no newer run replaced it) and keep its share free: they use (routers − silent) ÷ routers of each rate limit, and leave it its share of what was left of each daily budget. A router that shuts down cleanly leaves the live set and is not counted. A router that crashes and restarts with the same router ID (by default the hostname, so a container restart in the same pod) is replaced at the next round. One that disappears without a clean shutdown and never comes back with the same ID, such as a pod on a lost node, counts as silent for up to an hour: the others cannot tell it from a cut-off router that is still spending its share
Router starts while the store is unreachable	Same as above: it enforces its share with `--sync-min-routers` routers until the store answers
Store back	Routers publish the usage they held back, with the time it was admitted, and return to shared limits. Usage a rate bucket has refilled since is not charged again
A router cannot read today’s total for a key with a daily limit (it never saw the key today and the store is unreachable)	The router refuses that key’s requests at once with `429 token_limit_exceeded` until it can read the total. Keys without a daily limit are not affected. While the store is unreachable the refusal is immediate; while it hangs, the read waits at most 200 ms
Midnight UTC, or `auth.yaml` reloaded, while the store is unreachable	Keys the router already knew keep working: a new day starts at zero for every router, and a reload keeps today’s usage. Each router spends at most its share
The store is unreachable for ten sync intervals	The router reconnects from the configured addresses, in case Valkey or the Sentinels came back at new addresses
Occupancy while the store is unreachable	Each router keeps admitting within the credits it holds, capped at its share of the budget, and asks for no more. A router that holds no credits (for example, it started while the store was down) admits nothing until the store answers. A request larger than the router’s share is admitted when nothing else of its model is in flight on that router; every router may run one at once, so the engine may see up to the number of routers × the budget until the store is back
Occupancy after the store comes back, or after it lost its data	A grant made before a router noticed the loss is refused. Until every router seen before is back or known to be gone, each router holds at most the budget minus what those routers held when it last saw them. For a settle window after the store lost its data (the request timeout plus ten sync intervals plus 5 s), a router takes no new credits for a budget it had not seen before the loss, for example a router that started since: a router cut off from the store when it lost its data may still run requests no one else knows of, and they end within the window. The first deployment of a site waits the same window before it admits. While the agents report a connection to a router the store does not list, the others keep to their share, counting that router, for the same window; `hivenet_sync_unknown_routers` counts such routers, and a router that stays unknown usually has wrong store settings
Redirect tokens while the store is unreachable	Each router checks single use on its own

Output tokens are counted after the response, so they are never refused because of a router’s share. They are refused only when the shared daily budget is spent, as with one router.

Deploy Valkey

Use one Valkey per site, reachable only from the routers. For high availability run one primary, two replicas and three Sentinels on separate hosts. Sentinel promotes a replica when the primary fails; routers ask the Sentinels for the current primary. The store holds only recent usage, today’s totals and the occupancy credits. After a total loss the routers rebuild from what they hold in memory, but for the settle window (about 75 s with the default 60 s request timeout) a model or key no router had served before the loss gets 429 on every router. To keep restarts from causing that, run the primary with persistence (AOF, appendonly yes) and replicas behind Sentinel, so a restart or a failover keeps the data; a store that keeps its data is not treated as new. They stay within the budget as long as a router cut off from the store when it lost its data comes back, or ends its requests, within the settle window. One that stays cut off longer, and whose agents no other router hears from, can add up to its own share of the budget. Grants are acknowledged by a replica; the per-round rewrites of what each router holds are not, and are healed by the next round.

Configure the routers

Give every router of the site the same --sync-site (it prefixes every key in the store, as a hash tag: hivenet:{site-a}:…; up to 64 letters, digits, ., _ or -; it defaults to --site-id, else default) and a stable --router-id. Set --sync-min-routers to the number of routers: the router cannot know it while the store is unreachable, and with the default 1 routers that start during an outage each enforce the whole limit (the router logs a warning). In Kubernetes, set it to the StatefulSet’s replica count; a StatefulSet pod name works as --router-id; the default is the hostname.

hivenet-router \
  --sync-site site-a \
  --sync-store valkey \
  --sync-valkey-addrs sentinel-0:26379,sentinel-1:26379,sentinel-2:26379 \
  --sync-valkey-sentinel-master hivenet \
  --sync-valkey-password-file /run/secrets/valkey-password \
  --sync-min-routers 3

--sync-site names the site in the store only. --site-id does too when --sync-site is not set, but it also turns on cross-site redirect, which needs redirect secrets, allowed domains, and dynamic API keys or auth mode none.

Flag	Environment variable	Default	Description
`--sync-store`	`HIVENET_ROUTER_SYNC_STORE`	`none`	`none` or `valkey`
`--sync-valkey-addrs`	`HIVENET_ROUTER_SYNC_VALKEY_ADDRS`	Empty	Comma-separated `host:port` of the Sentinels, or the address of one Valkey without Sentinel (Valkey Cluster is not supported)
`--sync-valkey-sentinel-master`	`HIVENET_ROUTER_SYNC_VALKEY_SENTINEL_MASTER`	Empty	Sentinel master set name; empty connects to Valkey directly
`--sync-site`	`HIVENET_ROUTER_SYNC_SITE`	`--site-id`, else `default`	Name of the site in the store; the same on every router of the site
`--sync-valkey-password-file`	`HIVENET_ROUTER_SYNC_VALKEY_PASSWORD`	Empty	File holding the password for Valkey and Sentinel; the environment variable holds the password itself
`--sync-valkey-tls`	`HIVENET_ROUTER_SYNC_VALKEY_TLS`	`false`	Connect with TLS
`--sync-valkey-tls-ca-file`	`HIVENET_ROUTER_SYNC_VALKEY_TLS_CA_FILE`	Empty	PEM CA bundle that verifies Valkey and the Sentinels with TLS; empty uses the system roots
`--sync-interval`	`HIVENET_ROUTER_SYNC_INTERVAL`	`1s`	Between `100ms` and `10s`. Shorter intervals make limits stricter between rounds and cost more round trips
`--sync-min-routers`	`HIVENET_ROUTER_SYNC_MIN_ROUTERS`	`1`	Routers of the site assumed while the store is unreachable or before it first answers. Set it to the number of routers you run
`--router-id`	`HIVENET_ROUTER_ROUTER_ID`	Hostname	Stable identity of this router
`--advertise-grpc-addr`	`HIVENET_ROUTER_ADVERTISE_GRPC_ADDR`	Empty	gRPC address (`host:port`) agents reach this router at. The routers share it through the store, so an agent given one router finds the others. Empty: not advertised

--sync-store=valkey replaces HIVENET_ROUTER_QUOTA_BACKEND=badger: the store keeps today’s totals. The router refuses to start with both.

The load balancer must spread requests over the routers without changing the client’s API key. Every router must load the same auth.yaml, policies and key registry, so each one applies the same limits.

Connect agents to every router

Give each agent the gRPC address of every router of the site with --routers (or HIVENET_AGENT_ROUTERS):

HIVENET_ROUTER_JWT_SECRET=<site-secret> hivenet-agent \
  --engine vllm \
  --backend-url http://127.0.0.1:8000 \
  --routers router-0:50051,router-1:50051,router-2:50051

The agent keeps one independent session with each router: authentication, registration, heartbeat, routing signals and re-authentication. It reconnects each session on its own, with exponential backoff from 1 s up to 30 s, with jitter.

A router that is down or restarting does not affect the agent’s sessions with the other routers.
A router that comes up later is picked up automatically.
The agent polls the engine metrics and checks the backend health once, for all sessions.

Requirements:

Without discovery, each address reaches one specific router, never the client load balancer or a Kubernetes ClusterIP Service. Through a load balancer the agent gets a single session to whichever router answers, and the other routers never see the agent; nothing reports it. In Kubernetes, use the pod DNS names of a headless Service (router-0.router-headless:50051, …). With discovery, the list is only where the agent starts: it finds every router from the first one it reaches.
List every router, and update the list when you add one, or let the agents find every router. Every router divides the occupancy budgets by the live routers, whether or not agents reach it: a router no agent connects to lowers the others’ share and cannot serve requests itself.
Every router of the site uses the same JWT secret.
Each router sends its libp2p address in the authentication response. A router behind NAT must set --p2p-announce-addr to an address the agents can reach.

--router-grpc still connects an agent to a single router; it is used when --routers is empty.

Let agents find every router

Keeping every agent’s --routers up to date is easy to get wrong. Instead, set --advertise-grpc-addr on every router to the address agents reach it at:

# router-0 of a StatefulSet behind the headless Service router-headless
--advertise-grpc-addr router-0.router-headless:50051

The address must name a host agents can dial and a numeric port: a listen address such as :50051 or 0.0.0.0:50051 is refused at startup. Each router publishes its address with its heartbeat in the store. Every router hands the agents the live routers of the site that advertise one: in the authentication response, then in every heartbeat reply. An agent then:

opens a session with every listed router it has no session with, so a router added to the site gets the agents within one heartbeat (default 5 s);
needs only one reachable router in --routers, which becomes the starting list (it may be a DNS name);
recognizes a router it already reaches under another address (a --routers DNS name, the router listed under its pod IP) by its peer ID, and opens no second session to it. A router listed under a new address is dialed there once its old session is down;
closes the session of a discovered router only once that router has been missing from every list for 2 minutes and its session is down, or at once when one of its --routers entries already reaches it. A short or empty list (a router restarting) changes nothing, and the routers in --routers are never closed.

While the store is unreachable, the routers hand out no list, since they can no longer tell which routers are alive. An agent keeps at most 32 discovered routers. The list comes only from routers the agent authenticated with, and a listed address is useless without the site’s JWT secret. The list holds the same live routers the occupancy budgets are divided by, so a router that counts in the division also gets the agents. Older agents ignore the list; routers without --advertise-grpc-addr are not listed, and agents keep the sessions from their --routers. Each agent reports how many requests it has in flight from all routers. Every router ranks agents by the larger of that report and its own in-flight requests, so the routers spread work by the agent’s real load.

Monitor

Metric	Type	Meaning
`hivenet_sync_up`	gauge	`1` while the store answers; `0` while each router enforces its share
`hivenet_sync_peers`	gauge	Other live routers, as last seen
`hivenet_sync_lag_seconds`	gauge	Age of the latest peer usage when applied
`hivenet_sync_tick_seconds`	histogram	Duration of one sync round
`hivenet_sync_errors_total`	counter	Sync rounds that failed to reach the store
`hivenet_occupancy_credit_seconds`	histogram	Duration of one credit refill (`op="refill"`)
`hivenet_occupancy_credit_errors_total`	counter	Credit calls to the store that failed
`hivenet_occupancy_credit_reclaims_total`	counter	Routers given up early because no agent was connected to them
`hivenet_sync_unknown_routers`	gauge	Routers the agents report a connection to that the store does not list; one that stays above `0` usually has wrong store settings
`hivenet_admission_reaped_total`	counter	Occupancy reservations released because they outlived any request. Should stay at `0`: each one is a request exit path that skipped its release

Suggested alerts:

- alert: HivenetSyncDown
  expr: hivenet_sync_up == 0
  for: 1m
  labels: {severity: critical}
- alert: HivenetOccupancyLeak
  expr: increase(hivenet_admission_reaped_total[1h]) > 0
  labels: {severity: warning}
- alert: HivenetSyncPeersMissing
  expr: hivenet_sync_peers < 2
  for: 5m
  labels: {severity: warning}

Adjust hivenet_sync_peers to the number of routers you run minus one.

Getting started

Deploy Hivenet Router

Use the API

Routing and policies

Security and auth

Observability

Integrations

Reference

Project

What is shared

How it works

Occupancy credits

When the store is unreachable

Deploy Valkey

Configure the routers

Connect agents to every router

Let agents find every router

Monitor