Policy YAML reference - Hivenet Router

A Hivenet Router policy defines how the router filters, ranks, retries, and falls back across agents. Policies use YAML and can apply globally or to specific models.

Complete structure

models:
  - meta-llama/Llama-3.1-8B-Instruct

mode: reserved
max_input_tokens: 131072
images_max: 8
admit_budget_tokens: 262144
max_inflight: 16
shed_if:
  kv_cache_utilization:
    gt: 0.90
  waiting_requests:
    gt: 20

routing_policy:
  match:
    region: EU-France
    engine: vllm
    tags:
      - production

  exclude_if:
    kv_cache_utilization:
      gt: 0.90
    success_rate:
      lt: 0.95

  strategy: least-loaded
  max_tries: 3

fallback_chain:
  - name: any-vllm-region

    match:
      engine: vllm

    exclude_if:
      success_rate:
        lt: 0.9

    strategy: least-loaded
    max_tries: 2

fallback_provider:
  engine: openai
  model: gpt-4o-mini

Top-level fields

Field	Type	Required	Purpose
`models`	Array of strings	For per-model policy documents	Models that use this policy
`mode`	String	No	Admission profile: `reserved` (default) or `serverless`
`max_input_tokens`	Integer	No	Maximum learned prompt estimate per request; `0` disables B1
`images_max`	Integer	No	Maximum images per request; `0` disables the image check
`admit_budget_tokens`	Integer	No	Per-replica token-weighted occupancy capacity; `0` disables the token budget
`max_inflight`	Integer	No	Per-replica in-flight request backstop; `0` disables the count limit
`shed_if`	Object	No	Front-door thresholds for aggregate healthy-pool engine pressure
`routing_policy`	Object	Yes	Primary local routing step
`fallback_chain`	Array	No	Ordered local fallback steps
`fallback_provider`	Object	No	Final external provider fallback

The models field is ignored for a global policy loaded through:

--policy-file
_default.yaml
PUT /admin/policy

It is required for named per-model policy documents.

Admission control fields

The admission fields are top-level properties of the policy, not fields inside routing_policy. They apply to POST /v1/chat/completions and POST /v1/messages; the count-tokens, embedding, and reranking paths are exempt.

mode: reserved
max_input_tokens: 131072
images_max: 8
admit_budget_tokens: 262144
max_inflight: 16
shed_if:
  kv_cache_utilization:
    gt: 0.90
  waiting_requests:
    gt: 20

All numeric limits default to 0, which disables the corresponding check. Negative values are rejected. mode defaults to reserved and accepts only reserved or serverless.

`mode`

reserved enables the shared B1, B2, and B3 safety gates without serverless per-key caps. serverless enables the same safety gates plus the B4 fields resolved from the authenticated key. An empty value is normalized to reserved; any other value is a load-time error.

`max_input_tokens` and `images_max`

These are the B1 hard caps. The token check uses the same learned estimate used by occupancy and token quotas. It includes message text, the Anthropic top-level system prompt, and raw tool-definition JSON. The image check counts OpenAI image_url parts and Anthropic image blocks. Either breach returns HTTP 400 input_too_long.

`admit_budget_tokens` and `max_inflight`

Both values are per replica. The router recomputes the number of healthy replicas for every request and floors that count at 1:

effective token budget = floor(HIVENET_ROUTER_ADMIT_FRACTION
                               × admit_budget_tokens
                               × healthy_replicas)

effective request backstop = max_inflight × healthy_replicas

Over-budget requests may park for HIVENET_ROUTER_ADMIT_PARK_TIMEOUT before the router returns 429 concurrency_limit_exceeded. Existing reservations are not canceled when the healthy pool shrinks; the smaller limit affects new admissions until work completes. Benchmark-derived values are not loaded automatically from router_limits.yaml. Operators or provisioning tooling must copy the certified per-model values into the policy.

`shed_if`

shed_if:
  kv_cache_utilization:
    gt: 0.90
  waiting_requests:
    gt: 8

The loader accepts only kv_cache_utilization and waiting_requests in shed_if, with exactly one of gt, lt, gte, or lte per field. At admission, the router averages each reported signal across healthy agents serving the requested model. Missing signals pass. If a threshold fails, the router returns 429 concurrency_limit_exceeded with Retry-After: 1 before queueing the request. This is additive to per-agent exclude_if gates. Use exclude_if to remove an unhealthy or overloaded replica from a routing step; use shed_if to stop admitting new work when the serving pool as a whole is under pressure.

Cross-config validation

At startup and on auth or policy reload, the router validates every static and dynamic key against the effective serverless policies. Dynamic admin writes run the same checks. A configured occupancy share must be in (0, 1], and a nonzero input_tokens_per_minute must be at least max_input_tokens for every reachable serverless model. Invalid reloads leave the previous policy active. See Admission control for request-footprint calculation, parking, per-key limits, and error responses.

Primary routing policy

routing_policy:
  match: {}
  exclude_if: {}
  strategy: least-loaded
  max_tries: 3

Field	Type	Required	Default	Purpose
`match`	Object	No	Match any agent	Static metadata filters
`exclude_if`	Object	No	No metric gates	Live operational thresholds
`strategy`	String	Yes	None	Candidate-ranking strategy
`max_tries`	Integer	No	Router-wide value	Failed forwards allowed in this step

The router rejects a policy whose primary step has no strategy.

Static matching

The match object filters agents using metadata supplied during registration.

match:
  region: EU-France
  engine: vllm
  tags:
    - production
    - high-memory
  organization: ml-team
  machine: gpu-worker-1
  gpu_model: RTX4090

All non-empty conditions must match. Comparisons are exact and case-sensitive.

Field	Type	Agent setting or source
`region`	String	`--region`
`engine`	String	`--engine`
`tags`	Array of strings	`--tags`
`organization`	String	`--organization`
`machine`	String	`--machine`
`gpu_model`	String	`--gpu-model` or detected metadata

Engine values

Built-in agent engine names are:

vllm
ollama
sglang
llamacpp
infinity
custom

For example:

match:
  engine: llamacpp

Use llamacpp, not llama.cpp.

Match logic

This policy:

match:
  region: EU-France
  engine: vllm

matches only an agent whose region is exactly EU-France and whose engine is exactly vllm. It does not match:

Region: eu-france
Engine: vllm

or:

Region: EU
Engine: vllm

Tag logic

Every tag listed in the policy must be present on the agent.

match:
  tags:
    - production
    - high-memory

This matches an agent with:

production,high-memory,a100

It does not match an agent with only:

production

Additional agent tags are allowed.

Match every eligible agent

Use:

match: {}

or omit match when the policy should not restrict agents by metadata. Model, health, capability, previous failures, and capacity still apply as hard routing constraints.

Dynamic metric gates

The exclude_if object removes agents whose live metrics violate a threshold.

exclude_if:
  kv_cache_utilization:
    gt: 0.85

  gpu_temperature_c:
    gt: 82

  success_rate:
    lt: 0.95

An agent is excluded when it violates any configured rule.

Comparison operators

Each metric accepts exactly one operator.

Operator	Excludes the agent when
`gt`	Value is greater than the threshold
`gte`	Value is greater than or equal to the threshold
`lt`	Value is less than the threshold
`lte`	Value is less than or equal to the threshold

Valid:

success_rate:
  lt: 0.95

Invalid because it has no operator:

success_rate: {}

Invalid because it has two operators:

success_rate:
  lt: 0.95
  gt: 0.5

Unknown metric names are rejected when the policy loads. For example, this typo causes a validation error:

exclude_if:
  sucess_rate:
    lt: 0.95

Missing metrics

When a metric is unavailable for an agent, Hivenet Router skips that gate for the agent. For example:

exclude_if:
  kv_cache_utilization:
    gt: 0.85

does not exclude an Ollama agent merely because Ollama does not provide a KV-cache metric.

Missing data passes a metric gate.When a metric is required for the policy to be meaningful, combine the gate with a static engine or metadata filter.

For example:

match:
  engine: vllm

exclude_if:
  kv_cache_utilization:
    gt: 0.85

Metric units

Value type	Policy unit
Capacity, success rate, and utilization	Fraction from `0.0` to `1.0`
SRTT	Milliseconds
TTFT and ITL	Seconds
GPU temperature	Degrees Celsius
Request, queue, and failure values	Absolute counts

Use:

gpu_util_percent:
  gt: 0.95

for 95% GPU utilization. Do not use:

gpu_util_percent:
  gt: 95

Prometheus may display hardware utilization on a 0 to 100 scale. Hivenet Router normalizes these values to fractions before policy evaluation.

Universal metrics

These gates can apply to any agent once the necessary runtime history exists.

Field	Range	Meaning
`capacity_utilization`	`0.0`–`1.0`	Active requests divided by declared capacity
`success_rate`	`0.0`–`1.0`	Lifetime fraction of successful forwards
`srtt`	`0+`	Smoothed round-trip time in milliseconds
`consecutive_failures`	`0+`	Forward failures since the last success

Capacity utilization

exclude_if:
  capacity_utilization:
    gt: 0.8

This excludes agents above 80% of their declared capacity. Hivenet Router also applies a hard capacity gate. An agent is unavailable when:

active requests >= declared capacity

even when the policy contains no capacity_utilization rule.

Success rate

exclude_if:
  success_rate:
    lt: 0.95

A new agent with no request history has no success-rate value, so this rule is skipped until the agent has served requests.

Smoothed latency

exclude_if:
  srtt:
    gt: 500

The threshold is in milliseconds. SRTT becomes available after Hivenet Router has collected routing latency observations for the agent.

Consecutive failures

exclude_if:
  consecutive_failures:
    gte: 3

The counter increases after forward failures and resets after a successful request.

Engine metrics

These gates depend on metrics exposed by the inference backend.

Field	Range	Meaning
`kv_cache_utilization`	`0.0`–`1.0`	KV or token-cache fraction in use
`running_requests`	`0+`	Requests currently being processed
`waiting_requests`	`0+`	Requests waiting in the backend scheduler
`avg_ttft_seconds`	`0+`	Average time to first token
`p90_ttft_seconds`	`0+`	P90 time to first token
`avg_itl_seconds`	`0+`	Average inter-token latency
`p90_itl_seconds`	`0+`	P90 inter-token latency

Backend support differs:

Backend	Available engine signals
vLLM	Cache, running, waiting, TTFT, ITL, and preemption data
SGLang	Cache, running, waiting, and TTFT
llama.cpp with `--metrics`	Cache, running, waiting, TTFT, ITL, and throughput
Ollama	No engine-specific policy metrics
Infinity	No engine-specific policy metrics
Custom	No engine-specific policy metrics

Preemption and throughput values are available for observability but are not currently valid exclude_if fields.

Cache pressure

exclude_if:
  kv_cache_utilization:
    gt: 0.85

Backend queue

exclude_if:
  waiting_requests:
    gt: 0

Time to first token

exclude_if:
  p90_ttft_seconds:
    gt: 5

Hardware metrics

Hardware gates use the most recent snapshot reported by the agent.

Field	Range	Meaning
`gpu_temperature_c`	`0+`	Highest GPU temperature in Celsius
`gpu_util_percent`	`0.0`–`1.0`	Highest GPU compute utilization
`gpu_vram_used_percent`	`0.0`–`1.0`	Highest GPU VRAM-used fraction
`memory_used_percent`	`0.0`–`1.0`	System memory-used fraction
`cpu_usage_percent`	`0.0`–`1.0`	CPU-utilization fraction

When a host has several GPUs, Hivenet Router evaluates the highest reported value for:

temperature
compute utilization
VRAM utilization

For example:

exclude_if:
  gpu_temperature_c:
    gt: 82

  gpu_vram_used_percent:
    gt: 0.9

  memory_used_percent:
    gt: 0.9

A CPU-only agent has no GPU values, so GPU gates are skipped for that agent.

Ranking strategy

Every policy step requires a ranking strategy. The only implemented strategy is:

strategy: least-loaded

It ranks candidates by:

active requests / declared capacity

The lowest ratio ranks first. The following names are not currently supported:

lowest-srtt
round-robin
prefix-aware
lowest-kv-cache
lowest-queue
best-ttft
best-itl

A policy using one of these strategies is rejected.

Forward attempts

max_tries: 3

max_tries is the number of failed forwards allowed in one policy step before Hivenet Router advances to the next fallback step. A value of 0, a negative value, or an omitted field uses the router-wide default:

--max-tries-per-step 3

Examples of failures that consume a try include:

backend unavailability
backend overload or rate-limit responses
retryable backend errors
transport failures after connection recovery is exhausted

Request-level failures such as request_invalid, invalid_parameter, context_length_exceeded, and token_limit_exceeded stop immediately. They do not consume additional tries or advance through other local agents. The following do not consume a forward try:

no matching agents
agents excluded by policy gates
all agents already at capacity
losing an atomic capacity-slot race
the first connection-level recovery attempt for an agent

Each fallback step can define its own budget.

Fallback chain

fallback_chain contains ordered local policy steps.

fallback_chain:
  - name: any-vllm-region

    match:
      engine: vllm

    exclude_if:
      success_rate:
        lt: 0.9

    strategy: least-loaded
    max_tries: 2

  - name: any-engine

    match: {}

    strategy: least-loaded
    max_tries: 2

The primary routing_policy always runs first. Each fallback entry supports:

Field	Type	Required	Purpose
`name`	String	No	Label used in logs and metrics
`match`	Object	No	Static metadata filters
`exclude_if`	Object	No	Dynamic metric gates
`strategy`	String	Yes	Candidate-ranking strategy
`max_tries`	Integer	No	Step-specific forward budget

When name is omitted, Hivenet Router generates a name based on the step’s array position. A step advances when it has no viable candidates or exhausts its forward budget. Model and capability remain fixed throughout the fallback chain. A fallback step cannot route a request to another model or workload capability. See Fallback chains for examples and execution behavior.

Provider fallback

fallback_provider defines one external API used after every local step is exhausted.

fallback_provider:
  engine: openai
  model: gpt-4o-mini

It is a top-level field. Do not put it inside fallback_chain.

Field	Type	Required	Purpose
`engine`	String	Yes	External provider
`model`	String	Yes	Model sent to that provider

Supported engines are:

openai
anthropic

Configure credentials through environment variables:

Provider	Environment variable
OpenAI	`HIVENET_ROUTER_OPENAI_API_KEY`
Anthropic	`HIVENET_ROUTER_ANTHROPIC_API_KEY`

Credentials do not belong in the policy YAML. The router rejects a policy when:

the provider engine is unsupported
the corresponding API key is not configured
the provider model is empty

Provider fallback is intended for non-streaming Chat Completions requests.Embedding and reranking requests do not use it, and it is not a transparent fallback for Anthropic token counting or every Messages field.

See Provider fallback for request translation and feature limitations.

Global policy file

Load one global policy:

./bin/hivenet-router \
  --policy-file /etc/hivenet-router/policy.yaml

The environment-variable equivalent is:

export HIVENET_ROUTER_POLICY_FILE=/etc/hivenet-router/policy.yaml

When no global policy is configured, Hivenet Router uses:

routing_policy:
  match: {}
  strategy: least-loaded

with the router-wide forward-attempt limit. A parse or validation error in the configured global policy prevents the router from starting.

Per-model policy directory

Use a directory when different models need different policies:

./bin/hivenet-router \
  --policy-model-dir /etc/hivenet-router/policies

The environment-variable equivalent is:

export HIVENET_ROUTER_POLICY_MODEL_DIR=/etc/hivenet-router/policies

Example:

/etc/hivenet-router/policies/
├── _default.yaml
├── llama-8b.yaml
├── llama-70b.yaml
└── embeddings.yml

Only .yaml and .yml files are loaded.

`_default.yaml`

The special file:

_default.yaml

or:

_default.yml

defines the global policy for the directory. Its models field is ignored.

routing_policy:
  match: {}
  strategy: least-loaded
  max_tries: 3

When both are configured:

--policy-file
--policy-model-dir with _default.yaml

_default.yaml takes precedence. A parse or validation error in _default.yaml is fatal during router startup.

Avoid configuring both a global file and a policy directory unless you need that exact precedence behavior.When a policy directory is configured, SIGHUP reloads the directory rather than the separate --policy-file.

Named per-model files

Every other policy file must contain at least one model:

models:
  - meta-llama/Llama-3.1-8B-Instruct
  - meta-llama/Llama-3.1-70B-Instruct

routing_policy:
  match:
    engine: vllm

  strategy: least-loaded

The filename stem becomes the policy-document name. For:

llama-family.yaml

the name is:

llama-family

That name appears in the administration API and operational logs. A file without models is skipped with a warning. A model can belong to only one named policy document.

Model ownership conflicts

At initial startup, policy files are processed by modification time, oldest first. When two files claim the same model:

the older file claims the model
the later conflicting file is skipped entirely

When modification times are equal, filenames provide a deterministic tie-break. For example:

# production.yaml
models:
  - shared-model

and:

# experimental.yaml
models:
  - shared-model

cannot both load.

Do not rely on file modification times as an intentional priority system.Keep model ownership unique and explicit across the directory.

A conflict in one model causes the entire later file to be skipped, including any other models it contains.

Per-model file errors

During initial loading:

invalid _default.yaml stops router startup
an invalid named policy file is logged and skipped
a named file without models is logged and skipped
other valid named policies still load

This partial-failure behavior prevents one broken model policy from blocking unrelated models.

Policy selection

For each request, Hivenet Router:

looks for a named policy assigned to the requested model
uses that policy when one exists
otherwise uses the active global policy

A per-model policy replaces the entire global policy for that model. It does not inherit or merge:

global match values
global gates
global fallback steps
global provider fallback

Repeat any global behavior that the per-model policy still needs.

Reload policies with SIGHUP

Reload policies without restarting the router:

sudo kill -HUP "$(pgrep hivenet-router)"

For a single global file:

Hivenet Router parses and validates the file
valid changes replace the active policy
invalid changes are rejected
the previous active policy remains in use

For a policy directory:

unchanged files are not reapplied
valid changed files replace their named policies
deleted named files lose their policies
invalid existing files keep their previous active policy
conflicting changed files are skipped
unrelated valid policies remain active

This conservative reload behavior avoids removing a working policy merely because an edited file temporarily contains an error.

If _default.yaml is deleted during a directory reload, the current implementation reverts the global policy to Hivenet Router’s built-in least-loaded default.

Reload events are exposed through policy reload metrics and router logs.

Manage policies through the admin API

Global policy

GET /admin/policy
PUT /admin/policy

Update the global policy with a YAML body:

curl -X PUT \
  -H "Authorization: Bearer <admin-api-key>" \
  -H "Content-Type: text/yaml" \
  --data-binary @policy.yaml \
  http://localhost:8080/admin/policy

The models field is ignored for this endpoint.

Named policies

GET    /admin/policy/models
GET    /admin/policy/models/{name}
PUT    /admin/policy/models/{name}
DELETE /admin/policy/models/{name}

Create or replace a named policy:

curl -X PUT \
  -H "Authorization: Bearer <admin-api-key>" \
  -H "Content-Type: text/yaml" \
  --data-binary @llama-family.yaml \
  http://localhost:8080/admin/policy/models/llama-family

The YAML body must contain at least one model. API-created policies are ephemeral and disappear when the router restarts. Use files and --policy-model-dir for persistent configuration. See Admin endpoints for the complete API behavior.

Global example

routing_policy:
  match:
    region: EU-France
    engine: vllm
    tags:
      - production

  exclude_if:
    kv_cache_utilization:
      gt: 0.85

    gpu_temperature_c:
      gt: 82

    success_rate:
      lt: 0.95

    srtt:
      gt: 500

  strategy: least-loaded
  max_tries: 3

fallback_chain:
  - name: any-vllm-region

    match:
      engine: vllm

    exclude_if:
      success_rate:
        lt: 0.9

    strategy: least-loaded
    max_tries: 2

fallback_provider:
  engine: openai
  model: gpt-4o-mini

Per-model example

models:
  - BAAI/bge-m3

routing_policy:
  match:
    engine: infinity
    region: EU-France

  exclude_if:
    success_rate:
      lt: 0.95

  strategy: least-loaded
  max_tries: 3

fallback_chain:
  - name: any-region

    match:
      engine: infinity

    strategy: least-loaded
    max_tries: 2

Because BAAI/bge-m3 is an embedding model, a configured fallback_provider would not be used for its requests.

Validation checklist

Before loading a policy, check that:

routing_policy.strategy is present
every fallback step has a supported strategy
the strategy is least-loaded
every exclude_if field is spelled correctly
every metric has exactly one comparison operator
percentage thresholds use fractions
SRTT uses milliseconds
TTFT and ITL use seconds
provider credentials are set when provider fallback is configured
every named file declares at least one model
no two named files claim the same model
YAML indentation is valid

Troubleshooting

The router rejects the policy

Check its logs for the first validation error. Common causes include:

missing strategy
unsupported strategy
unknown metric field
missing comparison operator
several operators on one metric
missing provider engine or model
provider API key not configured
invalid YAML indentation

A policy file is ignored

For a named file, check that:

its extension is .yaml or .yml
it contains a non-empty models array
another file does not already claim one of its models
it passed YAML and policy validation

A model uses the global policy

Confirm that the model name in the file exactly matches the model registered by the agent:

curl \
  -H "Authorization: Bearer <admin-api-key>" \
  http://localhost:8080/admin/models \
  -H "Authorization: Bearer <admin-api-key>" \
  | jq -r '.data[].id'

Model names are case-sensitive.

A gate does not exclude an agent

Inspect the live value:

curl \
  -H "Authorization: Bearer <admin-api-key>" \
  http://localhost:8080/admin/routing-table \
  -H "Authorization: Bearer <admin-api-key>" \
  | jq '.agents[]'

The metric may be unavailable, use another unit, or remain unset until the agent has served requests.

A changed policy does not take effect

File changes are not watched continuously. Send SIGHUP:

sudo kill -HUP "$(pgrep hivenet-router)"

Then inspect the router logs for the reload result.

Next steps

Fallback chains

Build ordered local alternatives with independent filters and retry budgets.

Provider fallback

Configure OpenAI or Anthropic as a final fallback for chat requests.

Policy gates

Review practical gate patterns for health, latency, cache, and hardware pressure.

​Complete structure

​Top-level fields

​Admission control fields

​mode

​max_input_tokens and images_max

​admit_budget_tokens and max_inflight

​shed_if

​Cross-config validation

​Primary routing policy

​Static matching

​Engine values

​Match logic

​Tag logic

​Match every eligible agent

​Dynamic metric gates

​Comparison operators

​Missing metrics

​Metric units

​Universal metrics

​Capacity utilization

​Success rate

​Smoothed latency

​Consecutive failures

​Engine metrics

​Cache pressure

​Backend queue

​Time to first token

​Hardware metrics

​Ranking strategy

​Forward attempts

​Fallback chain

​Provider fallback

​Global policy file

​Per-model policy directory

​_default.yaml

​Named per-model files

​Model ownership conflicts

​Per-model file errors

​Policy selection

​Reload policies with SIGHUP

​Manage policies through the admin API

​Global policy

​Named policies

​Global example

​Per-model example

​Validation checklist

​Troubleshooting

​The router rejects the policy

​A policy file is ignored

​A model uses the global policy

​A gate does not exclude an agent

​A changed policy does not take effect

​Next steps

Fallback chains

Provider fallback

Policy gates

Complete structure

Top-level fields

Admission control fields

`mode`

`max_input_tokens` and `images_max`

`admit_budget_tokens` and `max_inflight`

`shed_if`

Cross-config validation

Primary routing policy

Static matching

Engine values

Match logic

Tag logic

Match every eligible agent

Dynamic metric gates

Comparison operators

Missing metrics

Metric units

Universal metrics

Capacity utilization

Success rate

Smoothed latency

Consecutive failures

Engine metrics

Cache pressure

Backend queue

Time to first token

Hardware metrics

Ranking strategy

Forward attempts

Fallback chain

Provider fallback

Global policy file

Per-model policy directory

`_default.yaml`

Named per-model files

Model ownership conflicts

Per-model file errors

Policy selection

Reload policies with SIGHUP

Manage policies through the admin API

Global policy

Named policies

Global example

Per-model example

Validation checklist

Troubleshooting

The router rejects the policy

A policy file is ignored

A model uses the global policy

A gate does not exclude an agent

A changed policy does not take effect

Next steps