Hardware-aware routing - Hivenet Router

Hardware-aware routing lets Hivenet Router prefer particular infrastructure and avoid agents under resource pressure. It combines two kinds of information:

static metadata that describes the intended hardware tier
live metrics that describe the current state of the agent host and inference engine

Static preference
  gpu_model, tags, region
          │
          ▼
Live protection
  temperature, utilization, VRAM,
  CPU, memory, cache, latency
          │
          ▼
least-loaded ranking

Hivenet Router does not automatically calculate the fastest, cheapest, or most energy-efficient hardware.You define the hardware tiers and operational limits. Hivenet Router applies those rules to the metadata and metrics reported by agents.

Static and dynamic hardware signals

Mechanism	Policy section	Purpose
GPU model	`match.gpu_model`	Select an operator-defined GPU tier
Tags	`match.tags`	Describe VRAM class, workload, cost tier, or operational role
Region	`match.region`	Prefer an operator-defined location
GPU temperature	`exclude_if.gpu_temperature_c`	Avoid overheated agents
GPU utilization	`exclude_if.gpu_util_percent`	Avoid highly active GPUs
VRAM utilization	`exclude_if.gpu_vram_used_percent`	Preserve device-memory headroom
CPU utilization	`exclude_if.cpu_usage_percent`	Avoid CPU-saturated hosts
System memory	`exclude_if.memory_used_percent`	Avoid hosts under RAM pressure
Engine cache	`exclude_if.kv_cache_utilization`	Avoid inference-cache pressure
Engine queue	`exclude_if.waiting_requests`	Avoid backends with queued work
SRTT	`exclude_if.srtt`	Avoid agents with high observed request latency
TTFT and ITL	`exclude_if._ttft_seconds`, `exclude_if._itl_seconds`	Avoid degraded generation latency

The static fields describe what an agent is. The dynamic fields describe how it is behaving now.

Configure GPU metadata

The hardware-specific static match field is:

gpu_model:

Set the value when starting the agent:

./bin/hivenet-agent \
  --gpu-model "NVIDIA H100" \
  ...

The environment-variable equivalent is:

export HIVENET_ROUTER_GPU_MODEL="NVIDIA H100"

Hivenet Router does not currently derive this registration value from NVML. The operator is responsible for assigning it consistently.

gpu_model matching is exact and case-sensitive.These values are different:

NVIDIA H100
H100
nvidia h100

Check the registered value:

curl \
  -H "Authorization: Bearer <admin-api-key>" \
  http://localhost:8080/admin/routing-table \
  | jq '.agents[] | {
      model: .metadata.model,
      machine: .metadata.machine,
      gpu_model: .metadata.gpu_model
    }'

Prefer a GPU tier

Use the primary policy for the preferred tier and fallback steps for acceptable alternatives.

routing_policy:
  match:
    gpu_model: "NVIDIA H100"

  strategy: least-loaded

fallback_chain:
  - name: a100-tier

    match:
      gpu_model: "NVIDIA A100"

    strategy: least-loaded

  - name: a10-tier

    match:
      gpu_model: "NVIDIA A10"

    strategy: least-loaded

This policy prefers:

H100 agents
A100 agents
A10 agents

Every agent must still:

serve the requested model
serve the required capability
be healthy
have free declared capacity

A hardware tier does not override the normal routing constraints.

Represent VRAM tiers

Hivenet Router cannot statically match total VRAM or require an absolute number of free bytes. This is not supported:

match:
  vram_gb: 80

The live VRAM gate uses a fraction of each device’s total VRAM:

exclude_if:
  gpu_vram_used_percent:
    gt: 0.9

It does not tell the policy whether the device has 24 GB, 40 GB, or 80 GB in total.

Use operator-defined tags

When total VRAM matters, describe the tier explicitly:

./bin/hivenet-agent \
  --gpu-model "NVIDIA A100" \
  --tags production,vram-80gb \
  ...

Then match it:

routing_policy:
  match:
    tags:
      - vram-80gb

  strategy: least-loaded

This is safer than assuming one GPU model always has one memory size. Hardware families may contain several VRAM variants.

Every tag listed in a policy must exist on the agent.Tag matching uses AND logic.

Thermal-aware routing

Exclude agents above your chosen temperature threshold:

routing_policy:
  match:
    engine: vllm

  exclude_if:
    gpu_temperature_c:
      gt: 80

  strategy: least-loaded

The threshold is in degrees Celsius. The appropriate value depends on:

GPU model
cooling design
ambient temperature
normal operating range
throttling behavior
hardware policy

Do not copy one thermal limit across unrelated hardware without testing.

Use graduated thermal limits

A fallback step can retain hotter agents as a last local option:

routing_policy:
  match:
    tags:
      - production

  exclude_if:
    gpu_temperature_c:
      gt: 75

  strategy: least-loaded

fallback_chain:
  - name: relaxed-thermal

    match:
      tags:
        - production

    exclude_if:
      gpu_temperature_c:
        gt: 82

    strategy: least-loaded

This policy:

prefers agents below or equal to 75°C
falls back to agents below or equal to 82°C
exhausts the local chain when every agent exceeds the relaxed limit

The fallback may intentionally send traffic to hardware that the primary policy considered too hot.Use the relaxed threshold only when continued service is preferable to rejecting or externally falling back.

Protect VRAM and engine cache

Host-level VRAM and engine KV cache describe different kinds of pressure.

routing_policy:
  match:
    engine: vllm

  exclude_if:
    gpu_vram_used_percent:
      gt: 0.9

    kv_cache_utilization:
      gt: 0.85

  strategy: least-loaded

Signal	What it describes
`gpu_vram_used_percent`	Total device memory used by every process and allocation
`kv_cache_utilization`	Fraction of the inference engine’s allocated KV cache in use

A backend can have:

high total VRAM use but moderate KV-cache pressure
low host-wide VRAM use but a nearly full engine cache
pressure in both layers

Using both signals gives a better picture than treating them as interchangeable.

Avoid highly utilized GPUs

routing_policy:
  exclude_if:
    gpu_util_percent:
      gt: 0.95

  strategy: least-loaded

Policy utilization values use fractions:

0.95 = 95%

Prometheus exposes the corresponding hardware value on a 0 to 100 scale:

hivenet_router_agent_gpu_utilization_percent > 95

GPU utilization and Hivenet Router capacity are independent signals.The hard capacity gate uses:

active requests >= declared capacity

A GPU can be highly utilized before the agent reaches capacity, or lightly utilized while every declared request slot is occupied.For streaming requests, Hivenet Router currently releases the declared capacity slot when response headers arrive while backend generation continues. Use engine running and waiting requests, KV-cache pressure, TTFT, and ITL alongside capacity-based gates for streaming-heavy workloads.

Avoid host CPU and memory pressure

Inference backends can depend on CPU and system memory for:

tokenization
request parsing
input preprocessing
model loading
memory-mapped files
network processing
CPU offloading

Use node-level gates:

routing_policy:
  exclude_if:
    cpu_usage_percent:
      gt: 0.9

    memory_used_percent:
      gt: 0.9

  strategy: least-loaded

These values use fractions from 0.0 to 1.0. The metrics describe the whole host, not only the inference process. Other workloads on the same machine can therefore exclude the agent.

Multi-GPU agents

When an agent reports several GPUs, Hivenet Router evaluates the highest value across the reported devices for:

temperature
compute utilization
VRAM-used fraction

For example:

GPU	Temperature	VRAM use
GPU 0	68°C	61%
GPU 1	84°C	72%

This policy:

exclude_if:
  gpu_temperature_c:
    gt: 82

excludes the complete agent because GPU 1 is above the threshold. This conservative behavior is useful when one unhealthy or overloaded GPU can affect the complete multi-GPU backend. Use --gpu-devices-file when the agent should report only the devices assigned to its inference engine.

Missing metrics pass

A hardware gate is skipped when its metric is unavailable for an agent. For example:

exclude_if:
  gpu_temperature_c:
    gt: 82

does not exclude:

a CPU-only agent
an agent without working NVML
an agent that has not yet reported a hardware snapshot

Missing data passes the gate.A GPU gate alone does not guarantee that traffic goes only to GPU agents with working hardware telemetry.

When metric availability is required, combine the gate with known static metadata:

routing_policy:
  match:
    tags:
      - gpu
      - hardware-metrics-enabled

  exclude_if:
    gpu_temperature_c:
      gt: 82

  strategy: least-loaded

The tags remain operator assertions. Hivenet Router does not verify that the labels accurately describe the machine.

Hardware does not change ranking

The only implemented ranking strategy is:

strategy: least-loaded

Hivenet Router does not currently support strategies such as:

coolest-gpu
lowest-vram
lowest-power
highest-throughput
lowest-srtt

Hardware and latency values can exclude agents. They do not sort the survivors. For example:

exclude_if:
  gpu_temperature_c:
    gt: 80

removes agents above 80°C. The remaining agents are still ranked by:

active requests / declared capacity

Use fallback steps to express ordered hardware preferences.

Real-time inference

A real-time policy can combine a preferred hardware tier with latency and pressure gates:

routing_policy:
  match:
    tags:
      - realtime
    gpu_model: "NVIDIA H100"
    engine: vllm

  exclude_if:
    srtt:
      gt: 200

    p90_ttft_seconds:
      gt: 0.5

    waiting_requests:
      gt: 0

    gpu_temperature_c:
      gt: 80

  strategy: least-loaded

fallback_chain:
  - name: relaxed-realtime

    match:
      tags:
        - realtime
      engine: vllm

    exclude_if:
      srtt:
        gt: 750

      waiting_requests:
        gt: 5

      gpu_temperature_c:
        gt: 84

    strategy: least-loaded

Units are:

Field	Unit
`srtt`	Milliseconds
`p90_ttft_seconds`	Seconds
`gpu_temperature_c`	Celsius

SRTT and TTFT are different:

SRTT is measured by Hivenet Router across the router-agent-backend request path.
TTFT is reported by the inference engine.

Use both only after establishing representative baselines for the workload.

Batch inference

Batch workloads may favor high-memory agents and tolerate more queueing or latency.

routing_policy:
  match:
    tags:
      - batch
      - vram-80gb

  exclude_if:
    gpu_vram_used_percent:
      gt: 0.8

    memory_used_percent:
      gt: 0.9

  strategy: least-loaded

The tag expresses the static VRAM tier. The gates preserve device and host-memory headroom. A batch policy may tolerate backend queues that would be unsuitable for interactive traffic:

exclude_if:
  waiting_requests:
    gt: 20

Choose the threshold from actual throughput and completion-time requirements.

Prefer a lower-cost tier

Hivenet Router does not know GPU prices, contracts, or operating costs. Represent the intended economic preference through operator-defined metadata:

./bin/hivenet-agent \
  --tags production,cost-tier-low \
  ...

Then build an ordered chain:

routing_policy:
  match:
    tags:
      - cost-tier-low

  strategy: least-loaded

fallback_chain:
  - name: standard-tier

    match:
      tags:
        - cost-tier-standard

    strategy: least-loaded

  - name: premium-tier

    match:
      tags:
        - cost-tier-premium

    strategy: least-loaded

This expresses your own cost classification without hard-coding assumptions about which GPU is cheaper.

The tags do not affect billing.They are routing metadata maintained by the operator.

Route by location or energy policy

Hivenet Router does not measure:

electricity source
carbon intensity
renewable-energy share
water use
facility efficiency

Do not infer those properties from a broad region name. When your organization has verified location or facility data, express it through controlled metadata:

./bin/hivenet-agent \
  --region "EU-France" \
  --tags production,approved-energy-site \
  ...

Then match it:

routing_policy:
  match:
    region: "EU-France"
    tags:
      - approved-energy-site

  strategy: least-loaded

This routes according to the metadata you supplied. Hivenet Router does not independently verify the environmental claim or update it from a live carbon-intensity source.

Power-aware monitoring

GPU power is available through Prometheus:

hivenet_router_agent_gpu_power_watts

Total current GPU power by machine:

sum by (machine) (
  hivenet_router_agent_gpu_power_watts
)

Power by region:

sum by (region) (
  hivenet_router_agent_gpu_power_watts
)

Power is not currently a valid exclude_if field. This is not supported:

exclude_if:
  gpu_power_watts:
    gt: 350

A ratio such as requests per watt can be useful as a rough dashboard signal:

sum(
  rate(
    hivenet_router_routing_requests_routed_total[5m]
  )
)
/
clamp_min(
  sum(
    hivenet_router_agent_gpu_power_watts
  ),
  1
)

This ratio is not a complete energy-efficiency measure.It mixes routed-request rate with instantaneous power and does not account for prompt size, output length, model quality, idle power, CPU use, or completed work.

Combine hardware and engine state

A production policy can protect several layers at once:

routing_policy:
  match:
    engine: vllm
    tags:
      - production
      - high-memory

  exclude_if:
    capacity_utilization:
      gte: 0.8

    kv_cache_utilization:
      gt: 0.85

    waiting_requests:
      gt: 5

    gpu_temperature_c:
      gt: 82

    gpu_util_percent:
      gt: 0.95

    gpu_vram_used_percent:
      gt: 0.9

    memory_used_percent:
      gt: 0.9

    cpu_usage_percent:
      gt: 0.9

    srtt:
      gt: 750

  strategy: least-loaded
  max_tries: 3

fallback_chain:
  - name: relaxed-production

    match:
      engine: vllm
      tags:
        - production

    exclude_if:
      kv_cache_utilization:
        gt: 0.95

      waiting_requests:
        gt: 20

      gpu_temperature_c:
        gt: 86

      memory_used_percent:
        gt: 0.95

      srtt:
        gt: 2000

    strategy: least-loaded
    max_tries: 2

  - name: any-local-agent

    match: {}

    strategy: least-loaded
    max_tries: 2

An agent is excluded when it violates any field in one exclude_if block. A long list of gates can drain the candidate pool more often than expected. Add rules gradually and monitor which gate exhausts each step.

Monitor hardware-aware routing

Current GPU state

hivenet_router_agent_gpu_utilization_percent

100
*
hivenet_router_agent_gpu_vram_used_bytes
/
clamp_min(
  hivenet_router_agent_gpu_vram_total_bytes,
  1
)

hivenet_router_agent_gpu_temperature_celsius

Current host state

hivenet_router_agent_cpu_usage_percent

hivenet_router_agent_memory_used_percent

Engine pressure

hivenet_router_agent_engine_kv_cache_utilization

hivenet_router_agent_engine_waiting_requests

Routing outcomes

sum by (model) (
  rate(
    hivenet_router_policy_primary_routed_total[5m]
  )
)

sum by (model) (
  rate(
    hivenet_router_policy_fallback_routed_total[5m]
  )
)

sum by (model) (
  rate(
    hivenet_router_policy_exhausted_total[5m]
  )
)

A rising fallback or exhaustion rate after adding hardware gates may mean:

thresholds are too strict
hardware metrics are genuinely degraded
static metadata is inconsistent
the preferred tier lacks capacity
one metric is behaving differently from your assumption

Diagnose an excluded agent

Inspect the current snapshot:

curl \
  -H "Authorization: Bearer <admin-api-key>" \
  http://localhost:8080/admin/routing-table \
  | jq '.agents[] | {
      model: .metadata.model,
      machine: .metadata.machine,
      gpu_model: .metadata.gpu_model,
      tags: .metadata.tags,
      status: .status,
      engine: .engine,
      hardware: .hardware,
      latency: {
        srtt_ms: .universal.srtt_ms,
        success_rate: .universal.success_rate
      }
    }'

Enable detailed policy logs:

export GOLOG_LOG_LEVEL="policy=debug,router=debug"

A policy-exhaustion record identifies which gate removed the candidates. For example:

routing_policy:
gate[7] exclude_if [
  gpu_temperature_c=1,
  gpu_vram_used_percent=2
]

This indicates that one candidate violated the temperature gate and two violated the VRAM gate.

Test a policy safely

A practical test sequence is:

Record current hardware and engine values.
Add one gate whose threshold should pass.
Reload the policy.
Send representative traffic.
Confirm primary routing continues.
Lower the threshold in a controlled environment.
Confirm that the intended fallback step receives traffic.
Inspect fallback and exhaustion metrics.
Restore the production threshold.
Add the next gate only after the first behaves as expected.

Avoid introducing many hardware gates at the same time. When the policy drains, it becomes harder to identify which assumption was wrong.

Capacity remains operator-defined

Hivenet Router does not derive agent capacity from:

total VRAM
free VRAM
GPU count
model parameters
KV-cache size
power limit

The agent’s:

--capacity

remains an operator-set concurrency limit. Do not rely on a general formula such as:

free VRAM / estimated KV cache per request

as the only capacity calculation. Real capacity depends on:

model architecture
quantization
context distribution
engine implementation
batching and scheduler settings
tensor parallelism
prefix reuse
CPU work
latency objectives
workload mix

Benchmark representative traffic and raise capacity gradually while observing engine queues, cache pressure, TTFT, ITL, hardware use, and failures.

Design guidance

Prefer explicit metadata

Use stable tags for operational facts such as:

vram-80gb
cost-tier-low
approved-energy-site
interactive
batch

Do not overload broad labels such as region or gpu_model with several unrelated meanings.

Keep policy and monitoring separate

A routing gate decides whether an agent should receive the current request. An alert decides whether an operator should investigate. The appropriate thresholds may differ. For example:

route away from a GPU briefly above 82°C
alert only when it remains above 85°C for five minutes

Use relaxed fallbacks deliberately

A relaxed step accepts a condition that the primary step rejected. Document why that compromise is acceptable.

Avoid false precision

Hardware values are sampled periodically and may be slightly stale by the time a request is routed. Use thresholds with practical headroom rather than treating a single decimal point as a precise safety boundary.

Troubleshooting

A GPU-tier policy matches nothing

Check the registered metadata:

curl \
  -H "Authorization: Bearer <admin-api-key>" \
  http://localhost:8080/admin/routing-table \
  | jq -r '.agents[].metadata.gpu_model'

Check exact spelling, capitalization, and whitespace. Hivenet Router does not populate gpu_model automatically.

A VRAM requirement cannot be expressed

Hivenet Router does not support an absolute VRAM match. Use an operator-defined tag such as:

vram-80gb

or a carefully standardized gpu_model value. Use gpu_vram_used_percent only for live pressure, not total device capacity.

A hardware gate does not exclude an agent

The metric may be missing. Check the routing-table hardware object. Also verify the units:

Prometheus	Policy
`95`	`0.95`
`82°C`	`82`

Missing metrics pass.

One hot GPU excludes a multi-GPU agent

This is expected. Hivenet Router uses the highest temperature, utilization, and VRAM fraction across the GPUs reported by the agent. Restrict collection with --gpu-devices-file if the agent should represent only a subset of the host’s devices.

Power cannot be used in `exclude_if`

GPU power is currently an observability metric only. Use static hardware metadata, tags, or one of the supported dynamic gate fields.

The preferred tier is skipped

Check the earlier hard constraints:

requested model
capability
agent health
static metadata
previous failures
declared capacity

A hardware match does not make an agent eligible when it fails one of those checks.

Fallback traffic rises after adding gates

Inspect:

policy-exhaustion logs
current metric distributions
missing-metric behavior
preferred-tier capacity
threshold units
whether one agent reports a persistent outlier

Relax one condition at a time rather than removing the complete policy.

Hardware data appears stale

Check agent logs for collection errors and confirm the latest heartbeat. The current system does not expose a dedicated hardware-snapshot age metric. A healthy heartbeat does not prove that every recent hardware sample succeeded.

Next steps

Prometheus metrics

Query the live hardware, engine, latency, capacity, and routing signals used in these policies.

Policy gates

Review the exact evaluation behavior and all supported gate fields.

Hardware metrics

Understand how GPU, CPU, memory, temperature, and power values are collected.

​Static and dynamic hardware signals

​Configure GPU metadata

​Prefer a GPU tier

​Represent VRAM tiers

​Use operator-defined tags

​Thermal-aware routing

​Use graduated thermal limits

​Protect VRAM and engine cache

​Avoid highly utilized GPUs

​Avoid host CPU and memory pressure

​Multi-GPU agents

​Missing metrics pass

​Hardware does not change ranking

​Real-time inference

​Batch inference

​Prefer a lower-cost tier

​Route by location or energy policy

​Power-aware monitoring

​Combine hardware and engine state

​Monitor hardware-aware routing

​Current GPU state

​Current host state

​Engine pressure

​Routing outcomes

​Diagnose an excluded agent

​Test a policy safely

​Capacity remains operator-defined

​Design guidance

​Prefer explicit metadata

​Keep policy and monitoring separate

​Use relaxed fallbacks deliberately

​Avoid false precision

​Troubleshooting

​A GPU-tier policy matches nothing

​A VRAM requirement cannot be expressed

​A hardware gate does not exclude an agent

​One hot GPU excludes a multi-GPU agent

​Power cannot be used in exclude_if

​The preferred tier is skipped

​Fallback traffic rises after adding gates

​Hardware data appears stale

​Next steps

Prometheus metrics

Policy gates

Hardware metrics

Static and dynamic hardware signals

Configure GPU metadata

Prefer a GPU tier

Represent VRAM tiers

Use operator-defined tags

Thermal-aware routing

Use graduated thermal limits

Protect VRAM and engine cache

Avoid highly utilized GPUs

Avoid host CPU and memory pressure

Multi-GPU agents

Missing metrics pass

Hardware does not change ranking

Real-time inference

Batch inference

Prefer a lower-cost tier

Route by location or energy policy

Power-aware monitoring

Combine hardware and engine state

Monitor hardware-aware routing

Current GPU state

Current host state

Engine pressure

Routing outcomes

Diagnose an excluded agent

Test a policy safely

Capacity remains operator-defined

Design guidance

Prefer explicit metadata

Keep policy and monitoring separate

Use relaxed fallbacks deliberately

Avoid false precision

Troubleshooting

A GPU-tier policy matches nothing

A VRAM requirement cannot be expressed

A hardware gate does not exclude an agent

One hot GPU excludes a multi-GPU agent

Power cannot be used in `exclude_if`

The preferred tier is skipped

Fallback traffic rises after adding gates

Hardware data appears stale

Next steps