Engine metrics - Hivenet Router

Hivenet Router agents collect operational metrics from supported inference engines and send them to the router. These metrics describe the state of the backend scheduler and model-serving process, including:

KV-cache pressure
running and waiting requests
request preemptions
time to first token
inter-token latency
prompt and generation lengths
completion finish reasons
llama.cpp token throughput

The router exposes the latest values through Prometheus. A subset is also available through the administration routing table.

Engine metrics describe the inference backend.They are separate from Hivenet Router’s universal request metrics and the host-level GPU, CPU, and memory metrics collected by the agent.

How engine metrics flow

For a supported engine, the agent:

requests the backend’s /metrics endpoint
extracts the metrics Hivenet Router understands
caches the latest successful snapshot
sends that snapshot through routing signals and heartbeats
continues serving requests when a metrics scrape fails

The router:

stores the current engine snapshot
exposes scalar values through /admin/routing-table
publishes Prometheus gauges, counters, and histograms
makes supported scalar fields available to routing-policy gates

Inference engine
  │
  │ GET /metrics
  ▼
Hivenet Router agent
  │
  │ routing signals and heartbeats
  ▼
Hivenet Router router
  │
  ├── /admin/routing-table
  └── /metrics

Prometheus scrapes the router rather than every inference host.

Collection interval

The default engine scrape interval is:

500 milliseconds

Change it on the agent:

./bin/hivenet-agent \
  --engine-sample-interval 1s \
  ...

Each scrape has a five-second timeout. The default routing-signal interval is also:

500 milliseconds

This lets a newly collected snapshot reach the router without an additional heartbeat delay. If the routing-signal interval is shorter than the engine scrape interval, the agent may send the same cached snapshot more than once.

Scrape failures and stale values

Engine metric failures do not stop inference routing. If a scrape fails:

the agent keeps the previous successful snapshot
routing signals continue carrying that cached data
the agent logs one warning when scraping changes from working to failing
it logs a recovery message after a later successful scrape
request forwarding continues

The warning resembles:

Engine metrics scrape started failing — routing signals will carry stale data

A previously reported metric may remain at its last successful value while scraping is failing.The current metric set does not include a dedicated engine-snapshot age gauge. Correlate suspicious values with agent logs, backend health, and heartbeat freshness.

A metric that an engine has never reported is absent rather than emitted as zero. When an agent disconnects and is removed from the router, Hivenet Router removes its engine series.

Engine support

The current implementation collects engine metrics from:

vLLM
SGLang
llama.cpp

The following integrations do not currently provide engine-specific metrics:

Ollama
Infinity
custom engines

They still report Hivenet Router request history, latency, capacity, and available host hardware metrics.

Metric availability

Metric	vLLM	SGLang	llama.cpp
KV-cache utilization	Yes	Yes	Yes
Running requests	Yes	Yes	Yes
Waiting requests	Yes	Yes	Yes
Preemptions	Yes	No	No
Average TTFT	Yes	Yes	Yes
P90 TTFT	Yes	Yes	Yes
Raw TTFT histogram	Yes	No	No
Average ITL	Yes	No	Yes
P90 ITL	Yes	No	Yes
Raw ITL histogram	Yes	No	No
Prompt-length histogram	Yes	No	No
Generation-length histogram	Yes	No	No
Finish reasons	Yes	No	No
Generation throughput	No	No	Yes
Prompt throughput	No	No	Yes

SGLang and llama.cpp expose histogram data that Hivenet Router uses to calculate their local average and P90 scalar values.In the current implementation, raw histogram snapshots are re-exported through Prometheus only for vLLM.

Enable backend metrics

vLLM
SGLang
llama.cpp

vLLM exposes its Prometheus endpoint without an additional Hivenet Router-specific flag.Check it:

curl http://localhost:8888/metrics \
  | head

Start SGLang with:

--enable-metrics

For example:

python3 -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 3000 \
  --enable-metrics

Check it:

curl http://localhost:3000/metrics \
  | head

Start llama.cpp with:

--metrics

For example:

llama-server \
  -m /path/to/model.gguf \
  -a "llama-3.1-8b" \
  --host 0.0.0.0 \
  --port 8888 \
  --metrics

Check it:

curl http://localhost:8888/metrics \
  | head

A missing or disabled backend metrics endpoint does not prevent the agent from registering and forwarding requests.

Prometheus labels

Engine metrics use:

peer_id
model
engine
organization
machine

as their common labels. For example:

hivenet_router_agent_engine_waiting_requests{
  model="meta-llama/Llama-3.1-8B-Instruct",
  engine="vllm"
}

These labels let you compare values by:

agent
model
engine
organization
machine

Keep agent metadata stable to avoid unnecessary time-series cardinality.

View the current engine snapshot

Use the administration routing table:

curl \
  -H "Authorization: Bearer <admin-api-key>" \
  http://localhost:8080/admin/routing-table \
  | jq '.agents[] | {
      peer_id,
      model: .metadata.model,
      engine_name: .metadata.engine,
      engine_metrics: .engine
    }'

A vLLM agent may resemble:

{
  "peer_id": "12D3KooW...",
  "model": "meta-llama/Llama-3.1-8B-Instruct",
  "engine_name": "vllm",
  "engine_metrics": {
    "kv_cache_utilization": 0.42,
    "running_requests": 5,
    "waiting_requests": 0,
    "preemptions_total": 2,
    "avg_ttft_seconds": 0.118,
    "p90_ttft_seconds": 0.231,
    "avg_itl_seconds": 0.008,
    "p90_itl_seconds": 0.015
  }
}

The values above are examples. The engine object is omitted when the router has not received an engine snapshot for that agent. The routing table currently exposes scalar engine values. Raw histograms, finish-reason counters, and llama.cpp throughput gauges are available through Prometheus.

KV-cache utilization

hivenet_router_agent_engine_kv_cache_utilization

KV-cache utilization is the fraction of the engine’s preallocated KV or token cache currently in use. The range is:

0.0 to 1.0

Backend sources are:

Backend	Source metric
vLLM	`vllm:kv_cache_usage_perc`
SGLang	`sglang:token_usage`
llama.cpp	`llamacpp:kv_cache_usage_ratio`

View current cache use:

hivenet_router_agent_engine_kv_cache_utilization

Find agents above 85%:

hivenet_router_agent_engine_kv_cache_utilization > 0.85

Average by model and engine:

avg by (model, engine) (
  hivenet_router_agent_engine_kv_cache_utilization
)

High cache utilization can lead to:

request preemption
recomputation
latency spikes
reduced throughput
backend instability under additional load

The operational meaning of a particular value depends on the engine, model, context lengths, scheduler, and workload. Avoid treating one threshold as universal across every deployment.

Running requests

hivenet_router_agent_engine_running_requests

This is the number of requests currently being processed by the inference engine. Backend sources are:

Backend	Source metric
vLLM	`vllm:num_requests_running`
SGLang	`sglang:num_running_reqs`
llama.cpp	`llamacpp:requests_processing`

View running requests:

hivenet_router_agent_engine_running_requests

Total by model:

sum by (model) (
  hivenet_router_agent_engine_running_requests
)

Running requests describe the engine scheduler. They are not necessarily identical to the Hivenet Router agent’s:

active_requests

The two values are observed at different layers and times. For example:

the agent may have accepted a request that has not entered the engine’s running batch yet
a long streaming generation can continue after Hivenet Router releases its declared capacity slot when response headers arrive
a backend may continue work briefly after a client disconnects
collection and routing snapshots may arrive at slightly different times

Waiting requests

hivenet_router_agent_engine_waiting_requests

Waiting requests are queued inside the inference-engine scheduler. Backend sources are:

Backend	Source metric
vLLM	`vllm:num_requests_waiting`
SGLang	`sglang:num_queue_reqs`
llama.cpp	`llamacpp:requests_deferred`

View backend queues:

hivenet_router_agent_engine_waiting_requests

Find agents with queued requests:

hivenet_router_agent_engine_waiting_requests > 0

Total waiting work by model:

sum by (model) (
  hivenet_router_agent_engine_waiting_requests
)

A growing waiting count normally means the backend has accepted more work than it can process immediately. This can increase:

time to first token
request latency
timeout risk
cache pressure

It is separate from Hivenet Router’s per-model capacity wait queue. One queue lives inside the inference engine; the other lives in the router before an agent is selected.

Preemptions

hivenet_router_agent_engine_preemptions_total

vLLM reports a cumulative count of requests preempted because of scheduler and KV-cache pressure. Hivenet Router exports the backend’s latest cumulative value as a Prometheus gauge, even though its name ends in _total. Backend source:

vllm:num_preemptions_total

View the current cumulative value:

hivenet_router_agent_engine_preemptions_total

Detect growth over a period:

delta(
  hivenet_router_agent_engine_preemptions_total[5m]
) > 0

Show increases by agent:

sum by (peer_id, model) (
  clamp_min(
    delta(
      hivenet_router_agent_engine_preemptions_total[5m]
    ),
    0
  )
)

Treat this metric as a gauge containing the backend’s latest cumulative value.It may fall when vLLM restarts. Ordinary counter assumptions do not fully apply.

A rising value together with high KV-cache utilization is evidence of cache pressure.

Time to first token

Time to first token, or TTFT, is the time between a request entering the inference engine and the first output token becoming available. Hivenet Router exposes two per-agent gauges:

hivenet_router_agent_engine_avg_ttft_seconds
hivenet_router_agent_engine_p90_ttft_seconds

Both are measured in seconds.

Average TTFT

hivenet_router_agent_engine_avg_ttft_seconds

This is calculated from the backend histogram’s cumulative sum and count.

P90 TTFT

hivenet_router_agent_engine_p90_ttft_seconds

The agent estimates this value from the backend histogram buckets. Hivenet Router keeps it as a scalar because routing policies need to read a current value directly during agent selection.

Raw TTFT histogram

For vLLM agents, Hivenet Router also exports:

hivenet_router_agent_engine_ttft_seconds

Prometheus exposes:

hivenet_router_agent_engine_ttft_seconds_bucket
hivenet_router_agent_engine_ttft_seconds_sum
hivenet_router_agent_engine_ttft_seconds_count

Fleet-wide P90 for vLLM:

histogram_quantile(
  0.90,
  sum by (le) (
    rate(
      hivenet_router_agent_engine_ttft_seconds_bucket[5m]
    )
  )
)

P90 by model:

histogram_quantile(
  0.90,
  sum by (le, model) (
    rate(
      hivenet_router_agent_engine_ttft_seconds_bucket[5m]
    )
  )
)

Do not average per-agent P90 gauges to calculate a fleet-wide P90.Use the raw histogram buckets where they are available.

High TTFT can result from:

a waiting scheduler queue
long prompt prefill
cold model loading
CPU-side input processing
GPU contention
cache pressure
large running batches

Establish a baseline for each model and workload rather than copying a generic target.

Inter-token latency

Inter-token latency, or ITL, is the time between generated output tokens during the decode phase. Hivenet Router exposes:

hivenet_router_agent_engine_avg_itl_seconds
hivenet_router_agent_engine_p90_itl_seconds

for:

vLLM
llama.cpp

SGLang does not currently supply ITL through this integration. Average ITL:

hivenet_router_agent_engine_avg_itl_seconds

P90 ITL:

hivenet_router_agent_engine_p90_itl_seconds

For vLLM, Hivenet Router also exports the raw histogram:

hivenet_router_agent_engine_itl_seconds

Fleet-wide vLLM P90:

histogram_quantile(
  0.90,
  sum by (le) (
    rate(
      hivenet_router_agent_engine_itl_seconds_bucket[5m]
    )
  )
)

High ITL can make streamed output feel slow or uneven. Possible causes include:

compute saturation
large running batches
long contexts
contention from other GPU workloads
memory pressure
backend configuration

Prompt-length histogram

vLLM supplies a histogram of prompt sizes:

hivenet_router_agent_engine_request_prompt_tokens

Prometheus exposes:

hivenet_router_agent_engine_request_prompt_tokens_bucket
hivenet_router_agent_engine_request_prompt_tokens_sum
hivenet_router_agent_engine_request_prompt_tokens_count

P50 prompt length:

histogram_quantile(
  0.50,
  sum by (le) (
    rate(
      hivenet_router_agent_engine_request_prompt_tokens_bucket[5m]
    )
  )
)

P90 by model:

histogram_quantile(
  0.90,
  sum by (le, model) (
    rate(
      hivenet_router_agent_engine_request_prompt_tokens_bucket[5m]
    )
  )
)

Long prompts can increase:

prefill time
TTFT
KV-cache use
memory pressure
per-request cost

Generation-length histogram

vLLM also supplies a distribution of generated output lengths:

hivenet_router_agent_engine_request_generation_tokens

P90 generation length:

histogram_quantile(
  0.90,
  sum by (le, model) (
    rate(
      hivenet_router_agent_engine_request_generation_tokens_bucket[5m]
    )
  )
)

Long generations:

occupy scheduler capacity for longer
retain KV-cache allocations
increase output-token use
can reduce effective concurrency

Use prompt and generation histograms together to understand the shape of the workload rather than relying only on request counts.

Finish reasons

vLLM reports cumulative completion counts grouped by finish reason. Hivenet Router converts changes between backend snapshots into the counter:

hivenet_router_agent_engine_request_success_total

Additional label:

finished_reason

Common values include:

stop
length
abort

Completion rate by reason:

sum by (finished_reason) (
  rate(
    hivenet_router_agent_engine_request_success_total[5m]
  )
)

Length-limited completions:

sum by (model) (
  rate(
    hivenet_router_agent_engine_request_success_total{
      finished_reason="length"
    }[5m]
  )
)

Aborted completions:

sum by (model) (
  rate(
    hivenet_router_agent_engine_request_success_total{
      finished_reason="abort"
    }[5m]
  )
)

Unlike the preemption gauge, this is a Hivenet Router Prometheus counter. The first snapshot establishes a baseline. Hivenet Router counts changes observed after it begins tracking that agent rather than importing the engine’s complete earlier lifetime count. If the backend count decreases after a restart, Hivenet Router treats the new value as post-restart activity.

llama.cpp token throughput

llama.cpp exposes two current throughput gauges:

hivenet_router_agent_engine_predicted_tps
hivenet_router_agent_engine_prompt_tps

Generation throughput

hivenet_router_agent_engine_predicted_tps

This is the backend’s recent generated-token rate.

hivenet_router_agent_engine_predicted_tps

Average by model:

avg by (model) (
  hivenet_router_agent_engine_predicted_tps
)

Prompt throughput

hivenet_router_agent_engine_prompt_tps

This is the backend’s recent prompt-ingestion rate.

hivenet_router_agent_engine_prompt_tps

These values are currently available through Prometheus but not through the administration routing-table response. A falling throughput value can indicate contention or a changed workload, but interpret it alongside:

request sizes
active and waiting requests
GPU use
context length
batch configuration
model quantization

Use engine metrics in routing policies

The supported engine metric gates are:

kv_cache_utilization
running_requests
waiting_requests
avg_ttft_seconds
p90_ttft_seconds
avg_itl_seconds
p90_itl_seconds

For example:

routing_policy:
  match:
    engine: vllm

  exclude_if:
    kv_cache_utilization:
      gt: 0.85

    waiting_requests:
      gt: 5

    p90_ttft_seconds:
      gt: 0.5

  strategy: least-loaded

Preemptions, finish reasons, request-shape histograms, and token-throughput values are available for monitoring but are not current policy-gate fields.

Missing metrics pass

When an agent does not expose a selected field, Hivenet Router skips that gate for the agent. For example:

exclude_if:
  avg_itl_seconds:
    gt: 0.05

does not exclude an SGLang agent because SGLang does not report ITL through this integration.

Combine engine-specific gates with a static engine or metadata match when the metric must be present.Missing data does not fail the gate.

Practical routing patterns

Avoid cache pressure

routing_policy:
  match:
    engine: vllm

  exclude_if:
    kv_cache_utilization:
      gt: 0.85

  strategy: least-loaded

Avoid backend queues

routing_policy:
  exclude_if:
    waiting_requests:
      gt: 5

  strategy: least-loaded

Protect TTFT

routing_policy:
  exclude_if:
    p90_ttft_seconds:
      gt: 0.5

  strategy: least-loaded

Use a relaxed fallback

routing_policy:
  match:
    engine: vllm

  exclude_if:
    kv_cache_utilization:
      gt: 0.85
    waiting_requests:
      gt: 5

  strategy: least-loaded

fallback_chain:
  - name: relaxed-vllm

    match:
      engine: vllm

    exclude_if:
      kv_cache_utilization:
        gt: 0.95
      waiting_requests:
        gt: 20

    strategy: least-loaded

This lets Hivenet Router prefer low-pressure agents while retaining a controlled local fallback under heavier load.

Grafana

The provisioned router dashboard includes engine panels for:

KV-cache utilization
running and waiting requests
preemptions
average and P90 TTFT
average and P90 ITL
llama.cpp prompt and generation throughput

Blank values may mean:

the backend does not support the metric
backend metrics are not enabled
no observations exist yet
the agent has not completed a successful scrape
dashboard filters exclude the series

See Grafana dashboards for provisioning and dashboard behavior.

Alerting examples

The thresholds below are examples. Establish appropriate limits from your models, hardware, and service objectives.

KV-cache pressure

groups:
  - name: hivenet-router-engine

    rules:
      - alert: HivenetRouterKVCachePressure

        expr: |
          hivenet_router_agent_engine_kv_cache_utilization > 0.90

        for: 2m

        labels:
          severity: warning

        annotations:
          summary: >-
            High KV-cache use on {{ $labels.peer_id }}

Backend queue growth

      - alert: HivenetRouterEngineQueue

        expr: |
          hivenet_router_agent_engine_waiting_requests > 10

        for: 2m

        labels:
          severity: warning

        annotations:
          summary: >-
            Backend queue on {{ $labels.peer_id }}

TTFT degradation

      - alert: HivenetRouterTTFTDegradation

        expr: |
          hivenet_router_agent_engine_p90_ttft_seconds > 0.5

        for: 5m

        labels:
          severity: warning

        annotations:
          summary: >-
            High P90 TTFT on {{ $labels.peer_id }}

vLLM preemptions increasing

      - alert: HivenetRouterVLLMPreemptions

        expr: |
          clamp_min(
            delta(
              hivenet_router_agent_engine_preemptions_total[5m]
            ),
            0
          ) > 0

        for: 5m

        labels:
          severity: warning

        annotations:
          summary: >-
            vLLM preemptions are increasing on {{ $labels.peer_id }}

Troubleshooting

No engine metrics appear

Check whether the selected engine supports metrics:

vllm
sglang
llamacpp

Then check the backend directly:

curl -i \
  http://localhost:<backend-port>/metrics

Inspect the agent logs:

systemd
Docker

sudo journalctl \
  -u hivenet-agent \
  | grep -i "engine metrics"

docker logs hivenet-agent \
  2>&1 \
  | grep -i "engine metrics"

SGLang metrics are missing

Start SGLang with:

--enable-metrics

Restart SGLang, then check:

curl http://localhost:3000/metrics \
  | head

llama.cpp metrics are missing

Start llama.cpp with:

--metrics

Restart the server, then check:

curl http://localhost:8888/metrics \
  | head

Values appear stale

Check the agent logs for:

Engine metrics scrape started failing

Test the backend metrics endpoint locally. The agent deliberately retains its latest successful snapshot during a scrape failure. Restarting the agent clears its local cache, but the better fix is to restore the backend metrics endpoint.

TTFT or ITL is absent

The backend may not expose that metric, or it may not have recorded any observations yet. Check:

engine type
backend metrics configuration
whether requests have completed
whether the expected histogram exists on the backend

SGLang does not provide ITL through the current integration.

Raw histogram queries are empty

The current Hivenet Router implementation re-exports raw engine histograms from vLLM. SGLang and llama.cpp currently supply scalar average and P90 values but not raw histogram series through the router. Filter for vLLM:

hivenet_router_agent_engine_ttft_seconds_count{
  engine="vllm"
}

Preemption calculations become negative

The backend cumulative value can reset when vLLM restarts. Use:

clamp_min(
  delta(
    hivenet_router_agent_engine_preemptions_total[5m]
  ),
  0
)

rather than treating the series as a normal Prometheus counter.

Engine values exist but policy gates do not behave as expected

Check:

policy field spelling
units
current routing-table value
whether the agent passes earlier routing gates
whether the metric is missing rather than zero
whether cached data is stale after scrape failure

Policy utilization fields use fractions rather than percentages.

Next steps

Latency tracking

Understand Hivenet Router’s RFC 6298 SRTT and RTTVAR measurements.

Hardware-aware routing

Combine engine pressure with GPU, CPU, memory, and hardware-tier policies.

Grafana dashboards

Explore the provisioned engine tables and time-series panels.

​How engine metrics flow

​Collection interval

​Scrape failures and stale values

​Engine support

​Metric availability

​Enable backend metrics

​Prometheus labels

​View the current engine snapshot

​KV-cache utilization

​Running requests

​Waiting requests

​Preemptions

​Time to first token

​Average TTFT

​P90 TTFT

​Raw TTFT histogram

​Inter-token latency

​Prompt-length histogram

​Generation-length histogram

​Finish reasons

​llama.cpp token throughput

​Generation throughput

​Prompt throughput

​Use engine metrics in routing policies

​Missing metrics pass

​Practical routing patterns

​Avoid cache pressure

​Avoid backend queues

​Protect TTFT

​Use a relaxed fallback

​Grafana

​Alerting examples

​KV-cache pressure

​Backend queue growth

​TTFT degradation

​vLLM preemptions increasing

​Troubleshooting

​No engine metrics appear

​SGLang metrics are missing

​llama.cpp metrics are missing

​Values appear stale

​TTFT or ITL is absent

​Raw histogram queries are empty

​Preemption calculations become negative

​Engine values exist but policy gates do not behave as expected

​Next steps

Latency tracking

Hardware-aware routing

Grafana dashboards

How engine metrics flow

Collection interval

Scrape failures and stale values

Engine support

Metric availability

Enable backend metrics

Prometheus labels

View the current engine snapshot

KV-cache utilization

Running requests

Waiting requests

Preemptions

Time to first token

Average TTFT

P90 TTFT

Raw TTFT histogram

Inter-token latency

Prompt-length histogram

Generation-length histogram

Finish reasons

llama.cpp token throughput

Generation throughput

Prompt throughput

Use engine metrics in routing policies

Missing metrics pass

Practical routing patterns

Avoid cache pressure

Avoid backend queues

Protect TTFT

Use a relaxed fallback

Grafana

Alerting examples

KV-cache pressure

Backend queue growth

TTFT degradation

vLLM preemptions increasing

Troubleshooting

No engine metrics appear

SGLang metrics are missing

llama.cpp metrics are missing

Values appear stale

TTFT or ITL is absent

Raw histogram queries are empty

Preemption calculations become negative

Engine values exist but policy gates do not behave as expected

Next steps