Prometheus metrics - Hivenet Router

Hivenet Router exports router, agent, inference-engine, hardware, policy, quota, and HTTP metrics in Prometheus format.

GET /metrics

The metrics server runs separately from the main HTTP API and listens on port 2112 by default.

curl http://localhost:2112/metrics

The metrics endpoint does not have built-in authentication.Restrict it to Prometheus and trusted operators through a private network, firewall, reverse proxy, or equivalent network control.

Start the metrics server

The router starts the Prometheus endpoint automatically. The default address is:

:2112

Change it with:

./bin/hivenet-router \
  --metrics-port :9212 \
  ...

The metrics endpoint is then available at:

http://<router-host>:9212/metrics

Configure Prometheus

A minimal scrape configuration is:

global:
  scrape_interval: 5s
  evaluation_interval: 5s

scrape_configs:
  - job_name: hivenet-router

    static_configs:
      - targets:
          - router.example.internal:2112

Reload Prometheus after changing its configuration. With the repository’s Docker Compose stack, Prometheus reaches the router through the internal Compose network:

scrape_configs:
  - job_name: hivenet-router

    static_configs:
      - targets:
          - router:2112

Port 2112 does not need to be published to the host when Prometheus runs in the same Compose network.

Check the scrape target

From the Prometheus interface, open:

Status → Targets

Or query the Prometheus API:

curl \
  http://localhost:9090/api/v1/targets \
  | jq '.data.activeTargets[] | {
      scrape_url: .scrapeUrl,
      health,
      last_error: .lastError
    }'

A healthy target reports:

{
  "health": "up",
  "last_error": ""
}

Check that Hivenet Router metrics exist:

curl -s \
  http://localhost:2112/metrics \
  | grep '^hivenet_router_'

How metrics reach the router

Prometheus scrapes only the router. Agents send operational information to the router through:

frequent routing-signal updates
periodic heartbeats
request outcomes

The router then exposes this information through its own Prometheus registry. You do not need to configure Prometheus to scrape every agent directly. For supported engines, each agent scrapes its local backend’s /metrics endpoint and forwards the resulting values to the router.

Metric groups

Group	What it covers
Routing	Registered agents, health, routed and failed requests
Per-agent history	Success, failure, tokens, capacity, latency, and disconnections
Engine	Cache, queue, latency, finish reasons, and throughput
Hardware	GPU, CPU, and system memory
Policies	Primary routing, fallback, exhaustion, reloads, and connection resets
Queue	Requests waiting for agent capacity
Tenant and quota	Requests, tokens, limits, last use, and quota failures
Admission control	Gate rejections, replica-scaled occupancy budgets, and in-flight concurrency
HTTP	Router endpoint latency, active requests, and provider calls

Metric labels and cardinality

Several metrics include labels such as:

peer_id
model
tenant_id
key_id
deployment_id
organization
machine
region
engine

These labels make operational breakdowns possible, but each distinct label combination creates another Prometheus time series. Admission-control metrics use only model, or reason and model. They do not expose tenant or key identity.

Do not use unbounded or frequently changing values for agent metadata.Values such as machine, organization, region, and deployment_id should remain stable. Avoid putting request IDs, user IDs, timestamps, or other high-cardinality data into agent metadata.

Routing metrics

Registered agents

These gauges use:

peer_id
region
engine
model
capacity
organization
machine

as labels.

hivenet_router_routing_agent_info

The value is 1 while an agent is registered. Count registered agents:

count(
  hivenet_router_routing_agent_info
)

Count agents by model:

count by (model) (
  hivenet_router_routing_agent_info
)

Agent health

hivenet_router_routing_agent_healthy

Values are:

Value	Meaning
`1`	Agent is healthy
`0`	Health monitor marked the agent unhealthy

Count healthy agents by model:

sum by (model) (
  hivenet_router_routing_agent_healthy
)

Find unhealthy agents:

hivenet_router_routing_agent_healthy == 0

Last heartbeat

hivenet_router_routing_agent_last_seen_timestamp

The value is a Unix timestamp in milliseconds. Show seconds since the most recent heartbeat:

time()
-
(
  hivenet_router_routing_agent_last_seen_timestamp
  / 1000
)

Routed requests

hivenet_router_routing_requests_routed_total

Labels:

region
engine
model
tenant_id

Request rate by model:

sum by (model) (
  rate(
    hivenet_router_routing_requests_routed_total[5m]
  )
)

Request rate by region and engine:

sum by (region, engine) (
  rate(
    hivenet_router_routing_requests_routed_total[5m]
  )
)

Failed routing

hivenet_router_routing_requests_failed_total

This counts inference requests that the router could not forward successfully. Failure rate:

sum(
  rate(
    hivenet_router_routing_requests_failed_total[5m]
  )
)
/
clamp_min(
  sum(
    rate(
      hivenet_router_routing_requests_routed_total[5m]
    )
  )
  +
  sum(
    rate(
      hivenet_router_routing_requests_failed_total[5m]
    )
  ),
  0.000001
)

Per-agent metrics

These metrics use:

peer_id
model
engine
organization
machine

as labels.

Successful and failed requests

hivenet_router_agent_requests_success_total
hivenet_router_agent_requests_failed_total

Request rate by agent:

sum by (peer_id, model) (
  rate(
    hivenet_router_agent_requests_success_total[5m]
  )
)

Success rate

hivenet_router_agent_success_rate

The value ranges from 0.0 to 1.0. Find agents below 95%:

hivenet_router_agent_success_rate < 0.95

A new agent may not expose a meaningful success rate until it has completed requests.

Capacity utilization

hivenet_router_agent_capacity_utilization

The value is:

active requests / declared capacity

For non-streaming requests, the slot normally remains occupied until the response completes. For streaming requests, the current implementation releases the agent capacity slot when response headers arrive and streaming begins, while backend generation can continue.

Do not use hivenet_router_agent_capacity_utilization as the authoritative count of ongoing streaming generations.For streaming-heavy workloads, compare it with engine running and waiting requests, KV-cache pressure, TTFT, and ITL.

Show agents above 80%:

hivenet_router_agent_capacity_utilization > 0.8

Tokens

hivenet_router_agent_input_tokens_total
hivenet_router_agent_output_tokens_total

Token rate by model:

sum by (model) (
  rate(
    hivenet_router_agent_input_tokens_total[5m]
  )
  +
  rate(
    hivenet_router_agent_output_tokens_total[5m]
  )
)

Capacity rejections

hivenet_router_agent_rejected_requests_total

This increases when Hivenet Router tries to acquire a capacity slot and the agent is already full.

sum by (peer_id, model) (
  rate(
    hivenet_router_agent_rejected_requests_total[5m]
  )
)

Disconnections and health failures

hivenet_router_agent_disconnections_total
hivenet_router_agent_failure_total
hivenet_router_model_backend_failure_total

Metric	Meaning
`hivenet_router_agent_disconnections_total`	Agent disconnection history
`hivenet_router_agent_failure_total`	Agent marked unhealthy after missed heartbeats
`hivenet_router_model_backend_failure_total`	Agent reported its inference backend as unhealthy

Compare agent-process and backend failures:

sum by (peer_id, model) (
  rate(
    hivenet_router_agent_failure_total[1h]
  )
)

sum by (peer_id, model) (
  rate(
    hivenet_router_model_backend_failure_total[1h]
  )
)

Smoothed latency

hivenet_router_agent_srtt_ms
hivenet_router_agent_rttvar_ms

These follow the RFC 6298 smoothed round-trip-time calculation. Show SRTT by agent:

hivenet_router_agent_srtt_ms

Find high or unstable latency:

hivenet_router_agent_srtt_ms > 5000

hivenet_router_agent_rttvar_ms > 2000

SRTT reflects the request path observed by Hivenet Router. It includes more than raw network latency.

Persistence and resets

Current agent metadata, health, hardware, and engine snapshots are held in memory. Per-agent lifetime counters and latency history are also stored in BadgerDB and reseeded into Prometheus when an agent registers again. These include:

successful and failed request counts
input and output tokens
capacity rejections
disconnections and failure counters
SRTT and RTTVAR

The router’s default persistent-entry lifetime is 30 days. Reset these values through:

POST /admin/metrics/reset

curl -X POST \
  -H "Authorization: Bearer <admin-api-key>" \
  http://localhost:8080/admin/metrics/reset

This clears persisted per-agent lifetime counters, latency history, matching in-memory state, and their Prometheus series.It does not reset tenant quota counters or every router-level metric.

Engine metrics

Engine metrics use:

peer_id
model
engine
organization
machine

as labels. Availability depends on the backend:

Backend	Engine metrics
vLLM	Full supported set
SGLang	Cache, queue, and TTFT subset
llama.cpp with `--metrics`	Cache, queue, TTFT, ITL, and throughput
Ollama	No engine-specific metrics
Infinity	No engine-specific metrics
Custom	No engine-specific metrics

A metric is absent until the agent reports it.

Cache and queue state

hivenet_router_agent_engine_kv_cache_utilization
hivenet_router_agent_engine_running_requests
hivenet_router_agent_engine_waiting_requests

Agents under cache pressure:

hivenet_router_agent_engine_kv_cache_utilization > 0.9

Agents with backend queues:

hivenet_router_agent_engine_waiting_requests > 0

Preemptions

hivenet_router_agent_engine_preemptions_total

Despite the _total suffix, this series is exported as a gauge representing the backend’s latest cumulative value. Use a change function to detect growth:

delta(
  hivenet_router_agent_engine_preemptions_total[5m]
) > 0

Do not assume ordinary counter-reset behavior across every backend restart.

Per-agent TTFT and ITL gauges

hivenet_router_agent_engine_avg_ttft_seconds
hivenet_router_agent_engine_p90_ttft_seconds
hivenet_router_agent_engine_avg_itl_seconds
hivenet_router_agent_engine_p90_itl_seconds

These are convenient for viewing one agent. For fleet-wide percentiles, use the histogram metrics instead of averaging per-agent percentile gauges.

Finish reasons

hivenet_router_agent_engine_request_success_total

Additional label:

finished_reason

For vLLM, common values include:

stop
length
abort

Completion rate by finish reason:

sum by (finished_reason) (
  rate(
    hivenet_router_agent_engine_request_success_total[5m]
  )
)

llama.cpp throughput

hivenet_router_agent_engine_predicted_tps
hivenet_router_agent_engine_prompt_tps

These expose the latest reported generation and prompt-ingestion throughput from llama.cpp.

Engine histograms

The current router re-exports raw engine histogram buckets for vLLM agents. SGLang and llama.cpp can contribute scalar average and P90 values, but their raw histogram buckets are not currently re-exported through Hivenet Router. The vLLM-backed histogram families are:

hivenet_router_agent_engine_ttft_seconds
hivenet_router_agent_engine_itl_seconds
hivenet_router_agent_engine_request_prompt_tokens
hivenet_router_agent_engine_request_generation_tokens

Prometheus exposes each histogram as:

<name>_bucket
<name>_sum
<name>_count

Fleet P90 TTFT

histogram_quantile(
  0.90,
  sum by (le) (
    rate(
      hivenet_router_agent_engine_ttft_seconds_bucket[5m]
    )
  )
)

P90 TTFT by model

histogram_quantile(
  0.90,
  sum by (le, model) (
    rate(
      hivenet_router_agent_engine_ttft_seconds_bucket[5m]
    )
  )
)

Fleet P90 ITL

histogram_quantile(
  0.90,
  sum by (le) (
    rate(
      hivenet_router_agent_engine_itl_seconds_bucket[5m]
    )
  )
)

P90 prompt length

histogram_quantile(
  0.90,
  sum by (le) (
    rate(
      hivenet_router_agent_engine_request_prompt_tokens_bucket[5m]
    )
  )
)

P90 generation length

histogram_quantile(
  0.90,
  sum by (le) (
    rate(
      hivenet_router_agent_engine_request_generation_tokens_bucket[5m]
    )
  )
)

Use histogram_quantile() over aggregated buckets for fleet percentiles.Averaging several per-agent P90 gauges does not produce a valid fleet P90.

GPU metrics

GPU metrics use:

peer_id
region
model
engine
gpu_index
gpu_id
organization
machine

as labels.

hivenet_router_agent_gpu_utilization_percent
hivenet_router_agent_gpu_vram_used_bytes
hivenet_router_agent_gpu_vram_free_bytes
hivenet_router_agent_gpu_vram_total_bytes
hivenet_router_agent_gpu_temperature_celsius
hivenet_router_agent_gpu_power_watts

Unlike policy gates, the Prometheus utilization value uses a 0 to 100 scale.

GPU utilization

hivenet_router_agent_gpu_utilization_percent

Average utilization by model:

avg by (model) (
  hivenet_router_agent_gpu_utilization_percent
)

VRAM percentage

100
*
hivenet_router_agent_gpu_vram_used_bytes
/
clamp_min(
  hivenet_router_agent_gpu_vram_total_bytes,
  1
)

Temperature

max by (peer_id, model) (
  hivenet_router_agent_gpu_temperature_celsius
)

Power

sum by (peer_id, model) (
  hivenet_router_agent_gpu_power_watts
)

CPU-only agents do not expose GPU series.

CPU and memory metrics

Labels:

peer_id
region
model
engine
organization
machine

hivenet_router_agent_cpu_usage_percent
hivenet_router_agent_memory_used_percent
hivenet_router_agent_memory_available_bytes
hivenet_router_agent_memory_total_bytes

CPU usage:

hivenet_router_agent_cpu_usage_percent

Memory use:

hivenet_router_agent_memory_used_percent

Available GiB:

hivenet_router_agent_memory_available_bytes
/
1024
/
1024
/
1024

Policy metrics

Primary and fallback routing

hivenet_router_policy_primary_routed_total
hivenet_router_policy_fallback_routed_total
hivenet_router_policy_provider_fallback_total
hivenet_router_policy_exhausted_total

All use the model label. Local fallback rate by model:

sum by (model) (
  rate(
    hivenet_router_policy_fallback_routed_total[5m]
  )
)

Provider fallback rate:

sum by (model) (
  rate(
    hivenet_router_policy_provider_fallback_total[5m]
  )
)

Policy exhaustion:

sum by (model) (
  rate(
    hivenet_router_policy_exhausted_total[5m]
  )
)

This counter describes an exhausted policy path, not one guaranteed final HTTP status. A request can proceed to provider fallback, and a failed provider call can end with a provider or backend error rather than a local 503.

Stale connection resets

hivenet_router_agent_connection_resets_total

Labels:

model
reason

A sustained rate can indicate repeated router-agent connection loss:

sum by (model, reason) (
  rate(
    hivenet_router_agent_connection_resets_total[5m]
  )
)

Policy reloads

hivenet_router_policy_reload_total

Labels:

trigger
result

Current values include:

trigger="api"
trigger="sighup"

result="success"
result="error"

Failed reloads:

sum by (trigger) (
  increase(
    hivenet_router_policy_reload_total{
      result="error"
    }[1h]
  )
)

Queue metrics

These metrics describe the per-model queue used when eligible agents exist but all declared capacity slots are occupied. They do not describe the router’s global pending-request channel, requests waiting for --max-concurrent, or work already forwarded to a backend.

Current queue depth

hivenet_router_queue_depth

Label:

model

hivenet_router_queue_depth > 0

Queue wait duration

hivenet_router_queue_wait_seconds

P95 queue wait by model:

histogram_quantile(
  0.95,
  sum by (le, model) (
    rate(
      hivenet_router_queue_wait_seconds_bucket[5m]
    )
  )
)

Tenant and quota metrics

Tenant metrics attribute activity to clients and deployments.

Label defaults

Label	Source	Default
`tenant_id`	Authenticated owner or tenant	`default` in no-auth mode
`key_id`	Dynamic registry entry ID	`anonymous` for static or no-auth access
`deployment_id`	Selected agent registration metadata	`unset` before routing or when absent
`model`	Request body	Requested model

A pre-routing failure cannot have a selected deployment, so its deployment_id is:

unset

Successful and failed requests

hivenet_router_tenant_requests_success_total
hivenet_router_tenant_requests_failed_total

Labels:

tenant_id
key_id
deployment_id
model

Request count by tenant and deployment:

sum by (tenant_id, deployment_id) (
  increase(
    hivenet_router_tenant_requests_success_total[7d]
  )
)

Token use

hivenet_router_tenant_input_tokens_total
hivenet_router_tenant_output_tokens_total

Token rate by tenant and model:

sum by (tenant_id, model) (
  rate(
    hivenet_router_tenant_input_tokens_total[5m]
  )
  +
  rate(
    hivenet_router_tenant_output_tokens_total[5m]
  )
)

Request-rate rejection

hivenet_router_tenant_rate_limited_total

Labels:

tenant_id
key_id

sum by (tenant_id, key_id) (
  rate(
    hivenet_router_tenant_rate_limited_total[5m]
  )
)

Token-budget rejection

hivenet_router_tenant_token_limited_total

Labels:

tenant_id
key_id
deployment_id
phase

phase is:

input
output

sum by (tenant_id, phase) (
  rate(
    hivenet_router_tenant_token_limited_total[5m]
  )
)

Flat quota limits

hivenet_router_tenant_quota_rpm_limit
hivenet_router_tenant_quota_tpd_limit

Label:

tenant_id

A value of 0 means unlimited. These gauges describe keys using the flat quota shape.

Per-model quota limits

hivenet_router_tenant_per_model_quota_rpm_limit
hivenet_router_tenant_per_model_quota_tpd_limit

Labels:

tenant_id
model

The RPM gauge reflects the current effective ceiling:

requests per minute per replica
× healthy replicas

It can change as agent health and fleet size change.

Tokens used today

hivenet_router_tenant_tokens_used_today

Label:

tenant_id

The value resets at midnight UTC. Token-budget utilization for flat quotas:

hivenet_router_tenant_tokens_used_today
/
clamp_min(
  hivenet_router_tenant_quota_tpd_limit,
  1
)

Last request timestamp

hivenet_router_tenant_last_request_timestamp

Labels:

tenant_id
key_id
deployment_id

Most recent request per API key:

max by (tenant_id, key_id) (
  hivenet_router_tenant_last_request_timestamp
)

Seconds since last request:

time()
-
max by (tenant_id, key_id) (
  hivenet_router_tenant_last_request_timestamp
)

Tenant request duration

hivenet_router_tenant_request_duration_seconds

Labels:

tenant_id
key_id
deployment_id
model

P95 by tenant and model:

histogram_quantile(
  0.95,
  sum by (le, tenant_id, model) (
    rate(
      hivenet_router_tenant_request_duration_seconds_bucket[5m]
    )
  )
)

Quota persistence errors

hivenet_router_quota_backend_errors_total

This increases when the Badger-backed daily token limiter cannot flush quota state to disk.

increase(
  hivenet_router_quota_backend_errors_total[15m]
) > 0

A persistence error may leave the in-memory quota path working while daily usage is not safely stored for restart recovery.Investigate any increase rather than treating it as a harmless background error.

Admission-control metrics

The LLM admission gates publish one rejection counter and four live pool gauges:

Metric	Type	Labels	Meaning
`hivenet_router_admission_rejections_total`	Counter	`reason`, `model`	Requests rejected by B1 through B4
`hivenet_router_admission_occupancy_tokens`	Gauge	`model`	Current token-weighted in-flight sum
`hivenet_router_admission_budget_tokens`	Gauge	`model`	Effective B2 budget after admit fraction and healthy-replica scaling
`hivenet_router_admission_inflight_requests`	Gauge	`model`	Current in-flight request count
`hivenet_router_admission_max_inflight`	Gauge	`model`	Effective `max_inflight × healthy_replicas` backstop

The reason label identifies the gate:

Reason	Trigger
`b1`	Input-token or image-count cap
`b2`	Global occupancy budget or in-flight backstop
`b3`	Live pool-pressure shed
`b4_occupancy`	Serverless per-key occupancy share
`b4_itpm`	Serverless per-key input tokens per minute
`b4_otpm`	Serverless per-key output tokens per minute
`b4_rpm`	Request-per-minute limit

The budget and backstop gauges can change as healthy replicas join or leave. Existing reservations remain visible in the occupancy gauges while the new denominators affect subsequent admissions. POST /v1/messages/count_tokens, embedding, and reranking requests do not enter B1 through B4, so they do not change these occupancy gauges. See Admission control metrics for panel queries and alert interpretation.

Per-request duration by agent

hivenet_router_request_duration_seconds

Labels:

tenant_id
peer_id
model
status_code

P95 duration by agent:

histogram_quantile(
  0.95,
  sum by (le, peer_id, model) (
    rate(
      hivenet_router_request_duration_seconds_bucket[5m]
    )
  )
)

HTTP server metrics

Request duration

http_server_request_duration_seconds

Labels:

method
route
status_code

P95 by route:

histogram_quantile(
  0.95,
  sum by (le, route) (
    rate(
      http_server_request_duration_seconds_bucket[5m]
    )
  )
)

Error rate by route:

sum by (route) (
  rate(
    http_server_request_duration_seconds_count{
      status_code=~"5.."
    }[5m]
  )
)

Active requests

http_server_active_requests

Labels:

method
route

http_server_active_requests{
  method="POST",
  route="/v1/chat/completions"
}

Provider HTTP metrics

http_client_request_duration_seconds

Labels:

provider
status_code

P95 provider latency:

histogram_quantile(
  0.95,
  sum by (le, provider) (
    rate(
      http_client_request_duration_seconds_bucket[5m]
    )
  )
)

Provider responses by status:

sum by (provider, status_code) (
  rate(
    http_client_request_duration_seconds_count[5m]
  )
)

Network failures use:

status_code="0"

Alerting examples

The thresholds below are examples. Adjust them to your hardware, workload, and service objectives.

Agent unhealthy

groups:
  - name: hivenet-router

    rules:
      - alert: HivenetRouterAgentUnhealthy

        expr: |
          hivenet_router_routing_agent_healthy == 0

        for: 1m

        labels:
          severity: warning

        annotations:
          summary: >-
            Hivenet Router agent {{ $labels.peer_id }} is unhealthy

High routing failure rate

      - alert: HivenetRouterHighRoutingFailureRate

        expr: |
          (
            sum(
              rate(
                hivenet_router_routing_requests_failed_total[5m]
              )
            )
            /
            clamp_min(
              sum(
                rate(
                  hivenet_router_routing_requests_routed_total[5m]
                )
              )
              +
              sum(
                rate(
                  hivenet_router_routing_requests_failed_total[5m]
                )
              ),
              0.000001
            )
          ) > 0.10

        for: 5m

        labels:
          severity: warning

        annotations:
          summary: >-
            More than 10% of Hivenet Router routing attempts are failing

KV-cache pressure

      - alert: HivenetRouterKVCachePressure

        expr: |
          hivenet_router_agent_engine_kv_cache_utilization > 0.90

        for: 2m

        labels:
          severity: warning

        annotations:
          summary: >-
            High KV-cache use on {{ $labels.peer_id }}

GPU temperature

      - alert: HivenetRouterGPUHighTemperature

        expr: |
          hivenet_router_agent_gpu_temperature_celsius > 85

        for: 5m

        labels:
          severity: critical

        annotations:
          summary: >-
            High GPU temperature on {{ $labels.peer_id }}

Policy exhaustion

      - alert: HivenetRouterPolicyExhaustion

        expr: |
          sum by (model) (
            rate(
              hivenet_router_policy_exhausted_total[5m]
            )
          ) > 0

        for: 5m

        labels:
          severity: warning

        annotations:
          summary: >-
            Routing policy is being exhausted for {{ $labels.model }}

Quota persistence failure

      - alert: HivenetRouterQuotaPersistenceFailure

        expr: |
          increase(
            hivenet_router_quota_backend_errors_total[15m]
          ) > 0

        labels:
          severity: critical

        annotations:
          summary: >-
            Hivenet Router could not persist daily quota state

Recording rules

Recording rules can simplify expensive or frequently reused dashboard queries.

groups:
  - name: hivenet-router-recording

    interval: 30s

    rules:
      - record: hivenet_router:model_request_rate5m

        expr: |
          sum by (model) (
            rate(
              hivenet_router_routing_requests_routed_total[5m]
            )
          )

      - record: hivenet_router:model_failure_rate5m

        expr: |
          sum by (model) (
            rate(
              hivenet_router_routing_requests_failed_total[5m]
            )
          )

      - record: hivenet_router:model_p95_ttft_seconds5m

        expr: |
          histogram_quantile(
            0.95,
            sum by (le, model) (
              rate(
                hivenet_router_agent_engine_ttft_seconds_bucket[5m]
              )
            )
          )

Troubleshooting

Prometheus cannot reach the router

Check the endpoint directly:

curl -i \
  http://<router-host>:2112/metrics

Check the router logs:

journalctl \
  -u hivenet-router \
  | grep -i prometheus

For Docker Compose:

docker compose exec prometheus \
  wget -qO- \
  http://router:2112/metrics \
  | head

Check firewall and container-network rules.

A metric is missing

A series often appears only after the relevant event or data has occurred. For example:

tenant metrics appear after requests
engine metrics appear after a successful engine scrape
TTFT and ITL appear after completions
GPU metrics require NVML and visible NVIDIA devices
policy counters appear after routing activity
per-model quota gauges appear when those quota paths are used or seeded

Search by prefix:

curl -s \
  http://localhost:2112/metrics \
  | grep 'hivenet_router_agent_engine'

An engine metric is missing

Check the backend endpoint locally on the agent host:

curl \
  http://localhost:8888/metrics \
  | head

Confirm the engine-specific requirement:

SGLang needs --enable-metrics
llama.cpp needs --metrics
Ollama, Infinity, and custom engines do not currently supply engine metrics

Inspect the agent logs for scrape errors. A failed metrics scrape does not stop request forwarding.

Prometheus shows duplicate or stale agents

The peer_id label identifies the agent. If an agent starts with a new identity after every restart, Prometheus sees a new series. Configure a persistent:

--identity-path

and preserve the file or mounted volume across restarts.

Tenant labels show `anonymous` or `default`

In no-auth mode:

tenant_id="default"
key_id="anonymous"

Static API keys currently use:

key_id="anonymous"

because stable key IDs belong to dynamic registry entries. deployment_id="unset" is expected before an agent is selected or when the agent does not advertise a deployment ID.

Histogram queries return no data

Check that the corresponding _bucket series exists:

curl -s \
  http://localhost:2112/metrics \
  | grep \
    'hivenet_router_agent_engine_ttft_seconds_bucket'

Use rate() over a window that contains completed observations. A new or idle engine may not have enough data yet.

Counter graphs drop after a restart

Prometheus counters are process-local series. Some per-agent lifetime counters are reseeded from BadgerDB when agents register again, but other router, policy, HTTP, tenant, and queue metrics begin again with the new router process. Use rate() or increase() rather than graphing raw counter values when restart behavior matters.

Next steps

Grafana dashboards

Explore the provisioned dashboards, data sources, variables, and panels.

Audit logging

Query structured request records through Loki or local JSONL files.

Hardware metrics

Review how agents collect GPU, CPU, and memory data.

⌘I

​Start the metrics server

​Configure Prometheus

​Check the scrape target

​How metrics reach the router

​Metric groups

​Metric labels and cardinality

​Routing metrics

​Registered agents

​Agent health

​Last heartbeat

​Routed requests

​Failed routing

​Per-agent metrics

​Successful and failed requests

​Success rate

​Capacity utilization

​Tokens

​Capacity rejections

​Disconnections and health failures

​Smoothed latency

​Persistence and resets

​Engine metrics

​Cache and queue state

​Preemptions

​Per-agent TTFT and ITL gauges

​Finish reasons

​llama.cpp throughput

​Engine histograms

​Fleet P90 TTFT

​P90 TTFT by model

​Fleet P90 ITL

​P90 prompt length

​P90 generation length

​GPU metrics

​GPU utilization

​VRAM percentage

​Temperature

​Power

​CPU and memory metrics

​Policy metrics

​Primary and fallback routing

​Stale connection resets

​Policy reloads

​Queue metrics

​Current queue depth

​Queue wait duration

​Tenant and quota metrics

​Label defaults

​Successful and failed requests

​Token use

​Request-rate rejection

​Token-budget rejection

​Flat quota limits

​Per-model quota limits

​Tokens used today

​Last request timestamp

​Tenant request duration

​Quota persistence errors

​Admission-control metrics

​Per-request duration by agent

​HTTP server metrics

​Request duration

​Active requests

​Provider HTTP metrics

​Alerting examples

​Agent unhealthy

​High routing failure rate

​KV-cache pressure

​GPU temperature

​Policy exhaustion

​Quota persistence failure

​Recording rules

​Troubleshooting

​Prometheus cannot reach the router

​A metric is missing

​An engine metric is missing

​Prometheus shows duplicate or stale agents

​Tenant labels show anonymous or default

​Histogram queries return no data

​Counter graphs drop after a restart

Start the metrics server

Configure Prometheus

Check the scrape target

How metrics reach the router

Metric groups

Metric labels and cardinality

Routing metrics

Registered agents

Agent health

Last heartbeat

Routed requests

Failed routing

Per-agent metrics

Successful and failed requests

Success rate

Capacity utilization

Tokens

Capacity rejections

Disconnections and health failures

Smoothed latency

Persistence and resets

Engine metrics

Cache and queue state

Preemptions

Per-agent TTFT and ITL gauges

Finish reasons

llama.cpp throughput

Engine histograms

Fleet P90 TTFT

P90 TTFT by model

Fleet P90 ITL

P90 prompt length

P90 generation length

GPU metrics

GPU utilization

VRAM percentage

Temperature

Power

CPU and memory metrics

Policy metrics

Primary and fallback routing

Stale connection resets

Policy reloads

Queue metrics

Current queue depth

Queue wait duration

Tenant and quota metrics

Label defaults

Successful and failed requests

Token use

Request-rate rejection

Token-budget rejection

Flat quota limits

Per-model quota limits

Tokens used today

Last request timestamp

Tenant request duration

Quota persistence errors

Admission-control metrics

Per-request duration by agent

HTTP server metrics

Request duration

Active requests

Provider HTTP metrics

Alerting examples

Agent unhealthy

High routing failure rate

KV-cache pressure

GPU temperature

Policy exhaustion

Quota persistence failure

Recording rules

Troubleshooting

Prometheus cannot reach the router

A metric is missing

An engine metric is missing

Prometheus shows duplicate or stale agents

Tenant labels show `anonymous` or `default`

Histogram queries return no data

Counter graphs drop after a restart