> ## Documentation Index
> Fetch the complete documentation index at: https://routerdocs.hivenet.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Prometheus metrics

> Scrape Hivenet Router metrics for routing, agents, engines, hardware, policies, queues, tenants, quotas, and HTTP traffic.

Hivenet Router exports router, agent, inference-engine, hardware, policy, quota, and HTTP metrics in Prometheus format.

```text theme={null}
GET /metrics
```

The metrics server runs separately from the main HTTP API and listens on port `2112` by default.

```bash theme={null}
curl http://localhost:2112/metrics
```

<Warning>
  The metrics endpoint does not have built-in authentication.

  Restrict it to Prometheus and trusted operators through a private network, firewall, reverse proxy, or equivalent network control.
</Warning>

## Start the metrics server

The router starts the Prometheus endpoint automatically.

The default address is:

```text theme={null}
:2112
```

Change it with:

```bash theme={null}
./bin/hivenet-router \
  --metrics-port :9212 \
  ...
```

The metrics endpoint is then available at:

```text theme={null}
http://<router-host>:9212/metrics
```

## Configure Prometheus

A minimal scrape configuration is:

```yaml theme={null}
global:
  scrape_interval: 5s
  evaluation_interval: 5s

scrape_configs:
  - job_name: hivenet-router

    static_configs:
      - targets:
          - router.example.internal:2112
```

Reload Prometheus after changing its configuration.

With the repository’s Docker Compose stack, Prometheus reaches the router through the internal Compose network:

```yaml theme={null}
scrape_configs:
  - job_name: hivenet-router

    static_configs:
      - targets:
          - router:2112
```

Port `2112` does not need to be published to the host when Prometheus runs in the same Compose network.

## Check the scrape target

From the Prometheus interface, open:

```text theme={null}
Status → Targets
```

Or query the Prometheus API:

```bash theme={null}
curl \
  http://localhost:9090/api/v1/targets \
  | jq '.data.activeTargets[] | {
      scrape_url: .scrapeUrl,
      health,
      last_error: .lastError
    }'
```

A healthy target reports:

```json theme={null}
{
  "health": "up",
  "last_error": ""
}
```

Check that Hivenet Router metrics exist:

```bash theme={null}
curl -s \
  http://localhost:2112/metrics \
  | grep '^hivenet_router_'
```

## How metrics reach the router

Prometheus scrapes only the router.

Agents send operational information to the router through:

* frequent routing-signal updates
* periodic heartbeats
* request outcomes

The router then exposes this information through its own Prometheus registry.

You do not need to configure Prometheus to scrape every agent directly.

For supported engines, each agent scrapes its local backend’s `/metrics` endpoint and forwards the resulting values to the router.

## Metric groups

| Group             | What it covers                                                               |
| ----------------- | ---------------------------------------------------------------------------- |
| Routing           | Registered agents, health, routed and failed requests                        |
| Per-agent history | Success, failure, tokens, capacity, latency, and disconnections              |
| Engine            | Cache, queue, latency, finish reasons, and throughput                        |
| Hardware          | GPU, CPU, and system memory                                                  |
| Policies          | Primary routing, fallback, exhaustion, reloads, and connection resets        |
| Queue             | Requests waiting for agent capacity                                          |
| Tenant and quota  | Requests, tokens, limits, last use, and quota failures                       |
| Admission control | Gate rejections, replica-scaled occupancy budgets, and in-flight concurrency |
| HTTP              | Router endpoint latency, active requests, and provider calls                 |

## Metric labels and cardinality

Several metrics include labels such as:

* `peer_id`
* `model`
* `tenant_id`
* `key_id`
* `deployment_id`
* `organization`
* `machine`
* `region`
* `engine`

These labels make operational breakdowns possible, but each distinct label combination creates another Prometheus time series.

Admission-control metrics use only `model`, or `reason` and `model`. They do not expose tenant or key identity.

<Warning>
  Do not use unbounded or frequently changing values for agent metadata.

  Values such as `machine`, `organization`, `region`, and `deployment_id` should remain stable. Avoid putting request IDs, user IDs, timestamps, or other high-cardinality data into agent metadata.
</Warning>

## Routing metrics

### Registered agents

These gauges use:

```text theme={null}
peer_id
region
engine
model
capacity
organization
machine
```

as labels.

```promql theme={null}
hivenet_router_routing_agent_info
```

The value is `1` while an agent is registered.

Count registered agents:

```promql theme={null}
count(
  hivenet_router_routing_agent_info
)
```

Count agents by model:

```promql theme={null}
count by (model) (
  hivenet_router_routing_agent_info
)
```

### Agent health

```promql theme={null}
hivenet_router_routing_agent_healthy
```

Values are:

| Value | Meaning                                   |
| ----: | ----------------------------------------- |
|   `1` | Agent is healthy                          |
|   `0` | Health monitor marked the agent unhealthy |

Count healthy agents by model:

```promql theme={null}
sum by (model) (
  hivenet_router_routing_agent_healthy
)
```

Find unhealthy agents:

```promql theme={null}
hivenet_router_routing_agent_healthy == 0
```

### Last heartbeat

```promql theme={null}
hivenet_router_routing_agent_last_seen_timestamp
```

The value is a Unix timestamp in **milliseconds**.

Show seconds since the most recent heartbeat:

```promql theme={null}
time()
-
(
  hivenet_router_routing_agent_last_seen_timestamp
  / 1000
)
```

### Routed requests

```promql theme={null}
hivenet_router_routing_requests_routed_total
```

Labels:

```text theme={null}
region
engine
model
tenant_id
```

Request rate by model:

```promql theme={null}
sum by (model) (
  rate(
    hivenet_router_routing_requests_routed_total[5m]
  )
)
```

Request rate by region and engine:

```promql theme={null}
sum by (region, engine) (
  rate(
    hivenet_router_routing_requests_routed_total[5m]
  )
)
```

### Failed routing

```promql theme={null}
hivenet_router_routing_requests_failed_total
```

This counts inference requests that the router could not forward successfully.

Failure rate:

```promql theme={null}
sum(
  rate(
    hivenet_router_routing_requests_failed_total[5m]
  )
)
/
clamp_min(
  sum(
    rate(
      hivenet_router_routing_requests_routed_total[5m]
    )
  )
  +
  sum(
    rate(
      hivenet_router_routing_requests_failed_total[5m]
    )
  ),
  0.000001
)
```

## Per-agent metrics

These metrics use:

```text theme={null}
peer_id
model
engine
organization
machine
```

as labels.

### Successful and failed requests

```promql theme={null}
hivenet_router_agent_requests_success_total
hivenet_router_agent_requests_failed_total
```

Request rate by agent:

```promql theme={null}
sum by (peer_id, model) (
  rate(
    hivenet_router_agent_requests_success_total[5m]
  )
)
```

### Success rate

```promql theme={null}
hivenet_router_agent_success_rate
```

The value ranges from `0.0` to `1.0`.

Find agents below 95%:

```promql theme={null}
hivenet_router_agent_success_rate < 0.95
```

A new agent may not expose a meaningful success rate until it has completed requests.

### Capacity utilization

```promql theme={null}
hivenet_router_agent_capacity_utilization
```

The value is:

```text theme={null}
active requests / declared capacity
```

For non-streaming requests, the slot normally remains occupied until the response completes. For streaming requests, the current implementation releases the agent capacity slot when response headers arrive and streaming begins, while backend generation can continue.

<Warning>
  Do not use `hivenet_router_agent_capacity_utilization` as the authoritative count of ongoing streaming generations.

  For streaming-heavy workloads, compare it with engine running and waiting requests, KV-cache pressure, TTFT, and ITL.
</Warning>

Show agents above 80%:

```promql theme={null}
hivenet_router_agent_capacity_utilization > 0.8
```

### Tokens

```promql theme={null}
hivenet_router_agent_input_tokens_total
hivenet_router_agent_output_tokens_total
```

Token rate by model:

```promql theme={null}
sum by (model) (
  rate(
    hivenet_router_agent_input_tokens_total[5m]
  )
  +
  rate(
    hivenet_router_agent_output_tokens_total[5m]
  )
)
```

### Capacity rejections

```promql theme={null}
hivenet_router_agent_rejected_requests_total
```

This increases when Hivenet Router tries to acquire a capacity slot and the agent is already full.

```promql theme={null}
sum by (peer_id, model) (
  rate(
    hivenet_router_agent_rejected_requests_total[5m]
  )
)
```

### Disconnections and health failures

```promql theme={null}
hivenet_router_agent_disconnections_total
hivenet_router_agent_failure_total
hivenet_router_model_backend_failure_total
```

| Metric                                       | Meaning                                           |
| -------------------------------------------- | ------------------------------------------------- |
| `hivenet_router_agent_disconnections_total`  | Agent disconnection history                       |
| `hivenet_router_agent_failure_total`         | Agent marked unhealthy after missed heartbeats    |
| `hivenet_router_model_backend_failure_total` | Agent reported its inference backend as unhealthy |

Compare agent-process and backend failures:

```promql theme={null}
sum by (peer_id, model) (
  rate(
    hivenet_router_agent_failure_total[1h]
  )
)
```

```promql theme={null}
sum by (peer_id, model) (
  rate(
    hivenet_router_model_backend_failure_total[1h]
  )
)
```

### Smoothed latency

```promql theme={null}
hivenet_router_agent_srtt_ms
hivenet_router_agent_rttvar_ms
```

These follow the RFC 6298 smoothed round-trip-time calculation.

Show SRTT by agent:

```promql theme={null}
hivenet_router_agent_srtt_ms
```

Find high or unstable latency:

```promql theme={null}
hivenet_router_agent_srtt_ms > 5000
```

```promql theme={null}
hivenet_router_agent_rttvar_ms > 2000
```

<Note>
  SRTT reflects the request path observed by Hivenet Router. It includes more than raw network latency.
</Note>

## Persistence and resets

Current agent metadata, health, hardware, and engine snapshots are held in memory.

Per-agent lifetime counters and latency history are also stored in BadgerDB and reseeded into Prometheus when an agent registers again. These include:

* successful and failed request counts
* input and output tokens
* capacity rejections
* disconnections and failure counters
* SRTT and RTTVAR

The router’s default persistent-entry lifetime is 30 days.

Reset these values through:

```text theme={null}
POST /admin/metrics/reset
```

```bash theme={null}
curl -X POST \
  -H "Authorization: Bearer <admin-api-key>" \
  http://localhost:8080/admin/metrics/reset
```

<Danger>
  This clears persisted per-agent lifetime counters, latency history, matching in-memory state, and their Prometheus series.

  It does not reset tenant quota counters or every router-level metric.
</Danger>

## Engine metrics

Engine metrics use:

```text theme={null}
peer_id
model
engine
organization
machine
```

as labels.

Availability depends on the backend:

| Backend                    | Engine metrics                          |
| -------------------------- | --------------------------------------- |
| vLLM                       | Full supported set                      |
| SGLang                     | Cache, queue, and TTFT subset           |
| llama.cpp with `--metrics` | Cache, queue, TTFT, ITL, and throughput |
| Ollama                     | No engine-specific metrics              |
| Infinity                   | No engine-specific metrics              |
| Custom                     | No engine-specific metrics              |

A metric is absent until the agent reports it.

### Cache and queue state

```promql theme={null}
hivenet_router_agent_engine_kv_cache_utilization
hivenet_router_agent_engine_running_requests
hivenet_router_agent_engine_waiting_requests
```

Agents under cache pressure:

```promql theme={null}
hivenet_router_agent_engine_kv_cache_utilization > 0.9
```

Agents with backend queues:

```promql theme={null}
hivenet_router_agent_engine_waiting_requests > 0
```

### Preemptions

```promql theme={null}
hivenet_router_agent_engine_preemptions_total
```

Despite the `_total` suffix, this series is exported as a gauge representing the backend’s latest cumulative value.

Use a change function to detect growth:

```promql theme={null}
delta(
  hivenet_router_agent_engine_preemptions_total[5m]
) > 0
```

Do not assume ordinary counter-reset behavior across every backend restart.

### Per-agent TTFT and ITL gauges

```promql theme={null}
hivenet_router_agent_engine_avg_ttft_seconds
hivenet_router_agent_engine_p90_ttft_seconds
hivenet_router_agent_engine_avg_itl_seconds
hivenet_router_agent_engine_p90_itl_seconds
```

These are convenient for viewing one agent.

For fleet-wide percentiles, use the histogram metrics instead of averaging per-agent percentile gauges.

### Finish reasons

```promql theme={null}
hivenet_router_agent_engine_request_success_total
```

Additional label:

```text theme={null}
finished_reason
```

For vLLM, common values include:

```text theme={null}
stop
length
abort
```

Completion rate by finish reason:

```promql theme={null}
sum by (finished_reason) (
  rate(
    hivenet_router_agent_engine_request_success_total[5m]
  )
)
```

### llama.cpp throughput

```promql theme={null}
hivenet_router_agent_engine_predicted_tps
hivenet_router_agent_engine_prompt_tps
```

These expose the latest reported generation and prompt-ingestion throughput from llama.cpp.

## Engine histograms

The current router re-exports raw engine histogram buckets for vLLM agents. SGLang and llama.cpp can contribute scalar average and P90 values, but their raw histogram buckets are not currently re-exported through Hivenet Router.

The vLLM-backed histogram families are:

```text theme={null}
hivenet_router_agent_engine_ttft_seconds
hivenet_router_agent_engine_itl_seconds
hivenet_router_agent_engine_request_prompt_tokens
hivenet_router_agent_engine_request_generation_tokens
```

Prometheus exposes each histogram as:

```text theme={null}
<name>_bucket
<name>_sum
<name>_count
```

### Fleet P90 TTFT

```promql theme={null}
histogram_quantile(
  0.90,
  sum by (le) (
    rate(
      hivenet_router_agent_engine_ttft_seconds_bucket[5m]
    )
  )
)
```

### P90 TTFT by model

```promql theme={null}
histogram_quantile(
  0.90,
  sum by (le, model) (
    rate(
      hivenet_router_agent_engine_ttft_seconds_bucket[5m]
    )
  )
)
```

### Fleet P90 ITL

```promql theme={null}
histogram_quantile(
  0.90,
  sum by (le) (
    rate(
      hivenet_router_agent_engine_itl_seconds_bucket[5m]
    )
  )
)
```

### P90 prompt length

```promql theme={null}
histogram_quantile(
  0.90,
  sum by (le) (
    rate(
      hivenet_router_agent_engine_request_prompt_tokens_bucket[5m]
    )
  )
)
```

### P90 generation length

```promql theme={null}
histogram_quantile(
  0.90,
  sum by (le) (
    rate(
      hivenet_router_agent_engine_request_generation_tokens_bucket[5m]
    )
  )
)
```

<Note>
  Use `histogram_quantile()` over aggregated buckets for fleet percentiles.

  Averaging several per-agent P90 gauges does not produce a valid fleet P90.
</Note>

## GPU metrics

GPU metrics use:

```text theme={null}
peer_id
region
model
engine
gpu_index
gpu_id
organization
machine
```

as labels.

```promql theme={null}
hivenet_router_agent_gpu_utilization_percent
hivenet_router_agent_gpu_vram_used_bytes
hivenet_router_agent_gpu_vram_free_bytes
hivenet_router_agent_gpu_vram_total_bytes
hivenet_router_agent_gpu_temperature_celsius
hivenet_router_agent_gpu_power_watts
```

Unlike policy gates, the Prometheus utilization value uses a `0` to `100` scale.

### GPU utilization

```promql theme={null}
hivenet_router_agent_gpu_utilization_percent
```

Average utilization by model:

```promql theme={null}
avg by (model) (
  hivenet_router_agent_gpu_utilization_percent
)
```

### VRAM percentage

```promql theme={null}
100
*
hivenet_router_agent_gpu_vram_used_bytes
/
clamp_min(
  hivenet_router_agent_gpu_vram_total_bytes,
  1
)
```

### Temperature

```promql theme={null}
max by (peer_id, model) (
  hivenet_router_agent_gpu_temperature_celsius
)
```

### Power

```promql theme={null}
sum by (peer_id, model) (
  hivenet_router_agent_gpu_power_watts
)
```

CPU-only agents do not expose GPU series.

## CPU and memory metrics

Labels:

```text theme={null}
peer_id
region
model
engine
organization
machine
```

```promql theme={null}
hivenet_router_agent_cpu_usage_percent
hivenet_router_agent_memory_used_percent
hivenet_router_agent_memory_available_bytes
hivenet_router_agent_memory_total_bytes
```

CPU usage:

```promql theme={null}
hivenet_router_agent_cpu_usage_percent
```

Memory use:

```promql theme={null}
hivenet_router_agent_memory_used_percent
```

Available GiB:

```promql theme={null}
hivenet_router_agent_memory_available_bytes
/
1024
/
1024
/
1024
```

## Policy metrics

### Primary and fallback routing

```promql theme={null}
hivenet_router_policy_primary_routed_total
hivenet_router_policy_fallback_routed_total
hivenet_router_policy_provider_fallback_total
hivenet_router_policy_exhausted_total
```

All use the `model` label.

Local fallback rate by model:

```promql theme={null}
sum by (model) (
  rate(
    hivenet_router_policy_fallback_routed_total[5m]
  )
)
```

Provider fallback rate:

```promql theme={null}
sum by (model) (
  rate(
    hivenet_router_policy_provider_fallback_total[5m]
  )
)
```

Policy exhaustion:

```promql theme={null}
sum by (model) (
  rate(
    hivenet_router_policy_exhausted_total[5m]
  )
)
```

This counter describes an exhausted policy path, not one guaranteed final HTTP status. A request can proceed to provider fallback, and a failed provider call can end with a provider or backend error rather than a local `503`.

### Stale connection resets

```promql theme={null}
hivenet_router_agent_connection_resets_total
```

Labels:

```text theme={null}
model
reason
```

A sustained rate can indicate repeated router-agent connection loss:

```promql theme={null}
sum by (model, reason) (
  rate(
    hivenet_router_agent_connection_resets_total[5m]
  )
)
```

### Policy reloads

```promql theme={null}
hivenet_router_policy_reload_total
```

Labels:

```text theme={null}
trigger
result
```

Current values include:

```text theme={null}
trigger="api"
trigger="sighup"

result="success"
result="error"
```

Failed reloads:

```promql theme={null}
sum by (trigger) (
  increase(
    hivenet_router_policy_reload_total{
      result="error"
    }[1h]
  )
)
```

## Queue metrics

These metrics describe the per-model queue used when eligible agents exist but all declared capacity slots are occupied. They do not describe the router’s global pending-request channel, requests waiting for `--max-concurrent`, or work already forwarded to a backend.

### Current queue depth

```promql theme={null}
hivenet_router_queue_depth
```

Label:

```text theme={null}
model
```

```promql theme={null}
hivenet_router_queue_depth > 0
```

### Queue wait duration

```promql theme={null}
hivenet_router_queue_wait_seconds
```

P95 queue wait by model:

```promql theme={null}
histogram_quantile(
  0.95,
  sum by (le, model) (
    rate(
      hivenet_router_queue_wait_seconds_bucket[5m]
    )
  )
)
```

## Tenant and quota metrics

Tenant metrics attribute activity to clients and deployments.

### Label defaults

| Label           | Source                               | Default                                  |
| --------------- | ------------------------------------ | ---------------------------------------- |
| `tenant_id`     | Authenticated owner or tenant        | `default` in no-auth mode                |
| `key_id`        | Dynamic registry entry ID            | `anonymous` for static or no-auth access |
| `deployment_id` | Selected agent registration metadata | `unset` before routing or when absent    |
| `model`         | Request body                         | Requested model                          |

A pre-routing failure cannot have a selected deployment, so its `deployment_id` is:

```text theme={null}
unset
```

### Successful and failed requests

```promql theme={null}
hivenet_router_tenant_requests_success_total
hivenet_router_tenant_requests_failed_total
```

Labels:

```text theme={null}
tenant_id
key_id
deployment_id
model
```

Request count by tenant and deployment:

```promql theme={null}
sum by (tenant_id, deployment_id) (
  increase(
    hivenet_router_tenant_requests_success_total[7d]
  )
)
```

### Token use

```promql theme={null}
hivenet_router_tenant_input_tokens_total
hivenet_router_tenant_output_tokens_total
```

Token rate by tenant and model:

```promql theme={null}
sum by (tenant_id, model) (
  rate(
    hivenet_router_tenant_input_tokens_total[5m]
  )
  +
  rate(
    hivenet_router_tenant_output_tokens_total[5m]
  )
)
```

### Request-rate rejection

```promql theme={null}
hivenet_router_tenant_rate_limited_total
```

Labels:

```text theme={null}
tenant_id
key_id
```

```promql theme={null}
sum by (tenant_id, key_id) (
  rate(
    hivenet_router_tenant_rate_limited_total[5m]
  )
)
```

### Token-budget rejection

```promql theme={null}
hivenet_router_tenant_token_limited_total
```

Labels:

```text theme={null}
tenant_id
key_id
deployment_id
phase
```

`phase` is:

```text theme={null}
input
output
```

```promql theme={null}
sum by (tenant_id, phase) (
  rate(
    hivenet_router_tenant_token_limited_total[5m]
  )
)
```

### Flat quota limits

```promql theme={null}
hivenet_router_tenant_quota_rpm_limit
hivenet_router_tenant_quota_tpd_limit
```

Label:

```text theme={null}
tenant_id
```

A value of `0` means unlimited.

These gauges describe keys using the flat quota shape.

### Per-model quota limits

```promql theme={null}
hivenet_router_tenant_per_model_quota_rpm_limit
hivenet_router_tenant_per_model_quota_tpd_limit
```

Labels:

```text theme={null}
tenant_id
model
```

The RPM gauge reflects the current effective ceiling:

```text theme={null}
requests per minute per replica
× healthy replicas
```

It can change as agent health and fleet size change.

### Tokens used today

```promql theme={null}
hivenet_router_tenant_tokens_used_today
```

Label:

```text theme={null}
tenant_id
```

The value resets at midnight UTC.

Token-budget utilization for flat quotas:

```promql theme={null}
hivenet_router_tenant_tokens_used_today
/
clamp_min(
  hivenet_router_tenant_quota_tpd_limit,
  1
)
```

### Last request timestamp

```promql theme={null}
hivenet_router_tenant_last_request_timestamp
```

Labels:

```text theme={null}
tenant_id
key_id
deployment_id
```

Most recent request per API key:

```promql theme={null}
max by (tenant_id, key_id) (
  hivenet_router_tenant_last_request_timestamp
)
```

Seconds since last request:

```promql theme={null}
time()
-
max by (tenant_id, key_id) (
  hivenet_router_tenant_last_request_timestamp
)
```

### Tenant request duration

```promql theme={null}
hivenet_router_tenant_request_duration_seconds
```

Labels:

```text theme={null}
tenant_id
key_id
deployment_id
model
```

P95 by tenant and model:

```promql theme={null}
histogram_quantile(
  0.95,
  sum by (le, tenant_id, model) (
    rate(
      hivenet_router_tenant_request_duration_seconds_bucket[5m]
    )
  )
)
```

### Quota persistence errors

```promql theme={null}
hivenet_router_quota_backend_errors_total
```

This increases when the Badger-backed daily token limiter cannot flush quota state to disk.

```promql theme={null}
increase(
  hivenet_router_quota_backend_errors_total[15m]
) > 0
```

<Warning>
  A persistence error may leave the in-memory quota path working while daily usage is not safely stored for restart recovery.

  Investigate any increase rather than treating it as a harmless background error.
</Warning>

## Admission-control metrics

The LLM admission gates publish one rejection counter and four live pool gauges:

| Metric                                       | Type    | Labels            | Meaning                                                              |
| -------------------------------------------- | ------- | ----------------- | -------------------------------------------------------------------- |
| `hivenet_router_admission_rejections_total`  | Counter | `reason`, `model` | Requests rejected by B1 through B4                                   |
| `hivenet_router_admission_occupancy_tokens`  | Gauge   | `model`           | Current token-weighted in-flight sum                                 |
| `hivenet_router_admission_budget_tokens`     | Gauge   | `model`           | Effective B2 budget after admit fraction and healthy-replica scaling |
| `hivenet_router_admission_inflight_requests` | Gauge   | `model`           | Current in-flight request count                                      |
| `hivenet_router_admission_max_inflight`      | Gauge   | `model`           | Effective `max_inflight × healthy_replicas` backstop                 |

The `reason` label identifies the gate:

| Reason         | Trigger                                       |
| -------------- | --------------------------------------------- |
| `b1`           | Input-token or image-count cap                |
| `b2`           | Global occupancy budget or in-flight backstop |
| `b3`           | Live pool-pressure shed                       |
| `b4_occupancy` | Serverless per-key occupancy share            |
| `b4_itpm`      | Serverless per-key input tokens per minute    |
| `b4_otpm`      | Serverless per-key output tokens per minute   |
| `b4_rpm`       | Request-per-minute limit                      |

The budget and backstop gauges can change as healthy replicas join or leave. Existing reservations remain visible in the occupancy gauges while the new denominators affect subsequent admissions.

`POST /v1/messages/count_tokens`, embedding, and reranking requests do not enter B1 through B4, so they do not change these occupancy gauges. See [Admission control metrics](/observability/admission-control-metrics) for panel queries and alert interpretation.

## Per-request duration by agent

```promql theme={null}
hivenet_router_request_duration_seconds
```

Labels:

```text theme={null}
tenant_id
peer_id
model
status_code
```

P95 duration by agent:

```promql theme={null}
histogram_quantile(
  0.95,
  sum by (le, peer_id, model) (
    rate(
      hivenet_router_request_duration_seconds_bucket[5m]
    )
  )
)
```

## HTTP server metrics

### Request duration

```promql theme={null}
http_server_request_duration_seconds
```

Labels:

```text theme={null}
method
route
status_code
```

P95 by route:

```promql theme={null}
histogram_quantile(
  0.95,
  sum by (le, route) (
    rate(
      http_server_request_duration_seconds_bucket[5m]
    )
  )
)
```

Error rate by route:

```promql theme={null}
sum by (route) (
  rate(
    http_server_request_duration_seconds_count{
      status_code=~"5.."
    }[5m]
  )
)
```

### Active requests

```promql theme={null}
http_server_active_requests
```

Labels:

```text theme={null}
method
route
```

```promql theme={null}
http_server_active_requests{
  method="POST",
  route="/v1/chat/completions"
}
```

## Provider HTTP metrics

```promql theme={null}
http_client_request_duration_seconds
```

Labels:

```text theme={null}
provider
status_code
```

P95 provider latency:

```promql theme={null}
histogram_quantile(
  0.95,
  sum by (le, provider) (
    rate(
      http_client_request_duration_seconds_bucket[5m]
    )
  )
)
```

Provider responses by status:

```promql theme={null}
sum by (provider, status_code) (
  rate(
    http_client_request_duration_seconds_count[5m]
  )
)
```

Network failures use:

```text theme={null}
status_code="0"
```

## Alerting examples

The thresholds below are examples. Adjust them to your hardware, workload, and service objectives.

### Agent unhealthy

```yaml theme={null}
groups:
  - name: hivenet-router

    rules:
      - alert: HivenetRouterAgentUnhealthy

        expr: |
          hivenet_router_routing_agent_healthy == 0

        for: 1m

        labels:
          severity: warning

        annotations:
          summary: >-
            Hivenet Router agent {{ $labels.peer_id }} is unhealthy
```

### High routing failure rate

```yaml theme={null}
      - alert: HivenetRouterHighRoutingFailureRate

        expr: |
          (
            sum(
              rate(
                hivenet_router_routing_requests_failed_total[5m]
              )
            )
            /
            clamp_min(
              sum(
                rate(
                  hivenet_router_routing_requests_routed_total[5m]
                )
              )
              +
              sum(
                rate(
                  hivenet_router_routing_requests_failed_total[5m]
                )
              ),
              0.000001
            )
          ) > 0.10

        for: 5m

        labels:
          severity: warning

        annotations:
          summary: >-
            More than 10% of Hivenet Router routing attempts are failing
```

### KV-cache pressure

```yaml theme={null}
      - alert: HivenetRouterKVCachePressure

        expr: |
          hivenet_router_agent_engine_kv_cache_utilization > 0.90

        for: 2m

        labels:
          severity: warning

        annotations:
          summary: >-
            High KV-cache use on {{ $labels.peer_id }}
```

### GPU temperature

```yaml theme={null}
      - alert: HivenetRouterGPUHighTemperature

        expr: |
          hivenet_router_agent_gpu_temperature_celsius > 85

        for: 5m

        labels:
          severity: critical

        annotations:
          summary: >-
            High GPU temperature on {{ $labels.peer_id }}
```

### Policy exhaustion

```yaml theme={null}
      - alert: HivenetRouterPolicyExhaustion

        expr: |
          sum by (model) (
            rate(
              hivenet_router_policy_exhausted_total[5m]
            )
          ) > 0

        for: 5m

        labels:
          severity: warning

        annotations:
          summary: >-
            Routing policy is being exhausted for {{ $labels.model }}
```

### Quota persistence failure

```yaml theme={null}
      - alert: HivenetRouterQuotaPersistenceFailure

        expr: |
          increase(
            hivenet_router_quota_backend_errors_total[15m]
          ) > 0

        labels:
          severity: critical

        annotations:
          summary: >-
            Hivenet Router could not persist daily quota state
```

## Recording rules

Recording rules can simplify expensive or frequently reused dashboard queries.

```yaml theme={null}
groups:
  - name: hivenet-router-recording

    interval: 30s

    rules:
      - record: hivenet_router:model_request_rate5m

        expr: |
          sum by (model) (
            rate(
              hivenet_router_routing_requests_routed_total[5m]
            )
          )

      - record: hivenet_router:model_failure_rate5m

        expr: |
          sum by (model) (
            rate(
              hivenet_router_routing_requests_failed_total[5m]
            )
          )

      - record: hivenet_router:model_p95_ttft_seconds5m

        expr: |
          histogram_quantile(
            0.95,
            sum by (le, model) (
              rate(
                hivenet_router_agent_engine_ttft_seconds_bucket[5m]
              )
            )
          )
```

## Troubleshooting

### Prometheus cannot reach the router

Check the endpoint directly:

```bash theme={null}
curl -i \
  http://<router-host>:2112/metrics
```

Check the router logs:

```bash theme={null}
journalctl \
  -u hivenet-router \
  | grep -i prometheus
```

For Docker Compose:

```bash theme={null}
docker compose exec prometheus \
  wget -qO- \
  http://router:2112/metrics \
  | head
```

Check firewall and container-network rules.

### A metric is missing

A series often appears only after the relevant event or data has occurred.

For example:

* tenant metrics appear after requests
* engine metrics appear after a successful engine scrape
* TTFT and ITL appear after completions
* GPU metrics require NVML and visible NVIDIA devices
* policy counters appear after routing activity
* per-model quota gauges appear when those quota paths are used or seeded

Search by prefix:

```bash theme={null}
curl -s \
  http://localhost:2112/metrics \
  | grep 'hivenet_router_agent_engine'
```

### An engine metric is missing

Check the backend endpoint locally on the agent host:

```bash theme={null}
curl \
  http://localhost:8888/metrics \
  | head
```

Confirm the engine-specific requirement:

* SGLang needs `--enable-metrics`
* llama.cpp needs `--metrics`
* Ollama, Infinity, and custom engines do not currently supply engine metrics

Inspect the agent logs for scrape errors.

A failed metrics scrape does not stop request forwarding.

### Prometheus shows duplicate or stale agents

The `peer_id` label identifies the agent.

If an agent starts with a new identity after every restart, Prometheus sees a new series.

Configure a persistent:

```text theme={null}
--identity-path
```

and preserve the file or mounted volume across restarts.

### Tenant labels show `anonymous` or `default`

In no-auth mode:

```text theme={null}
tenant_id="default"
key_id="anonymous"
```

Static API keys currently use:

```text theme={null}
key_id="anonymous"
```

because stable key IDs belong to dynamic registry entries.

`deployment_id="unset"` is expected before an agent is selected or when the agent does not advertise a deployment ID.

### Histogram queries return no data

Check that the corresponding `_bucket` series exists:

```bash theme={null}
curl -s \
  http://localhost:2112/metrics \
  | grep \
    'hivenet_router_agent_engine_ttft_seconds_bucket'
```

Use `rate()` over a window that contains completed observations.

A new or idle engine may not have enough data yet.

### Counter graphs drop after a restart

Prometheus counters are process-local series.

Some per-agent lifetime counters are reseeded from BadgerDB when agents register again, but other router, policy, HTTP, tenant, and queue metrics begin again with the new router process.

Use `rate()` or `increase()` rather than graphing raw counter values when restart behavior matters.

## Next steps

<CardGroup cols={3}>
  <Card title="Grafana dashboards" href="/observability/grafana-dashboards">
    Explore the provisioned dashboards, data sources, variables, and panels.
  </Card>

  <Card title="Audit logging" href="/observability/audit-logging">
    Query structured request records through Loki or local JSONL files.
  </Card>

  <Card title="Hardware metrics" href="/observability/hardware-metrics">
    Review how agents collect GPU, CPU, and memory data.
  </Card>
</CardGroup>