Grafana dashboards - Hivenet Router

The Hivenet Router Docker Compose stack provisions Grafana with dashboards and data sources for metrics, audit logs, and traces. The repository includes three dashboards:

Dashboard	UID	Data source	Purpose
Hivenet Router Router	`hivenet-router`	Prometheus	Agents, routing, hardware, and engine state
Hivenet Router — Tenant Quota	`hivenet-router-tenants`	Prometheus	Tenant activity, quotas, tokens, and latency
Hivenet Router Audit	`hivenet-router-audit`	Loki	Structured request records and errors

Grafana also includes a Tempo data source for exploring distributed traces.

Open Grafana

When you use the repository’s Docker Compose stack, Grafana is available at:

http://<router-host>:3000

For a local deployment:

http://localhost:3000

The checked-in Compose file uses:

Username: admin
Password: changeme

Change the Grafana administrator password before exposing port 3000 outside a trusted environment.Prefer a VPN, private network, reverse proxy, or SSH tunnel instead of exposing Grafana directly to the internet.

Change the initial password in docker-compose.yml:

grafana:
  environment:
    - GF_SECURITY_ADMIN_PASSWORD=<strong-password>
    - GF_USERS_ALLOW_SIGN_UP=false

Then recreate the Grafana container:

docker compose up -d \
  --force-recreate \
  grafana

Use an SSH tunnel

You can access Grafana without opening port 3000 publicly:

ssh -L 3000:localhost:3000 \
  <user>@<router-host>

Then open:

http://localhost:3000

Open a dashboard directly

Use the dashboard UID in the URL.

Router dashboard

http://<router-host>:3000/d/hivenet-router/

Tenant quota dashboard

http://<router-host>:3000/d/hivenet-router-tenants/

Audit dashboard

http://<router-host>:3000/d/hivenet-router-audit/

Grafana may add a human-readable dashboard name and organization query parameter to the URL after opening it.

Provisioned data sources

The repository provisions three Grafana data sources.

Name	UID	Internal URL	Purpose
Prometheus	`prometheus`	`http://prometheus:9090`	Metrics and dashboards
Loki	`loki`	`http://loki:3100`	Audit and router logs
Tempo	`tempo`	`http://tempo:3200`	Distributed traces

Prometheus is the default data source. The Tempo data source is configured to:

correlate traces with Loki logs by trace ID
support node-graph visualization
search Loki from trace context

These hostnames resolve inside the Docker Compose network. They are not intended as public addresses.

Hivenet Router Router dashboard

The Hivenet Router Router dashboard is the main operational view. It contains filters for:

model
engine
peer ID
organization
machine

Use these filters to narrow every compatible panel to one model, backend type, agent, team, or host.

The current router dashboard does not include a region filter. Region is still available as a Prometheus label and can be used in custom panels or Explore queries.

Router summary

The top row shows:

Panel	Meaning
Registered Agents	Current number of registered agents
Healthy Agents	Current number of healthy registered agents
Primary Routed	Requests served by the primary policy
Fallback Routed	Requests served by local fallback steps
Exhausted → 503	Requests whose policy chain was exhausted; the panel title is shorthand, because a failed provider fallback can end with another status such as `502`
Provider Fallback	Requests served by an external provider fallback

The routing panels use cumulative counters rather than rates. Their values represent the current Prometheus series totals, not necessarily activity within the selected Grafana time range. Use rate() or increase() in Explore when you need activity during a specific period. For example, primary requests during the last hour:

sum(
  increase(
    hivenet_router_policy_primary_routed_total[1h]
  )
)

Agent routing table

The first table combines:

peer ID
model
engine
organization
machine
region and capacity labels where available
current health
last heartbeat
agent failures
backend failures
disconnections

Use it to answer questions such as:

Which agents are currently registered?
Which agents are unhealthy?
Is the problem with the agent process or its backend?
Has one peer repeatedly disconnected?
Is the expected model registered on the expected host?

Universal routing state

The Routing Table — Universal section includes:

success rate
capacity utilization
SRTT
RTTVAR
successful requests
failed requests
input tokens
output tokens
capacity rejections

These values come from Hivenet Router’s universal per-agent history and can apply across backend types.

Success rate

hivenet_router_agent_success_rate

A value below your normal baseline may indicate backend errors, request incompatibility, or connection instability.

Capacity utilization

hivenet_router_agent_capacity_utilization

This is:

active requests / declared capacity

A high value means Hivenet Router has assigned most of the capacity currently represented by its routing slots. For streaming requests, the current implementation releases the agent slot when response headers arrive, while backend generation can continue. Compare this panel with engine running and waiting requests, KV-cache pressure, TTFT, and ITL when the workload is stream-heavy.

SRTT and RTTVAR

hivenet_router_agent_srtt_ms
hivenet_router_agent_rttvar_ms

SRTT represents smoothed request time observed by Hivenet Router. RTTVAR shows how much that timing varies. A high RTTVAR can indicate unstable or bursty latency even when average latency remains acceptable.

Hardware tables

The hardware section includes two tables.

Node resources

The node table shows:

CPU use
system-memory use
available memory
total memory

GPU resources

The GPU table shows:

GPU utilization
VRAM used
VRAM free
total VRAM
temperature
power draw

One agent may produce several rows when it reports several GPUs. CPU-only agents do not produce GPU rows.

Hardware trends

The router dashboard includes time-series panels for:

GPU utilization
VRAM used
GPU temperature
GPU power draw
CPU usage
memory available

Use these panels to correlate routing or latency changes with system pressure. For example:

rising VRAM use may precede out-of-memory failures
high temperature may coincide with thermal throttling
falling memory availability may indicate another process competing with inference
sustained GPU saturation may explain queue growth

The thresholds that matter depend on the hardware and workload. Dashboard values should inform investigation rather than act as universal operating limits.

Engine metrics table

The Engine — Backend Metrics table shows the most recent engine-specific values reported by each agent. These can include:

KV-cache utilization
running requests
waiting requests
cumulative preemptions
average TTFT
P90 TTFT
average ITL
P90 ITL
generated tokens per second
prompt tokens per second

Backend support differs:

Backend	Dashboard engine data
vLLM	Broadest supported set
SGLang	Cache, running, waiting, and TTFT
llama.cpp with `--metrics`	Cache, queue, TTFT, ITL, and throughput
Ollama	No engine-specific values
Infinity	No engine-specific values
Custom	No engine-specific values

A blank cell may mean the backend does not expose the metric, the metric endpoint is disabled, or the agent has not reported a value yet.

Engine trends

The dashboard includes time-series panels for:

KV-cache utilization
running and waiting requests
preemption rate
time to first token
inter-token latency
token throughput

Preemption changes

Hivenet Router exports the backend’s latest cumulative preemption value as a gauge, despite the metric’s _total suffix. A normal rate() query assumes counter semantics and can mislead around backend restarts or resets. Use this in Explore when you need recent positive growth:

clamp_min(
  delta(
    hivenet_router_agent_engine_preemptions_total[5m]
  ),
  0
)

Treat any provisioned panel that applies rate() directly to this series as a dashboard limitation until its query is updated.

TTFT and ITL

The dashboard shows the per-agent average and P90 gauges supplied by Hivenet Router. For a mathematically correct fleet-wide percentile, use the histogram buckets in Prometheus rather than averaging per-agent P90 values. For example:

histogram_quantile(
  0.95,
  sum by (le, model) (
    rate(
      hivenet_router_agent_engine_ttft_seconds_bucket[5m]
    )
  )
)

Tenant quota dashboard

The Hivenet Router — Tenant Quota dashboard focuses on authenticated tenant activity. Its filters are:

tenant ID
model

The dashboard includes:

configured RPM limit
configured daily token limit
request count
success rate
RPM rejections
token rejections
most recent request
time until the UTC budget reset
request rate compared with the RPM limit
token-budget utilization
input and output token split
average tokens per successful request
estimated budget runway
tokens remaining
request latency percentiles
request distribution by model

Tenant snapshot values

The first row is titled:

Snapshot — last 24 h

However, several stat panels currently query raw cumulative counters rather than using increase(...[24h]). Treat those panels as current cumulative values from the available Prometheus series, not as guaranteed 24-hour totals. The time-series panels use Grafana’s selected rate interval and respond more directly to the chosen dashboard time range.

Daily token budget

Daily token panels use:

hivenet_router_tenant_tokens_used_today
hivenet_router_tenant_quota_tpd_limit

The budget resets at midnight UTC. The dashboard can show:

percentage used
absolute tokens used
prompt and completion split
current token-consumption rate
estimated hours of runway
tokens remaining

A quota limit of 0 means unlimited. Panels that depend on a finite budget may show no value for unlimited tenants.

Current tenant-dashboard limitation

The current tenant selector is populated from:

hivenet_router_tenant_quota_rpm_limit

and several limit panels use the flat quota gauges:

hivenet_router_tenant_quota_rpm_limit
hivenet_router_tenant_quota_tpd_limit

Tenants using only quota.per_model may therefore be missing from the selector or may not have their limits represented fully in this dashboard. Per-model quota metrics are still available in Prometheus:

hivenet_router_tenant_per_model_quota_rpm_limit
hivenet_router_tenant_per_model_quota_tpd_limit

Use Grafana Explore or create additional panels when per-model quota visibility is required.

Audit dashboard

The Hivenet Router Audit dashboard uses structured request records stored in Loki. It requires:

audit logging enabled in the router
the audit JSONL path mounted into Promtail
Promtail sending records to Loki
the Loki data source available in Grafana

The dashboard filters are:

tenant ID
status code
model
error code

It contains:

Panel	Purpose
Recent Requests	Search and inspect individual request records
Request Rate by Status	Compare successful and unsuccessful responses
P95 Latency	Calculate latency from audit records
Top Models	Show models with the most audit events
Error Rate	Calculate unsuccessful events as a share of all events

See Audit logging for the record schema, Promtail setup, and LogQL examples.

Explore traces

The repository does not include a dedicated Tempo dashboard. Use Grafana’s Explore view and select the Tempo data source. The Docker Compose router configuration includes:

OTEL_EXPORTER_OTLP_ENDPOINT=tempo:4317

When tracing is enabled, Hivenet Router sends spans to Tempo over OTLP gRPC. From a trace, Grafana can:

show the span tree
display a node graph
search related Loki logs using the trace ID

Trace availability depends on the router receiving the OpenTelemetry endpoint and successfully reaching Tempo.

Provisioning files

The repository stores dashboards at:

deploy/grafana/provisioning/dashboards/

Files:

hivenet-router.json
tenants.json
audit.json
dashboard.yml

Data sources live at:

deploy/grafana/provisioning/datasources/

Files:

prometheus.yml
loki.yml
tempo.yml

Docker Compose mounts the complete provisioning directory:

volumes:
  - ./deploy/grafana/provisioning:/etc/grafana/provisioning:ro

Grafana then loads the dashboards and data sources when the container starts.

Dashboard provisioning behavior

The dashboard provider is configured with:

apiVersion: 1

providers:
  - name: Hivenet Router
    type: file
    disableDeletion: true
    updateIntervalSeconds: 30

    options:
      path: /etc/grafana/provisioning/dashboards

This means:

the JSON files are the durable source of truth
Grafana checks the directory every 30 seconds
provisioned dashboards cannot be deleted through the UI
durable dashboard changes should be made in the repository files

Do not rely on an unsaved or manually duplicated UI edit as the maintained version of a provisioned dashboard.Update the dashboard JSON and review it through the repository workflow when the change should survive recreation and deployment.

Import dashboards manually

When you are not using the repository’s Compose stack, you can import the JSON files into another Grafana instance. In Grafana:

Open Dashboards.
Choose New and then Import.
Upload one of the dashboard JSON files.
Map its data source references where prompted.
Complete the import.

Import:

deploy/grafana/provisioning/dashboards/hivenet-router.json
deploy/grafana/provisioning/dashboards/tenants.json
deploy/grafana/provisioning/dashboards/audit.json

The target Grafana instance needs:

a Prometheus data source with Hivenet Router metrics
a Loki data source for the audit dashboard
a Tempo data source when trace exploration is needed

The provisioned dashboard JSON refers to data-source UIDs:

prometheus
loki
tempo

Use those UIDs or update the imported dashboard references.

Dashboard variables

The current dashboards define these variables.

Router dashboard

Variable	Source
`$model`	Models from `hivenet_router_routing_agent_info`
`$engine`	Engine values from agent registrations
`$peer_id`	Agent peer IDs
`$organization`	Agent organization metadata
`$machine`	Agent machine metadata

Tenant dashboard

Variable	Source
`$tenant_id`	Tenant quota metrics
`$model`	Tenant success and failure metrics

Audit dashboard

Variable	Source
`$tenant_id`	Loki audit labels
`$status_code`	Loki audit labels
`$model`	Loki audit labels
`$error_code`	Loki audit labels

Variables may remain empty until the corresponding metrics or logs exist.

Build a custom dashboard

Use Grafana Explore to test a query before adding it to a dashboard. For example, request rate by model:

sum by (model) (
  rate(
    hivenet_router_routing_requests_routed_total[5m]
  )
)

Policy exhaustion by model:

sum by (model) (
  rate(
    hivenet_router_policy_exhausted_total[5m]
  )
)

P95 tenant latency:

histogram_quantile(
  0.95,
  sum by (le, tenant_id, model) (
    rate(
      hivenet_router_tenant_request_duration_seconds_bucket[5m]
    )
  )
)

Audit failures:

{job="hivenet-router", log_type="audit"}
  | json
  | status_code >= 400

When the panel is useful:

add it to a dashboard
export the dashboard JSON
review the resulting changes
update the appropriate file under deploy/grafana/provisioning/dashboards/

Alerting

The repository provisions dashboards but does not currently provision Grafana alert rules. You can create alert rules in Grafana or manage them through Prometheus. For a version-controlled infrastructure setup, Prometheus rule files are usually easier to review, reproduce, and deploy consistently. See Prometheus metrics for example alert rules.

Troubleshooting

Grafana does not start

Check the container:

docker compose ps grafana

Inspect its logs:

docker compose logs grafana \
  | tail -100

Common causes include:

invalid provisioning YAML
unreadable mounted files
an unwritable Grafana data volume
an occupied host port 3000

Dashboards are missing

Check the mounted directory:

docker compose exec grafana \
  find /etc/grafana/provisioning \
  -maxdepth 3 \
  -type f

Confirm that it contains the dashboard JSON files and dashboard.yml. Check provisioning logs:

docker compose logs grafana \
  | grep -i "provision\|dashboard"

A data source reports an error

Test connectivity from the Grafana container. Prometheus:

docker compose exec grafana \
  wget -qO- \
  http://prometheus:9090/-/ready

Loki:

docker compose exec grafana \
  wget -qO- \
  http://loki:3100/ready

Tempo:

docker compose exec grafana \
  wget -qO- \
  http://tempo:3200/ready

Inspect the corresponding service logs when a check fails.

The router dashboard has no data

Check the Prometheus target:

docker compose exec prometheus \
  wget -qO- \
  http://localhost:9090/api/v1/targets \
  | jq '.data.activeTargets[] | {
      health,
      lastError
    }'

Query one registration metric:

docker compose exec prometheus \
  wget -qO- \
  'http://localhost:9090/api/v1/query?query=hivenet_router_routing_agent_info' \
  | jq .

If the query is empty:

confirm that agents are registered
confirm that Prometheus scrapes router:2112
inspect router and Prometheus logs
check whether the selected dashboard filters exclude all agents

Hardware panels are empty

Check whether the agent reports hardware data:

curl \
  -H "Authorization: Bearer <admin-api-key>" \
  http://localhost:8080/admin/routing-table \
  | jq '.agents[] | {
      peer_id,
      hardware
    }'

For GPU panels, check:

NVIDIA drivers
NVML availability
container GPU access
the agent process permissions

CPU-only agents do not produce GPU series.

Engine panels are empty

Check backend support and configuration:

SGLang needs --enable-metrics
llama.cpp needs --metrics
Ollama, Infinity, and custom engines do not currently supply engine metrics

Test the engine endpoint on the agent host:

curl http://localhost:8888/metrics \
  | head

Inspect the agent logs for scrape errors.

The tenant dashboard has no tenants

Tenant metrics appear only after authenticated or no-auth traffic creates them. Check:

curl -s \
  http://localhost:2112/metrics \
  | grep hivenet_router_tenant

A tenant using only per-model quota gauges may not appear in the current selector. Query the per-model metrics directly in Explore when necessary.

The audit dashboard is empty

Confirm that audit records exist:

docker compose exec router \
  ls -la /var/log/hivenet-router

Check Promtail:

docker compose logs promtail \
  | tail -100

Check Loki:

docker compose logs loki \
  | tail -100

Query Loki through its API:

docker compose exec grafana \
  wget -qO- \
  'http://loki:3100/loki/api/v1/query?query=%7Bjob%3D%22hivenet-router%22%2Clog_type%3D%22audit%22%7D' \
  | head

Trace search is empty

Check that the router received:

OTEL_EXPORTER_OTLP_ENDPOINT=tempo:4317

Inspect the router logs for tracing initialization or export errors. Confirm that Tempo is ready:

docker compose exec grafana \
  wget -qO- \
  http://tempo:3200/ready

Dashboard edits disappear

The dashboards are provisioned from files. Make durable changes in:

deploy/grafana/provisioning/dashboards/

then allow Grafana’s provisioning interval to reload them, or restart Grafana:

docker compose restart grafana

Next steps

Audit logging

Configure structured request records and query them through Loki.

Hardware metrics

Understand the GPU, CPU, and memory values shown in Grafana.

Engine metrics

Review cache, queue, TTFT, ITL, and throughput metrics by backend.

​Open Grafana

​Use an SSH tunnel

​Open a dashboard directly

​Router dashboard

​Tenant quota dashboard

​Audit dashboard

​Provisioned data sources

​Hivenet Router Router dashboard

​Router summary

​Agent routing table

​Universal routing state

​Success rate

​Capacity utilization

​SRTT and RTTVAR

​Hardware tables

​Node resources

​GPU resources

​Hardware trends

​Engine metrics table

​Engine trends

​Preemption changes

​TTFT and ITL

​Tenant quota dashboard

​Tenant snapshot values

​Daily token budget

​Current tenant-dashboard limitation

​Audit dashboard

​Explore traces

​Provisioning files

​Dashboard provisioning behavior

​Import dashboards manually

​Dashboard variables

​Router dashboard

​Tenant dashboard

​Audit dashboard

​Build a custom dashboard

​Alerting

​Troubleshooting

​Grafana does not start

​Dashboards are missing

​A data source reports an error

​The router dashboard has no data

​Hardware panels are empty

​Engine panels are empty

​The tenant dashboard has no tenants

​The audit dashboard is empty

​Trace search is empty

​Dashboard edits disappear

​Next steps

Audit logging

Hardware metrics

Engine metrics

Open Grafana

Use an SSH tunnel

Open a dashboard directly

Router dashboard

Tenant quota dashboard

Audit dashboard

Provisioned data sources

Hivenet Router Router dashboard

Router summary

Agent routing table

Universal routing state

Success rate

Capacity utilization

SRTT and RTTVAR

Hardware tables

Node resources

GPU resources

Hardware trends

Engine metrics table

Engine trends

Preemption changes

TTFT and ITL

Tenant quota dashboard

Tenant snapshot values

Daily token budget

Current tenant-dashboard limitation

Audit dashboard

Explore traces

Provisioning files

Dashboard provisioning behavior

Import dashboards manually

Dashboard variables

Router dashboard

Tenant dashboard

Audit dashboard

Build a custom dashboard

Alerting

Troubleshooting

Grafana does not start

Dashboards are missing

A data source reports an error

The router dashboard has no data

Hardware panels are empty

Engine panels are empty

The tenant dashboard has no tenants

The audit dashboard is empty

Trace search is empty

Dashboard edits disappear

Next steps