> ## Documentation Index
> Fetch the complete documentation index at: https://routerdocs.hivenet.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Grafana dashboards

> Use the provisioned Grafana dashboards to inspect router health, agents, hardware, engine behavior, tenant quotas, audit logs, and traces.

The Hivenet Router Docker Compose stack provisions Grafana with dashboards and data sources for metrics, audit logs, and traces.

The repository includes three dashboards:

| Dashboard                     | UID                      | Data source | Purpose                                      |
| ----------------------------- | ------------------------ | ----------- | -------------------------------------------- |
| Hivenet Router Router         | `hivenet-router`         | Prometheus  | Agents, routing, hardware, and engine state  |
| Hivenet Router — Tenant Quota | `hivenet-router-tenants` | Prometheus  | Tenant activity, quotas, tokens, and latency |
| Hivenet Router Audit          | `hivenet-router-audit`   | Loki        | Structured request records and errors        |

Grafana also includes a Tempo data source for exploring distributed traces.

## Open Grafana

When you use the repository’s Docker Compose stack, Grafana is available at:

```text theme={null}
http://<router-host>:3000
```

For a local deployment:

```text theme={null}
http://localhost:3000
```

The checked-in Compose file uses:

```text theme={null}
Username: admin
Password: changeme
```

<Warning>
  Change the Grafana administrator password before exposing port `3000` outside a trusted environment.

  Prefer a VPN, private network, reverse proxy, or SSH tunnel instead of exposing Grafana directly to the internet.
</Warning>

Change the initial password in `docker-compose.yml`:

```yaml theme={null}
grafana:
  environment:
    - GF_SECURITY_ADMIN_PASSWORD=<strong-password>
    - GF_USERS_ALLOW_SIGN_UP=false
```

Then recreate the Grafana container:

```bash theme={null}
docker compose up -d \
  --force-recreate \
  grafana
```

## Use an SSH tunnel

You can access Grafana without opening port `3000` publicly:

```bash theme={null}
ssh -L 3000:localhost:3000 \
  <user>@<router-host>
```

Then open:

```text theme={null}
http://localhost:3000
```

## Open a dashboard directly

Use the dashboard UID in the URL.

### Router dashboard

```text theme={null}
http://<router-host>:3000/d/hivenet-router/
```

### Tenant quota dashboard

```text theme={null}
http://<router-host>:3000/d/hivenet-router-tenants/
```

### Audit dashboard

```text theme={null}
http://<router-host>:3000/d/hivenet-router-audit/
```

Grafana may add a human-readable dashboard name and organization query parameter to the URL after opening it.

## Provisioned data sources

The repository provisions three Grafana data sources.

| Name       | UID          | Internal URL             | Purpose                |
| ---------- | ------------ | ------------------------ | ---------------------- |
| Prometheus | `prometheus` | `http://prometheus:9090` | Metrics and dashboards |
| Loki       | `loki`       | `http://loki:3100`       | Audit and router logs  |
| Tempo      | `tempo`      | `http://tempo:3200`      | Distributed traces     |

Prometheus is the default data source.

The Tempo data source is configured to:

* correlate traces with Loki logs by trace ID
* support node-graph visualization
* search Loki from trace context

These hostnames resolve inside the Docker Compose network. They are not intended as public addresses.

## Hivenet Router Router dashboard

The **Hivenet Router Router** dashboard is the main operational view.

It contains filters for:

* model
* engine
* peer ID
* organization
* machine

Use these filters to narrow every compatible panel to one model, backend type, agent, team, or host.

<Note>
  The current router dashboard does not include a region filter. Region is still available as a Prometheus label and can be used in custom panels or Explore queries.
</Note>

## Router summary

The top row shows:

| Panel             | Meaning                                                                                                                                               |
| ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
| Registered Agents | Current number of registered agents                                                                                                                   |
| Healthy Agents    | Current number of healthy registered agents                                                                                                           |
| Primary Routed    | Requests served by the primary policy                                                                                                                 |
| Fallback Routed   | Requests served by local fallback steps                                                                                                               |
| Exhausted → 503   | Requests whose policy chain was exhausted; the panel title is shorthand, because a failed provider fallback can end with another status such as `502` |
| Provider Fallback | Requests served by an external provider fallback                                                                                                      |

The routing panels use cumulative counters rather than rates. Their values represent the current Prometheus series totals, not necessarily activity within the selected Grafana time range.

Use `rate()` or `increase()` in Explore when you need activity during a specific period.

For example, primary requests during the last hour:

```promql theme={null}
sum(
  increase(
    hivenet_router_policy_primary_routed_total[1h]
  )
)
```

## Agent routing table

The first table combines:

* peer ID
* model
* engine
* organization
* machine
* region and capacity labels where available
* current health
* last heartbeat
* agent failures
* backend failures
* disconnections

Use it to answer questions such as:

* Which agents are currently registered?
* Which agents are unhealthy?
* Is the problem with the agent process or its backend?
* Has one peer repeatedly disconnected?
* Is the expected model registered on the expected host?

## Universal routing state

The **Routing Table — Universal** section includes:

* success rate
* capacity utilization
* SRTT
* RTTVAR
* successful requests
* failed requests
* input tokens
* output tokens
* capacity rejections

These values come from Hivenet Router’s universal per-agent history and can apply across backend types.

### Success rate

```text theme={null}
hivenet_router_agent_success_rate
```

A value below your normal baseline may indicate backend errors, request incompatibility, or connection instability.

### Capacity utilization

```text theme={null}
hivenet_router_agent_capacity_utilization
```

This is:

```text theme={null}
active requests / declared capacity
```

A high value means Hivenet Router has assigned most of the capacity currently represented by its routing slots.

For streaming requests, the current implementation releases the agent slot when response headers arrive, while backend generation can continue. Compare this panel with engine running and waiting requests, KV-cache pressure, TTFT, and ITL when the workload is stream-heavy.

### SRTT and RTTVAR

```text theme={null}
hivenet_router_agent_srtt_ms
hivenet_router_agent_rttvar_ms
```

SRTT represents smoothed request time observed by Hivenet Router.

RTTVAR shows how much that timing varies. A high RTTVAR can indicate unstable or bursty latency even when average latency remains acceptable.

## Hardware tables

The hardware section includes two tables.

### Node resources

The node table shows:

* CPU use
* system-memory use
* available memory
* total memory

### GPU resources

The GPU table shows:

* GPU utilization
* VRAM used
* VRAM free
* total VRAM
* temperature
* power draw

One agent may produce several rows when it reports several GPUs.

CPU-only agents do not produce GPU rows.

## Hardware trends

The router dashboard includes time-series panels for:

* GPU utilization
* VRAM used
* GPU temperature
* GPU power draw
* CPU usage
* memory available

Use these panels to correlate routing or latency changes with system pressure.

For example:

* rising VRAM use may precede out-of-memory failures
* high temperature may coincide with thermal throttling
* falling memory availability may indicate another process competing with inference
* sustained GPU saturation may explain queue growth

The thresholds that matter depend on the hardware and workload. Dashboard values should inform investigation rather than act as universal operating limits.

## Engine metrics table

The **Engine — Backend Metrics** table shows the most recent engine-specific values reported by each agent.

These can include:

* KV-cache utilization
* running requests
* waiting requests
* cumulative preemptions
* average TTFT
* P90 TTFT
* average ITL
* P90 ITL
* generated tokens per second
* prompt tokens per second

Backend support differs:

| Backend                    | Dashboard engine data                   |
| -------------------------- | --------------------------------------- |
| vLLM                       | Broadest supported set                  |
| SGLang                     | Cache, running, waiting, and TTFT       |
| llama.cpp with `--metrics` | Cache, queue, TTFT, ITL, and throughput |
| Ollama                     | No engine-specific values               |
| Infinity                   | No engine-specific values               |
| Custom                     | No engine-specific values               |

A blank cell may mean the backend does not expose the metric, the metric endpoint is disabled, or the agent has not reported a value yet.

## Engine trends

The dashboard includes time-series panels for:

* KV-cache utilization
* running and waiting requests
* preemption rate
* time to first token
* inter-token latency
* token throughput

### Preemption changes

Hivenet Router exports the backend’s latest cumulative preemption value as a gauge, despite the metric’s `_total` suffix. A normal `rate()` query assumes counter semantics and can mislead around backend restarts or resets.

Use this in Explore when you need recent positive growth:

```promql theme={null}
clamp_min(
  delta(
    hivenet_router_agent_engine_preemptions_total[5m]
  ),
  0
)
```

Treat any provisioned panel that applies `rate()` directly to this series as a dashboard limitation until its query is updated.

### TTFT and ITL

The dashboard shows the per-agent average and P90 gauges supplied by Hivenet Router.

For a mathematically correct fleet-wide percentile, use the histogram buckets in Prometheus rather than averaging per-agent P90 values.

For example:

```promql theme={null}
histogram_quantile(
  0.95,
  sum by (le, model) (
    rate(
      hivenet_router_agent_engine_ttft_seconds_bucket[5m]
    )
  )
)
```

## Tenant quota dashboard

The **Hivenet Router — Tenant Quota** dashboard focuses on authenticated tenant activity.

Its filters are:

* tenant ID
* model

The dashboard includes:

* configured RPM limit
* configured daily token limit
* request count
* success rate
* RPM rejections
* token rejections
* most recent request
* time until the UTC budget reset
* request rate compared with the RPM limit
* token-budget utilization
* input and output token split
* average tokens per successful request
* estimated budget runway
* tokens remaining
* request latency percentiles
* request distribution by model

## Tenant snapshot values

The first row is titled:

```text theme={null}
Snapshot — last 24 h
```

However, several stat panels currently query raw cumulative counters rather than using `increase(...[24h])`.

Treat those panels as current cumulative values from the available Prometheus series, not as guaranteed 24-hour totals.

The time-series panels use Grafana’s selected rate interval and respond more directly to the chosen dashboard time range.

## Daily token budget

Daily token panels use:

```text theme={null}
hivenet_router_tenant_tokens_used_today
hivenet_router_tenant_quota_tpd_limit
```

The budget resets at midnight UTC.

The dashboard can show:

* percentage used
* absolute tokens used
* prompt and completion split
* current token-consumption rate
* estimated hours of runway
* tokens remaining

A quota limit of `0` means unlimited. Panels that depend on a finite budget may show no value for unlimited tenants.

## Current tenant-dashboard limitation

The current tenant selector is populated from:

```text theme={null}
hivenet_router_tenant_quota_rpm_limit
```

and several limit panels use the flat quota gauges:

```text theme={null}
hivenet_router_tenant_quota_rpm_limit
hivenet_router_tenant_quota_tpd_limit
```

Tenants using only `quota.per_model` may therefore be missing from the selector or may not have their limits represented fully in this dashboard.

Per-model quota metrics are still available in Prometheus:

```text theme={null}
hivenet_router_tenant_per_model_quota_rpm_limit
hivenet_router_tenant_per_model_quota_tpd_limit
```

Use Grafana Explore or create additional panels when per-model quota visibility is required.

## Audit dashboard

The **Hivenet Router Audit** dashboard uses structured request records stored in Loki.

It requires:

* audit logging enabled in the router
* the audit JSONL path mounted into Promtail
* Promtail sending records to Loki
* the Loki data source available in Grafana

The dashboard filters are:

* tenant ID
* status code
* model
* error code

It contains:

| Panel                  | Purpose                                                |
| ---------------------- | ------------------------------------------------------ |
| Recent Requests        | Search and inspect individual request records          |
| Request Rate by Status | Compare successful and unsuccessful responses          |
| P95 Latency            | Calculate latency from audit records                   |
| Top Models             | Show models with the most audit events                 |
| Error Rate             | Calculate unsuccessful events as a share of all events |

See [Audit logging](/observability/audit-logging) for the record schema, Promtail setup, and LogQL examples.

## Explore traces

The repository does not include a dedicated Tempo dashboard.

Use Grafana’s **Explore** view and select the Tempo data source.

The Docker Compose router configuration includes:

```yaml theme={null}
OTEL_EXPORTER_OTLP_ENDPOINT=tempo:4317
```

When tracing is enabled, Hivenet Router sends spans to Tempo over OTLP gRPC.

From a trace, Grafana can:

* show the span tree
* display a node graph
* search related Loki logs using the trace ID

Trace availability depends on the router receiving the OpenTelemetry endpoint and successfully reaching Tempo.

## Provisioning files

The repository stores dashboards at:

```text theme={null}
deploy/grafana/provisioning/dashboards/
```

Files:

```text theme={null}
hivenet-router.json
tenants.json
audit.json
dashboard.yml
```

Data sources live at:

```text theme={null}
deploy/grafana/provisioning/datasources/
```

Files:

```text theme={null}
prometheus.yml
loki.yml
tempo.yml
```

Docker Compose mounts the complete provisioning directory:

```yaml theme={null}
volumes:
  - ./deploy/grafana/provisioning:/etc/grafana/provisioning:ro
```

Grafana then loads the dashboards and data sources when the container starts.

## Dashboard provisioning behavior

The dashboard provider is configured with:

```yaml theme={null}
apiVersion: 1

providers:
  - name: Hivenet Router
    type: file
    disableDeletion: true
    updateIntervalSeconds: 30

    options:
      path: /etc/grafana/provisioning/dashboards
```

This means:

* the JSON files are the durable source of truth
* Grafana checks the directory every 30 seconds
* provisioned dashboards cannot be deleted through the UI
* durable dashboard changes should be made in the repository files

<Warning>
  Do not rely on an unsaved or manually duplicated UI edit as the maintained version of a provisioned dashboard.

  Update the dashboard JSON and review it through the repository workflow when the change should survive recreation and deployment.
</Warning>

## Import dashboards manually

When you are not using the repository’s Compose stack, you can import the JSON files into another Grafana instance.

In Grafana:

1. Open **Dashboards**.
2. Choose **New** and then **Import**.
3. Upload one of the dashboard JSON files.
4. Map its data source references where prompted.
5. Complete the import.

Import:

```text theme={null}
deploy/grafana/provisioning/dashboards/hivenet-router.json
deploy/grafana/provisioning/dashboards/tenants.json
deploy/grafana/provisioning/dashboards/audit.json
```

The target Grafana instance needs:

* a Prometheus data source with Hivenet Router metrics
* a Loki data source for the audit dashboard
* a Tempo data source when trace exploration is needed

The provisioned dashboard JSON refers to data-source UIDs:

```text theme={null}
prometheus
loki
tempo
```

Use those UIDs or update the imported dashboard references.

## Dashboard variables

The current dashboards define these variables.

### Router dashboard

| Variable        | Source                                          |
| --------------- | ----------------------------------------------- |
| `$model`        | Models from `hivenet_router_routing_agent_info` |
| `$engine`       | Engine values from agent registrations          |
| `$peer_id`      | Agent peer IDs                                  |
| `$organization` | Agent organization metadata                     |
| `$machine`      | Agent machine metadata                          |

### Tenant dashboard

| Variable     | Source                             |
| ------------ | ---------------------------------- |
| `$tenant_id` | Tenant quota metrics               |
| `$model`     | Tenant success and failure metrics |

### Audit dashboard

| Variable       | Source            |
| -------------- | ----------------- |
| `$tenant_id`   | Loki audit labels |
| `$status_code` | Loki audit labels |
| `$model`       | Loki audit labels |
| `$error_code`  | Loki audit labels |

Variables may remain empty until the corresponding metrics or logs exist.

## Build a custom dashboard

Use Grafana Explore to test a query before adding it to a dashboard.

For example, request rate by model:

```promql theme={null}
sum by (model) (
  rate(
    hivenet_router_routing_requests_routed_total[5m]
  )
)
```

Policy exhaustion by model:

```promql theme={null}
sum by (model) (
  rate(
    hivenet_router_policy_exhausted_total[5m]
  )
)
```

P95 tenant latency:

```promql theme={null}
histogram_quantile(
  0.95,
  sum by (le, tenant_id, model) (
    rate(
      hivenet_router_tenant_request_duration_seconds_bucket[5m]
    )
  )
)
```

Audit failures:

```logql theme={null}
{job="hivenet-router", log_type="audit"}
  | json
  | status_code >= 400
```

When the panel is useful:

1. add it to a dashboard
2. export the dashboard JSON
3. review the resulting changes
4. update the appropriate file under `deploy/grafana/provisioning/dashboards/`

## Alerting

The repository provisions dashboards but does not currently provision Grafana alert rules.

You can create alert rules in Grafana or manage them through Prometheus.

For a version-controlled infrastructure setup, Prometheus rule files are usually easier to review, reproduce, and deploy consistently.

See [Prometheus metrics](/observability/prometheus-metrics#alerting-examples) for example alert rules.

## Troubleshooting

### Grafana does not start

Check the container:

```bash theme={null}
docker compose ps grafana
```

Inspect its logs:

```bash theme={null}
docker compose logs grafana \
  | tail -100
```

Common causes include:

* invalid provisioning YAML
* unreadable mounted files
* an unwritable Grafana data volume
* an occupied host port `3000`

### Dashboards are missing

Check the mounted directory:

```bash theme={null}
docker compose exec grafana \
  find /etc/grafana/provisioning \
  -maxdepth 3 \
  -type f
```

Confirm that it contains the dashboard JSON files and `dashboard.yml`.

Check provisioning logs:

```bash theme={null}
docker compose logs grafana \
  | grep -i "provision\|dashboard"
```

### A data source reports an error

Test connectivity from the Grafana container.

Prometheus:

```bash theme={null}
docker compose exec grafana \
  wget -qO- \
  http://prometheus:9090/-/ready
```

Loki:

```bash theme={null}
docker compose exec grafana \
  wget -qO- \
  http://loki:3100/ready
```

Tempo:

```bash theme={null}
docker compose exec grafana \
  wget -qO- \
  http://tempo:3200/ready
```

Inspect the corresponding service logs when a check fails.

### The router dashboard has no data

Check the Prometheus target:

```bash theme={null}
docker compose exec prometheus \
  wget -qO- \
  http://localhost:9090/api/v1/targets \
  | jq '.data.activeTargets[] | {
      health,
      lastError
    }'
```

Query one registration metric:

```bash theme={null}
docker compose exec prometheus \
  wget -qO- \
  'http://localhost:9090/api/v1/query?query=hivenet_router_routing_agent_info' \
  | jq .
```

If the query is empty:

* confirm that agents are registered
* confirm that Prometheus scrapes `router:2112`
* inspect router and Prometheus logs
* check whether the selected dashboard filters exclude all agents

### Hardware panels are empty

Check whether the agent reports hardware data:

```bash theme={null}
curl \
  -H "Authorization: Bearer <admin-api-key>" \
  http://localhost:8080/admin/routing-table \
  | jq '.agents[] | {
      peer_id,
      hardware
    }'
```

For GPU panels, check:

* NVIDIA drivers
* NVML availability
* container GPU access
* the agent process permissions

CPU-only agents do not produce GPU series.

### Engine panels are empty

Check backend support and configuration:

* SGLang needs `--enable-metrics`
* llama.cpp needs `--metrics`
* Ollama, Infinity, and custom engines do not currently supply engine metrics

Test the engine endpoint on the agent host:

```bash theme={null}
curl http://localhost:8888/metrics \
  | head
```

Inspect the agent logs for scrape errors.

### The tenant dashboard has no tenants

Tenant metrics appear only after authenticated or no-auth traffic creates them.

Check:

```bash theme={null}
curl -s \
  http://localhost:2112/metrics \
  | grep hivenet_router_tenant
```

A tenant using only per-model quota gauges may not appear in the current selector. Query the per-model metrics directly in Explore when necessary.

### The audit dashboard is empty

Confirm that audit records exist:

```bash theme={null}
docker compose exec router \
  ls -la /var/log/hivenet-router
```

Check Promtail:

```bash theme={null}
docker compose logs promtail \
  | tail -100
```

Check Loki:

```bash theme={null}
docker compose logs loki \
  | tail -100
```

Query Loki through its API:

```bash theme={null}
docker compose exec grafana \
  wget -qO- \
  'http://loki:3100/loki/api/v1/query?query=%7Bjob%3D%22hivenet-router%22%2Clog_type%3D%22audit%22%7D' \
  | head
```

### Trace search is empty

Check that the router received:

```text theme={null}
OTEL_EXPORTER_OTLP_ENDPOINT=tempo:4317
```

Inspect the router logs for tracing initialization or export errors.

Confirm that Tempo is ready:

```bash theme={null}
docker compose exec grafana \
  wget -qO- \
  http://tempo:3200/ready
```

### Dashboard edits disappear

The dashboards are provisioned from files.

Make durable changes in:

```text theme={null}
deploy/grafana/provisioning/dashboards/
```

then allow Grafana’s provisioning interval to reload them, or restart Grafana:

```bash theme={null}
docker compose restart grafana
```

## Next steps

<CardGroup cols={3}>
  <Card title="Audit logging" href="/observability/audit-logging">
    Configure structured request records and query them through Loki.
  </Card>

  <Card title="Hardware metrics" href="/observability/hardware-metrics">
    Understand the GPU, CPU, and memory values shown in Grafana.
  </Card>

  <Card title="Engine metrics" href="/observability/engine-metrics">
    Review cache, queue, TTFT, ITL, and throughput metrics by backend.
  </Card>
</CardGroup>