Audit logging - Hivenet Router

Hivenet Router writes one structured JSON audit record after each HTTP request completes. Audit records help operators investigate request outcomes, tenant activity, latency, model use, authentication failures, routing errors, and token consumption without storing prompt or response content.

Audit records still contain potentially sensitive metadata, including tenant identifiers, model names, source IP addresses, and request timing.Protect the files, restrict Loki and Grafana access, and define retention according to your organization’s legal and operational requirements.

What Hivenet Router audits

The audit middleware covers HTTP requests handled by the router, including:

inference requests under /v1/*
model-discovery requests
administration requests under /admin/*
authentication and quota failures
successful and unsuccessful backend responses

Hivenet Router deliberately skips:

GET /health liveness probes
CORS OPTIONS preflight requests

These high-frequency infrastructure requests do not carry useful inference or tenant information.

Audit record format

The default file is:

/var/log/hivenet-router/audit.jsonl

Each request produces one JSON object on one line:

{
  "log_type": "audit",
  "level": "info",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "request_id": "8b9e4a6d-dfb4-4b95-bada-319e7e5b878a",
  "tenant_id": "acme-corp",
  "key_id": "acme-prod",
  "model": "meta-llama/Llama-3.1-8B-Instruct",
  "status_code": 200,
  "latency_ms": 1523,
  "input_tokens": 15,
  "output_tokens": 42,
  "agent_id": "12D3KooW...",
  "error_code": "",
  "source_ip": "10.0.0.5",
  "ts": "2026-07-27T16:20:10.123+02:00"
}

The timestamp and values above are examples.

Audit fields

Field	Description
`log_type`	Always `audit`
`level`	`info` below HTTP `400`; `error` for HTTP `400` and above
`trace_id`	OpenTelemetry trace ID; empty when no valid trace is active
`request_id`	Valid client-supplied UUID or a generated UUID
`tenant_id`	Authenticated key owner; `default` in no-auth mode; empty for failures before identity resolution
`key_id`	Dynamic key-registry ID; normally empty for static keys and no-auth traffic
`model`	Top-level request model when the handler could read it
`status_code`	Final HTTP response status
`latency_ms`	Time from request entry until the response completes
`input_tokens`	Input-token count available to the request handler
`output_tokens`	Output-token count available to the request handler
`agent_id`	Selected agent peer ID or provider marker; empty when no dispatch occurred
`error_code`	Hivenet Router domain error code; empty for successful responses
`source_ip`	Client IP resolved by Gin
`ts`	Request start time in the `Europe/Rome` time zone, including the active UTC offset

Request IDs

Hivenet Router accepts an incoming:

X-Request-ID

only when its value is a valid UUID. A valid UUID is preserved:

curl \
  -H "X-Request-ID: 8b9e4a6d-dfb4-4b95-bada-319e7e5b878a" \
  http://localhost:8080/v1/models

An absent, empty, or invalid value is replaced with a new UUID v4. The selected value is:

returned in the X-Request-ID response header
stored in the audit record
attached to the active trace span

This prevents arbitrary request-ID text from being injected into audit records.

Trace IDs

When OpenTelemetry tracing is active, trace_id contains the current span’s trace ID. Hivenet Router also returns a W3C:

traceparent

response header when a valid trace context exists. Use either identifier to move between:

an application request
the JSONL audit record
Loki logs
a Tempo trace

When tracing is disabled or unavailable:

{
  "trace_id": ""
}

Tenant and key identity

For authenticated client requests:

tenant_id

comes from the key’s owner. For dynamic keys:

key_id

contains the stable registry entry ID. Static keys do not currently have a stable key ID, so their audit records normally contain:

{
  "key_id": ""
}

In no-auth mode, successful API requests use:

{
  "tenant_id": "default",
  "key_id": ""
}

Authentication failures happen before Hivenet Router can resolve a tenant or key, so those fields remain empty.

Audit records and Prometheus labels use different empty-key conventions.Static and no-auth audit records normally use an empty key_id, while tenant metrics use key_id="anonymous" for those requests.

Agent and provider identity

For a successful local inference request:

agent_id

contains the selected agent’s libp2p peer ID. For external provider fallback, the value may identify the provider instead:

provider:openai

or:

provider:anthropic

The field remains empty when the request fails before an agent or provider is selected.

Token fields

For successful non-streaming chat requests, Hivenet Router records the usage available from the completed response. For streaming chat requests, Hivenet Router updates the audit values after the stream closes and the token meter has finished. Embedding and reranking requests currently record:

{
  "input_tokens": 0,
  "output_tokens": 0
}

Requests rejected before inference may also contain zero token values.

Audit token values describe what Hivenet Router could account for during that request. Their precision depends on the backend response and the request type.

What is not logged

The dedicated audit record does not include:

prompt or message content
response or completion content
request body
response body
authorization header
raw API keys
provider credentials
complete HTTP headers
HTTP method
request path

Hivenet Router’s separate application log records method, path, status, latency, and client IP, but deliberately excludes the authorization header.

Metadata-only logging reduces content exposure but does not make the records anonymous.A model name, tenant ID, source IP, request time, or token count may still be personal, confidential, or commercially sensitive.

Error codes

When a request handler provides a specific Hivenet Router domain code, the audit record preserves it. Common inference codes include:

Error code	Typical HTTP status	Meaning
`request_invalid`	`400`	Malformed request or missing required data
`context_length_exceeded`	`400`	Backend rejected the context length
`invalid_parameter`	`400`	Backend rejected an inference parameter
`unauthorized`	`401`	Missing or invalid credentials
`model_forbidden`	`403`	Key cannot use the requested model
`model_not_found`	`404`	Requested model is unavailable or hidden
`rate_limit_exceeded`	`429`	Request-rate quota or undeclared per-model quota
`token_limit_exceeded`	`429`	Daily token budget exhausted
`backend_error`	`500` or `502`	Internal, backend, provider, or response-format failure
`no_agents_available`	`503`	No healthy matching agent
`no_capacity`	`503`	Matching agents are full
`agent_disconnected`	`503`	Selected agent became unreachable
`backend_unavailable`	`503`	Inference backend was not ready
`queue_full`	`503`	The router’s global pending-request channel remained full during the enqueue window
`request_timeout`	`504`	Request exceeded its deadline

When no more specific domain code was attached, the middleware derives a fallback code from the HTTP status:

HTTP status	Fallback audit code
`400`	`request_invalid`
`401`	`unauthorized`
`403`	`model_forbidden`
`404`	`model_not_found`
`429`	`rate_limit_exceeded`
`503`	`no_agents_available`
`504`	`request_timeout`
Other error status	`backend_error`

An oversized /v1/* request can be rejected with HTTP 413 before the normal inference-handler error path. In that case, treat status_code as authoritative; the audit error_code may be a broad status-derived value rather than a dedicated request-size code. Some administration endpoints return simpler endpoint-specific errors without attaching a domain code. Their audit record may therefore contain the broader status-derived value. See Error codes for the inference error reference.

Configure the file path

Set the case-sensitive environment variable:

export HIVENET_ROUTER_AUDIT_LOG_PATH=/custom/path/audit.jsonl

Then start the router:

./bin/hivenet-router \
  ...

The parent directory is created automatically when possible. The file is opened for append when the router process starts.

Restart the router after changing HIVENET_ROUTER_AUDIT_LOG_PATH. The audit logger does not reopen the file dynamically after an environment change.

Fallback to stdout

If Hivenet Router cannot create the directory or open the file, the router still starts and writes audit JSON to standard output. It also writes an explanatory message to standard error:

audit: cannot open <path>: <error> — falling back to stdout

This is useful for local development, but it can mix structured audit entries with application logs and change the expected Promtail pipeline. Check startup logs whenever the audit file remains empty.

Bare-metal permissions

Create a protected directory for a service running as hivenet-router:

sudo install -d \
  -o hivenet-router \
  -g hivenet-router \
  -m 0750 \
  /var/log/hivenet-router

Set the environment variable in the router service:

[Service]
Environment=HIVENET_ROUTER_AUDIT_LOG_PATH=/var/log/hivenet-router/audit.jsonl

Hivenet Router creates the file when it starts. Its final permissions are affected by the process umask. With the hardened systemd unit from the bare-metal guide:

UMask=0027

the requested file mode is reduced accordingly. Check the result:

sudo ls -l \
  /var/log/hivenet-router/audit.jsonl

Docker Compose pipeline

The repository’s Compose stack already connects the router’s audit file to Promtail. The router writes to a named volume:

router:
  volumes:
    - audit_logs:/var/log/hivenet-router

Promtail mounts the same volume read-only:

promtail:
  volumes:
    - audit_logs:/var/log/hivenet-router:ro

The volume declaration is:

volumes:
  audit_logs:

Start the full stack:

docker compose up -d --build

Check that the router writes records:

docker compose exec router \
  tail -n 5 \
  /var/log/hivenet-router/audit.jsonl

Check Promtail:

docker compose logs promtail \
  | tail -100

Check Loki readiness:

docker compose exec grafana \
  wget -qO- \
  http://loki:3100/ready

Promtail configuration

The repository separates application and audit logs into two jobs. The audit job reads only the JSONL file:

- job_name: hivenet-router-audit

  static_configs:
    - targets:
        - localhost

      labels:
        job: hivenet-router
        log_type: audit
        __path__: /var/log/hivenet-router/audit.jsonl

  pipeline_stages:
    - json:
        expressions:
          tenant_id: tenant_id
          status_code: status_code
          model: model
          error_code: error_code
          level: level

    - labels:
        tenant_id:
        status_code:
        model:
        error_code:
        level:

The resulting Loki stream selector is:

{job="hivenet-router", log_type="audit"}

Promtail promotes these JSON fields to labels:

tenant_id
status_code
model
error_code
level

Other fields remain available through query-time JSON parsing.

Label-cardinality guidance

Tenant and model labels are useful for dashboards, but they increase the number of Loki streams. For a larger deployment, review whether these should remain indexed labels:

tenant_id
model

High numbers of tenants, models, or frequently changing label values can increase Loki memory and storage use. A lower-cardinality alternative is to label only stable fields such as:

job
log_type
level
status_code

and parse tenant and model at query time with:

| json

Any pipeline change also requires corresponding dashboard-variable and query updates.

Query audit records

All audit records

{job="hivenet-router", log_type="audit"}
  | json

Errors

{job="hivenet-router", log_type="audit"}
  | json
  | status_code >= 400

You can also use the promoted label:

{
  job="hivenet-router",
  log_type="audit",
  level="error"
}

One tenant

{
  job="hivenet-router",
  log_type="audit",
  tenant_id="acme-corp"
}
  | json

One model

{
  job="hivenet-router",
  log_type="audit",
  model="meta-llama/Llama-3.1-8B-Instruct"
}
  | json

One request ID

request_id is not a Loki label, so parse the JSON field:

{job="hivenet-router", log_type="audit"}
  | json
  | request_id =
      "8b9e4a6d-dfb4-4b95-bada-319e7e5b878a"

One trace

{job="hivenet-router", log_type="audit"}
  | json
  | trace_id =
      "4bf92f3577b34da6a3ce929d0e0e4736"

Slow requests

{job="hivenet-router", log_type="audit"}
  | json
  | latency_ms > 5000

Authentication failures

{
  job="hivenet-router",
  log_type="audit",
  error_code="unauthorized"
}
  | json

Model-access failures

{
  job="hivenet-router",
  log_type="audit",
  error_code="model_forbidden"
}
  | json

Provider fallback requests

{job="hivenet-router", log_type="audit"}
  | json
  | agent_id =~ "provider:.*"

Aggregate audit data

Request rate by status

sum by (status_code) (
  rate(
    {
      job="hivenet-router",
      log_type="audit"
    }[5m]
  )
)

Because status_code is a promoted label in the repository configuration, it remains available to the aggregation.

Error rate

sum(
  rate(
    {
      job="hivenet-router",
      log_type="audit",
      level="error"
    }[5m]
  )
)
/
sum(
  rate(
    {
      job="hivenet-router",
      log_type="audit"
    }[5m]
  )
)

Requests by model

sum by (model) (
  count_over_time(
    {
      job="hivenet-router",
      log_type="audit"
    }[1h]
  )
)

Output tokens by tenant

sum by (tenant_id) (
  sum_over_time(
    {
      job="hivenet-router",
      log_type="audit"
    }
      | json
      | unwrap output_tokens
      | __error__ = ""
    [1d]
  )
)

Average latency by tenant

avg by (tenant_id) (
  avg_over_time(
    {
      job="hivenet-router",
      log_type="audit"
    }
      | json
      | unwrap latency_ms
      | __error__ = ""
    [1h]
  )
)

P95 latency

quantile_over_time(
  0.95,
  {
    job="hivenet-router",
    log_type="audit"
  }
    | json
    | unwrap latency_ms
    | __error__ = ""
  [5m]
)

Use the Grafana audit dashboard

The provisioned dashboard is available at:

http://<router-host>:3000/d/hivenet-router-audit/

It includes:

recent request records
request rate by status
P95 latency
top models
error rate

Filters are available for:

tenant ID
status code
model
error code

See Grafana dashboards for provisioning and troubleshooting.

Correlate logs and traces

A practical investigation flow is:

Obtain the request’s X-Request-ID or traceparent response header.
Search the audit logs for request_id or trace_id.
Inspect the status, tenant, model, agent, latency, and error code.
Open the matching trace in Tempo.
Review router application logs for the same request ID.
Inspect Prometheus metrics for the selected agent and model.

For example:

{job="hivenet-router", log_type="audit"}
  | json
  | request_id =
      "8b9e4a6d-dfb4-4b95-bada-319e7e5b878a"

The audit record provides the trace ID:

{
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736"
}

Use that value in Grafana Explore with the Tempo data source.

Source IP and proxies

The source_ip field comes from Gin’s client-IP resolution. Its accuracy depends on:

whether the router is reached directly
which reverse proxies are trusted
whether forwarding headers are replaced or preserved
whether clients can send spoofed forwarding headers

Do not treat source_ip as a verified user or tenant identity. When Hivenet Router sits behind a proxy:

restrict direct access to the router
configure the proxy to replace untrusted forwarding headers
review Gin’s trusted-proxy behavior for your deployment
test the value recorded in the audit file

Use authenticated tenant and key information for authorization and accountability.

Retention and rotation

Hivenet Router opens the audit file when the process starts and keeps the file descriptor open. A rotation method that renames the file and creates a new one can leave the router writing to the renamed file until restart. For a simple bare-metal setup, use copytruncate:

/etc/logrotate.d/hivenet-router

/var/log/hivenet-router/audit.jsonl {
    daily
    rotate 30
    compress
    delaycompress
    missingok
    notifempty
    copytruncate
}

copytruncate has a small race window in which lines written during the copy and truncate operation can be lost.For stricter audit requirements, use a log-shipping and retention design that avoids local copy-and-truncate rotation, or restart the router as part of a controlled rename-based rotation.

The repository’s Loki configuration uses filesystem storage and does not set an explicit production retention period. For a production deployment, configure and test:

Loki retention
local audit-file retention
backup requirements
deletion workflows
access controls
storage capacity monitoring

Deleting the Docker loki_data volume removes the stored Loki data.

Privacy and governance

Audit logging can support operational investigations and evidence collection, but it does not by itself establish compliance with any law or standard. Before production use, decide:

which teams may access audit data
how long records are retained
whether source IP addresses should be stored
whether tenant and model names reveal sensitive information
how deletion and access requests are handled
whether records must be exported to a separate security system
how clock, integrity, backup, and incident procedures are controlled

Hivenet Router does not log prompt and response bodies in the dedicated audit record, but other components in the request path may have their own logging behavior. Review:

reverse proxies
inference backends
client applications
provider APIs
container logging
observability agents

as part of the complete data-flow assessment.

Troubleshooting

The audit file does not exist

Check the configured path:

echo "$HIVENET_ROUTER_AUDIT_LOG_PATH"

Check the router logs for:

audit: cannot create log dir

or:

audit: cannot open

Confirm that the router process can create the directory and file. For systemd:

sudo -u hivenet-router \
  touch /var/log/hivenet-router/test-write

Remove the test file afterward:

sudo rm \
  /var/log/hivenet-router/test-write

Audit JSON appears in application stdout

The file could not be opened, so Hivenet Router fell back to stdout. Correct the path or permissions, then restart the router.

Promtail does not ingest records

Check that the shared file is visible:

docker compose exec promtail \
  tail -n 5 \
  /var/log/hivenet-router/audit.jsonl

Inspect Promtail logs:

docker compose logs promtail \
  | tail -100

Check its positions file and Loki connection.

Loki has no audit stream

Query the labels API:

docker compose exec grafana \
  wget -qO- \
  http://loki:3100/loki/api/v1/labels

Then query the stream:

docker compose exec grafana \
  wget -qO- \
  'http://loki:3100/loki/api/v1/query?query=%7Bjob%3D%22hivenet-router%22%2Clog_type%3D%22audit%22%7D' \
  | head

Check that Promtail uses:

job="hivenet-router"
log_type="audit"

rather than values from the older example configuration.

A field is empty

The field may not apply to that request. Examples:

auth failure: empty tenant and key
model-list request: empty model
pre-routing rejection: empty agent ID
no tracing: empty trace ID
static key: empty key ID
embedding or reranking: zero token counts

The request ID was replaced

Hivenet Router accepts only valid UUIDs in X-Request-ID. Use a UUID such as:

8b9e4a6d-dfb4-4b95-bada-319e7e5b878a

Other values are replaced with a generated UUID.

The source IP is unexpected

Check the proxy path and forwarding headers. The recorded value may be the proxy address or a header-derived address, depending on how the router is reached and configured.

Token totals are zero

Possible causes include:

the request failed before inference
it was an embedding or reranking request
the backend did not report usage
streaming usage could not be measured
the handler did not have token information for that route

Compare the audit record with the backend response and Prometheus tenant-token metrics.

Records continue in a rotated file

The router keeps the audit file open. Use copytruncate, or restart the router after a rename-based rotation so it opens the new path.

Next steps

Hardware metrics

Review the GPU, CPU, and memory data reported by agents.

Engine metrics

Understand cache, queue, latency, and throughput metrics by backend.

Error codes

Review the structured errors recorded in audit entries.

​What Hivenet Router audits

​Audit record format

​Audit fields

​Request IDs

​Trace IDs

​Tenant and key identity

​Agent and provider identity

​Token fields

​What is not logged

​Error codes

​Configure the file path

​Fallback to stdout

​Bare-metal permissions

​Docker Compose pipeline

​Promtail configuration

​Label-cardinality guidance

​Query audit records

​All audit records

​Errors

​One tenant

​One model

​One request ID

​One trace

​Slow requests

​Authentication failures

​Model-access failures

​Provider fallback requests

​Aggregate audit data

​Request rate by status

​Error rate

​Requests by model

​Output tokens by tenant

​Average latency by tenant

​P95 latency

​Use the Grafana audit dashboard

​Correlate logs and traces

​Source IP and proxies

​Retention and rotation

​Privacy and governance

​Troubleshooting

​The audit file does not exist

​Audit JSON appears in application stdout

​Promtail does not ingest records

​Loki has no audit stream

​A field is empty

​The request ID was replaced

​The source IP is unexpected

​Token totals are zero

​Records continue in a rotated file

​Next steps

Hardware metrics

Engine metrics

Error codes

What Hivenet Router audits

Audit record format

Audit fields

Request IDs

Trace IDs

Tenant and key identity

Agent and provider identity

Token fields

What is not logged

Error codes

Configure the file path

Fallback to stdout

Bare-metal permissions

Docker Compose pipeline

Promtail configuration

Label-cardinality guidance

Query audit records

All audit records

Errors

One tenant

One model

One request ID

One trace

Slow requests

Authentication failures

Model-access failures

Provider fallback requests

Aggregate audit data

Request rate by status

Error rate

Requests by model

Output tokens by tenant

Average latency by tenant

P95 latency

Use the Grafana audit dashboard

Correlate logs and traces

Source IP and proxies

Retention and rotation

Privacy and governance

Troubleshooting

The audit file does not exist

Audit JSON appears in application stdout

Promtail does not ingest records

Loki has no audit stream

A field is empty

The request ID was replaced

The source IP is unexpected

Token totals are zero

Records continue in a rotated file

Next steps