Latency tracking - Hivenet Router

Hivenet Router tracks per-agent request latency using a smoothed round-trip-time estimate. The router records one latency sample whenever it forwards a request to an agent and receives the agent’s HTTP response headers. It then updates two values:

SRTT is the smoothed round-trip time.
RTTVAR describes how much recent samples vary from the smoothed value.

Hivenet Router adapts the update rules from RFC 6298, which defines TCP’s retransmission-time estimator. It uses the RFC’s initialization and variance calculation, but changes the smoothing factor when latency improves.

Hivenet Router uses SRTT as an operational routing signal.It does not calculate or use TCP’s retransmission timeout, and its samples measure application-level request behavior rather than TCP packet round trips.

What Hivenet Router measures

The router starts timing immediately before it sends the HTTP request over libp2p. The sample ends when the router receives the HTTP response headers from the agent.

Router sends request
        │
        │ libp2p request
        ▼
Agent forwards to backend
        │
        │ backend response begins
        ▼
Agent sends response headers
        │
        ▼
Router records latency sample

The resulting value includes:

router-to-agent transport time
work performed by the agent
time spent waiting for the inference backend
backend response timing
agent-to-router transport time

It is measured in milliseconds.

Streaming and non-streaming requests

The meaning of one sample depends on how the response is delivered.

Streaming Chat Completions

For a streaming response, the agent forwards the backend’s response headers as soon as the stream begins. The sample therefore approximates:

router-agent transport
+ backend queueing and prompt setup
+ time until the backend begins its streaming response

This boundary can be close to time to first token, but it is not the same measurement. SRTT ends when response headers reach the router; engine TTFT ends when the backend records its first generated token. It does not include the time needed to generate and deliver the complete response. This response-header boundary is also where the current implementation releases the agent’s declared capacity slot and the router’s forwarding slot for a streaming request. Backend generation can continue after the SRTT sample has been recorded and those slots have been released.

Non-streaming Chat Completions

For the native non-streaming Chat Completions path, the agent buffers the complete backend response before returning headers to the router. The sample therefore approximates:

router-agent transport
+ backend queueing
+ complete inference
+ response processing

Other passthrough endpoints

For passthrough routes, the exact point at which headers become available depends on the backend and endpoint behavior.

Do not compare SRTT values blindly across different request types.A short streaming request, a long non-streaming completion, an embedding batch, and a reranking request measure different amounts of backend work.

Use SRTT to compare agents serving similar workloads and request shapes.

What SRTT does not measure

SRTT does not include:

client-to-router network latency
time spent waiting in Hivenet Router’s per-model capacity queue
time needed to send a complete stream to the client
delays caused by a slow client reading streamed output
application processing before the request reaches Hivenet Router
application processing after the response leaves Hivenet Router

This keeps the value centered on the router-agent-backend path. For complete client-visible latency, use the router HTTP, tenant-duration, audit, and tracing data instead.

Why Hivenet Router does not use end-to-end latency

End-to-end latency would mix agent performance with conditions the agent cannot control. For example:

a distant client would make the selected agent look slower
a request waiting in the router queue would inflate the agent’s value
a client reading an SSE stream slowly would extend the measured duration
clients in different locations would make agent comparisons inconsistent

SRTT measures every agent from the router’s point of view. That makes values more useful for comparing agents inside one Hivenet Router deployment.

First latency sample

For the first valid sample, Hivenet Router initializes the values as:

SRTT   = R
RTTVAR = R / 2

where:

R = the first measured request latency

For a first sample of 200 milliseconds:

SRTT   = 200 ms
RTTVAR = 100 ms

The routing table reports the latency state as:

UNKNOWN

before the first sample and:

KNOWN

after the estimator has been initialized.

Subsequent samples

Hivenet Router updates RTTVAR before SRTT.

RTTVAR =
  0.75 × previous RTTVAR
  + 0.25 × |previous SRTT - new sample|

The SRTT calculation uses an asymmetric smoothing factor.

When latency improves

When the new sample is lower than the current SRTT:

SRTT =
  0.5 × previous SRTT
  + 0.5 × new sample

This gives the new, faster sample a weight of:

α = 0.5

When latency worsens

When the new sample is equal to or higher than the current SRTT:

SRTT =
  0.875 × previous SRTT
  + 0.125 × new sample

This uses:

α = 0.125

The result adapts quickly when an agent becomes faster and more cautiously when one slow sample appears.

Why the smoothing is asymmetric

Language-model backends can produce large one-off latency samples. Examples include:

loading a model after startup
a cold cache
garbage collection
temporary host contention
a large prompt
an unusually long non-streaming response

Using a larger weight when latency improves helps an agent recover quickly after a cold or anomalous request. Using a smaller weight when latency worsens prevents one brief spike from immediately dominating its routing history.

This is a Hivenet Router adaptation.RFC 6298 normally uses α = 1/8 for subsequent SRTT updates in both directions. Hivenet Router uses α = 1/2 only for improving samples.

Example

Assume an agent has:

SRTT   = 200 ms
RTTVAR = 50 ms

It then produces a 5,000-millisecond sample.

After the slow sample

Update RTTVAR first:

RTTVAR =
  0.75 × 50
  + 0.25 × |200 - 5000|

RTTVAR = 1237.5 ms

Because the sample is worse than SRTT:

SRTT =
  0.875 × 200
  + 0.125 × 5000

SRTT = 800 ms

The single spike raises SRTT from 200 to 800 milliseconds rather than immediately replacing it with 5,000 milliseconds.

After a normal 200-millisecond sample

The new sample is better than the current SRTT, so Hivenet Router uses the faster downward update:

SRTT =
  0.5 × 800
  + 0.5 × 200

SRTT = 500 ms

Further 200-millisecond samples produce:

Sample	Updated SRTT
Initial state	`800 ms`
1	`500 ms`
2	`350 ms`
3	`275 ms`
4	`237.5 ms`
5	`218.75 ms`

The estimate moves back toward normal behavior without forgetting the spike immediately.

Successful and failed requests

Successful requests update SRTT and RTTVAR. Failed requests also update the estimator when the router obtained a positive latency sample before the failure. Examples include:

a backend returning an error response
an unsuccessful response received from an agent
a transport failure after the request was sent

Failures that occur before Hivenet Router can obtain a meaningful sample do not update latency. For example:

failure to create the outbound request
failure before a connection to the agent is established
another local error with a zero sample

This lets latency history include slow or failed backend responses without fabricating a measurement for work that never reached an agent.

Live routing values

Routing policies use the current in-memory SRTT. They do not read the periodically flushed BadgerDB record for every request. This matters because the persistent record may be up to one flush interval behind an actively serving agent. The default universal-history flush interval is:

30 seconds

The live in-memory value is updated immediately after a recorded request outcome.

Persistence

Hivenet Router persists SRTT and RTTVAR in BadgerDB under the agent’s peer ID. The router flushes universal agent history:

every 30 seconds by default
when an agent disconnects
during graceful router shutdown

Change the periodic interval:

./bin/hivenet-router \
  --universal-flush-interval 1m \
  ...

The persistent record uses the router’s normal disk retention period, which defaults to:

30 days

Warm start

When an agent reconnects with the same peer ID, Hivenet Router restores its previous SRTT and RTTVAR. For example:

Persisted SRTT: 195 ms
Router restarts
Agent reconnects with the same peer ID
First new sample: 190 ms
Updated SRTT: 192.5 ms

Without the persisted baseline, the first request would initialize a new estimator.

Warm start depends on a stable agent peer ID.Configure --identity-path and preserve the identity file or volume across agent restarts. An agent with a new peer ID begins with no previous latency history.

Reset latency history

The administration metrics-reset endpoint clears SRTT and RTTVAR together with the other persisted per-agent lifetime metrics.

POST /admin/metrics/reset

curl -X POST \
  -H "Authorization: Bearer <admin-api-key>" \
  http://localhost:8080/admin/metrics/reset

After the reset, the next valid request sample initializes the estimator again.

This operation also clears other per-agent historical request, token, failure, and disconnection counters.It is not a latency-only reset.

View current values

Use the routing table:

curl \
  -H "Authorization: Bearer <admin-api-key>" \
  http://localhost:8080/admin/routing-table \
  | jq '.agents[] | {
      peer_id,
      model: .metadata.model,
      engine: .metadata.engine,
      latency_state: .universal.latency_state,
      srtt_ms: .universal.srtt_ms,
      rttvar_ms: .universal.rttvar_ms
    }'

A response may resemble:

{
  "peer_id": "12D3KooW...",
  "model": "meta-llama/Llama-3.1-8B-Instruct",
  "engine": "vllm",
  "latency_state": "KNOWN",
  "srtt_ms": 214.5,
  "rttvar_ms": 38.2
}

Before the agent has a sample:

{
  "latency_state": "UNKNOWN",
  "srtt_ms": null,
  "rttvar_ms": null
}

Prometheus metrics

Hivenet Router exports:

hivenet_router_agent_srtt_ms
hivenet_router_agent_rttvar_ms

Labels:

peer_id
model
engine
organization
machine

SRTT by agent

hivenet_router_agent_srtt_ms

Average SRTT by model

avg by (model) (
  hivenet_router_agent_srtt_ms
)

SRTT by engine

avg by (engine) (
  hivenet_router_agent_srtt_ms
)

Agents above 500 milliseconds

hivenet_router_agent_srtt_ms > 500

High variation

hivenet_router_agent_rttvar_ms > 300

SRTT compared with variation

hivenet_router_agent_srtt_ms
+
4
*
hivenet_router_agent_rttvar_ms

This final query can be useful for visualization, but Hivenet Router does not use it as a TCP-style retransmission timeout.

Interpret RTTVAR

RTTVAR represents the smoothed absolute difference between recent samples and SRTT. A low value suggests relatively consistent observed latency. A high value suggests greater variation, which may come from:

variable prompt or generation lengths
queueing
cache misses
mixed streaming and non-streaming traffic
backend contention
network instability
cold starts
different request types sharing one agent

High RTTVAR does not identify the cause of variation.Correlate it with request shape, engine queues, TTFT, ITL, hardware pressure, logs, and traces.

Use SRTT in a routing policy

The policy field is:

srtt

The threshold uses milliseconds.

routing_policy:
  exclude_if:
    srtt:
      gt: 500

  strategy: least-loaded

This excludes an agent whose current SRTT is above 500 milliseconds.

Use srtt, not srtt_ms, in policy YAML.srtt_ms is the JSON and Prometheus naming convention. The policy field is srtt.

Missing latency history passes

A new agent with no request samples has no SRTT value. When a policy contains:

exclude_if:
  srtt:
    gt: 500

the gate is skipped for that new agent until latency history exists. The agent can therefore receive traffic before it has an established SRTT. Use tags or a staged policy when new agents need a warm-up period before production traffic.

SRTT is not a ranking strategy

The only currently implemented ranking strategy is:

least-loaded

This is not supported:

strategy: lowest-srtt

A policy using it is rejected during validation. SRTT can currently be used as an exclusion gate and observability signal.

Choose a threshold

A useful SRTT threshold depends on:

streaming or non-streaming behavior
model size
prompt length
expected output length
backend engine
hardware
agent location
service objective

A 500-millisecond threshold may be reasonable for one streaming workload and impossible for a long non-streaming completion. Use observed distributions before setting a gate. A practical process is:

collect several days of representative traffic
group SRTT by model and engine
identify normal and degraded ranges
compare SRTT with RTTVAR and backend metrics
set a conservative threshold
create a relaxed fallback step
monitor fallback and policy-exhaustion rates

Example:

routing_policy:
  match:
    engine: vllm

  exclude_if:
    srtt:
      gt: 500

  strategy: least-loaded

fallback_chain:
  - name: relaxed-latency

    match:
      engine: vllm

    exclude_if:
      srtt:
        gt: 1500

    strategy: least-loaded

Alerting examples

The thresholds below are examples. Establish values from your own workload.

High SRTT

groups:
  - name: hivenet-router-latency

    rules:
      - alert: HivenetRouterHighSRTT

        expr: |
          hivenet_router_agent_srtt_ms > 500

        for: 5m

        labels:
          severity: warning

        annotations:
          summary: >-
            High smoothed request latency on {{ $labels.peer_id }}

High RTTVAR

      - alert: HivenetRouterUnstableLatency

        expr: |
          hivenet_router_agent_rttvar_ms > 300

        for: 10m

        labels:
          severity: warning

        annotations:
          summary: >-
            Unstable request latency on {{ $labels.peer_id }}

Model-level degradation

      - alert: HivenetRouterModelLatencyDegradation

        expr: |
          avg by (model) (
            hivenet_router_agent_srtt_ms
          ) > 750

        for: 5m

        labels:
          severity: warning

        annotations:
          summary: >-
            High average SRTT for {{ $labels.model }}

Debug latency changes

Enable debug logs:

export GOLOG_LOG_LEVEL="router=debug,metrics=debug"

Start the router, then search for latency-related records:

systemd
Docker Compose

sudo journalctl \
  -fu hivenet-router \
  | grep -i "srtt\|rtt"

docker compose logs -f router \
  | grep -i "srtt\|rtt"

Use tracing when you need to break the request into router, agent, backend, and provider spans. Use audit or tenant-duration metrics when you need client-visible request duration rather than router-agent SRTT.

Troubleshooting

SRTT remains unknown

The agent has not produced a valid recorded request sample. Check that:

the agent is registered
it has served a request
the request reached the agent
the router recorded a success or a failure with a positive sample

Inspect:

curl \
  -H "Authorization: Bearer <admin-api-key>" \
  http://localhost:8080/admin/routing-table \
  | jq '.agents[] | {
      model: .metadata.model,
      latency_state: .universal.latency_state,
      successful: .universal.successful_requests_total,
      failed: .universal.failed_requests_total
    }'

SRTT is much higher than TTFT

The requests may be non-streaming. For native non-streaming Chat Completions, SRTT includes the complete inference before the agent returns headers. Engine TTFT measures only the start of model output. Also check:

router-agent network time
backend queueing
response parsing
mixed request types
different observation windows

SRTT changes after switching to streaming

This is expected. Streaming usually records latency near response start, while non-streaming native chat records latency after the complete backend response is ready. Establish separate baselines when both modes are used heavily.

SRTT remains high after a cold start

The estimator moves downward quickly, but it still needs successful faster samples. Send representative requests and check whether:

the backend is fully loaded
queue pressure has cleared
new samples are actually lower
the agent is still serving unusually large requests

SRTT resets after an agent restart

The agent may have started with a new peer ID. Check that --identity-path points to persistent storage and compare the current peer ID with the previous deployment. Also confirm that:

the router uses a persistent BadgerDB directory
universal history was not reset
the previous record has not expired
the router shut down or flushed recently enough to preserve the latest value

Policy does not exclude a slow agent

Check that the field is:

srtt:

rather than:

srtt_ms:

Inspect the live routing-table value. A new agent with no SRTT passes the gate. Also confirm that the agent survived the earlier model, health, capability, match, failure, and capacity gates.

Prometheus shows zero before traffic

The SRTT gauge may be seeded at zero when the agent first registers without latency history. Use the routing table’s:

latency_state

when you need to distinguish an initialized low value from an unknown estimator.

Next steps

Hardware-aware routing

Combine latency with GPU, CPU, memory, cache, and queue pressure.

Routing concepts

See where the SRTT gate fits inside the complete routing pipeline.

Performance characteristics

Review performance measurements, scope, and testing considerations.

​What Hivenet Router measures

​Streaming and non-streaming requests

​Streaming Chat Completions

​Non-streaming Chat Completions

​Other passthrough endpoints

​What SRTT does not measure

​Why Hivenet Router does not use end-to-end latency

​First latency sample

​Subsequent samples

​When latency improves

​When latency worsens

​Why the smoothing is asymmetric

​Example

​After the slow sample

​After a normal 200-millisecond sample

​Successful and failed requests

​Live routing values

​Persistence

​Warm start

​Reset latency history

​View current values

​Prometheus metrics

​SRTT by agent

​Average SRTT by model

​SRTT by engine

​Agents above 500 milliseconds

​High variation

​SRTT compared with variation

​Interpret RTTVAR

​Use SRTT in a routing policy

​Missing latency history passes

​SRTT is not a ranking strategy

​Choose a threshold

​Alerting examples

​High SRTT

​High RTTVAR

​Model-level degradation

​Debug latency changes

​Troubleshooting

​SRTT remains unknown

​SRTT is much higher than TTFT

​SRTT changes after switching to streaming

​SRTT remains high after a cold start

​SRTT resets after an agent restart

​Policy does not exclude a slow agent

​Prometheus shows zero before traffic

​Next steps

Hardware-aware routing

Routing concepts

Performance characteristics

What Hivenet Router measures

Streaming and non-streaming requests

Streaming Chat Completions

Non-streaming Chat Completions

Other passthrough endpoints

What SRTT does not measure

Why Hivenet Router does not use end-to-end latency

First latency sample

Subsequent samples

When latency improves

When latency worsens

Why the smoothing is asymmetric

Example

After the slow sample

After a normal 200-millisecond sample

Successful and failed requests

Live routing values

Persistence

Warm start

Reset latency history

View current values

Prometheus metrics

SRTT by agent

Average SRTT by model

SRTT by engine

Agents above 500 milliseconds

High variation

SRTT compared with variation

Interpret RTTVAR

Use SRTT in a routing policy

Missing latency history passes

SRTT is not a ranking strategy

Choose a threshold

Alerting examples

High SRTT

High RTTVAR

Model-level degradation

Debug latency changes

Troubleshooting

SRTT remains unknown

SRTT is much higher than TTFT

SRTT changes after switching to streaming

SRTT remains high after a cold start

SRTT resets after an agent restart

Policy does not exclude a slow agent

Prometheus shows zero before traffic

Next steps