vLLM agent

Connect a vLLM inference server to Hivenet Router by running an agent beside the backend. The agent discovers the model served by vLLM, registers it with the router, forwards requests, and reports engine and hardware metrics that Hivenet Router can use for routing and observability.

Each Hivenet Router agent registers one model. If a backend exposes several models, run a separate agent for each model and use --model to select the model that agent represents.

Before you start

You need:

a running Hivenet Router router
an NVIDIA GPU with CUDA support
network access between the agent host and the router
access to the model you want to serve
either Docker or a Python environment supported by vLLM
the Hivenet Router agent binary or Docker image
the same JWT secret used by the router

The examples use:

Router IP: 192.168.1.100
Router gRPC port: 50051
Router libp2p port: 9000
Agent IP: 192.168.1.101
vLLM port: 8888
Model: meta-llama/Llama-3.1-8B-Instruct

Replace these values with addresses and model names from your deployment.

Start vLLM

Docker
Bare metal

Start vLLM with host networking:

docker run -d \
  --name vllm \
  --restart unless-stopped \
  --gpus all \
  --network host \
  vllm/vllm-openai:latest \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 8888 \
  --tensor-parallel-size 1 \
  --max-num-seqs 256

If the model requires authentication or license acceptance, provide the required credentials through your normal secrets-management process.Follow the startup logs:

docker logs -f vllm

Install vLLM:

pip install vllm

Start the server:

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 8888 \
  --tensor-parallel-size 1 \
  --max-num-seqs 256

Wait for the model to load, then check the backend:

curl -i http://localhost:8888/health

A ready vLLM server returns HTTP status 200. List the models reported by vLLM:

curl http://localhost:8888/v1/models \
  | jq '.data[].id'

The model ID returned here is the value clients must use when sending requests through Hivenet Router.

Hivenet Router forwards each request’s ID to vLLM in the X-Request-ID header, and vLLM uses it as the request ID in its logs, so the router’s and vLLM’s logs show the same ID. The router generates a new ID for every request, and a retry of the same request gets the ID followed by -2, -3 and so on, so vLLM never receives the same ID twice. When a retry’s response comes back, the client sees the router’s request ID in X-Request-ID and the retry’s ID, for example <id>-2, in X-Upstream-Request-ID.

Prepare the Hivenet Router agent

Docker
Bare metal

Pull the prebuilt agent image from the GitHub Container Registry:

docker pull ghcr.io/hivenetoss/hivenet-agent:latest

Use :nightly if you want the moving pre-release tag built from the latest commit on main. Immutable :nightly-<shortsha> tags are also published.To build a custom image from source instead, clone the Hivenet Router repository and run:

docker build \
  -f Dockerfile.agent \
  -t hivenet-router/agent:latest \
  .

Create a directory for the shared secret and persistent agent identity:

sudo mkdir -p /opt/hivenet-router
sudo chown "$USER":"$USER" /opt/hivenet-router

Place the router’s JWT secret at:

/opt/hivenet-router/jwt.secret

Protect it:

chmod 600 /opt/hivenet-router/jwt.secret

Build the agent from the Hivenet Router repository:

go build -o bin/hivenet-agent ./cmd/agent/

Place the shared JWT secret somewhere the agent process can read, for example:

/opt/hivenet-router/jwt.secret

Create a writable directory for the persistent agent identity:

sudo mkdir -p /var/lib/hivenet-router/agent
sudo chown "$USER":"$USER" /var/lib/hivenet-router/agent

Start the agent

Docker
Bare metal

docker run -d \
  --name hivenet-agent \
  --restart unless-stopped \
  --network host \
  --gpus all \
  -v /opt/hivenet-router/jwt.secret:/jwt.secret:ro \
  -v /opt/hivenet-router:/app/data \
  ghcr.io/hivenetoss/hivenet-agent:latest \
  --engine vllm \
  --backend-url http://localhost:8888 \
  --router-grpc 192.168.1.100:50051 \
  --jwt-secret-file /jwt.secret \
  --capacity 256 \
  --region EU-France \
  --organization ml-team \
  --machine gpu-worker-1 \
  --identity-path /app/data/agent-identity.key

The /app/data mount preserves the agent’s peer identity across container replacement and restarts.The agent container receives GPU access so it can collect NVIDIA hardware metrics. Inference still runs in the separate vLLM process or container.

./bin/hivenet-agent \
  --engine vllm \
  --backend-url http://localhost:8888 \
  --router-grpc 192.168.1.100:50051 \
  --jwt-secret-file /opt/hivenet-router/jwt.secret \
  --capacity 256 \
  --region EU-France \
  --organization ml-team \
  --machine gpu-worker-1 \
  --identity-path /var/lib/hivenet-router/agent/identity.key

The agent waits for vLLM to become healthy before registering. If vLLM is still loading, the agent continues polling rather than exiting permanently.

Understand the agent settings

Setting	Purpose
`--engine vllm`	Selects the vLLM backend integration
`--backend-url`	Base URL of the local vLLM server
`--router-grpc`	Router endpoint used for agent authentication
`--jwt-secret-file`	Shared secret used to authenticate the agent
`--capacity`	Maximum concurrent requests the router may assign to the agent
`--region`	Region metadata available to routing policies
`--organization`	Organization or infrastructure-provider metadata
`--machine`	Stable machine identifier
`--identity-path`	Persistent private key that keeps the peer ID stable

You can also add comma-separated routing tags:

--tags production,a100,high-memory

Tags can be matched in routing policies.

Select the model

The agent can discover the model automatically or use an explicit model name.

Automatic discovery

When --model is omitted, the agent requests:

GET <backend-url>/v1/models

It registers the first model in the response. For a vLLM server exposing one model, automatic discovery is usually sufficient.

Explicit model selection

Use --model when you want to pin the agent to a specific model:

--model meta-llama/Llama-3.1-8B-Instruct

The explicit value overrides automatic discovery.

The model name registered by the agent must match the model name clients send to Hivenet Router.

Use a vLLM server that requires an API key

If you start vLLM with --api-key, it rejects every /v1/* request that lacks Authorization: Bearer <key>, including the model discovery request the agent sends before any client traffic exists. Give the agent the same key:

--backend-api-key-file /opt/hivenet-router/vllm.key

export HIVENET_ROUTER_BACKEND_API_KEY="<vllm-api-key>"

When a backend API key is set, the agent sends Authorization: Bearer <key> on every request to the backend: readiness and health checks, model discovery, metrics scrapes, and inference. The client’s router API key never reaches vLLM, with or without a backend key: the agent removes the client’s credentials before every request. When both are provided, --backend-api-key-file wins.

The agent never forwards the client’s router credentials to the backend: Authorization, X-Api-Key, Cookie and Proxy-Authorization are removed first. If the backend requires authentication, set a backend API key.

The setting is not specific to vLLM. It works the same way for every engine and any OpenAI-compatible backend that expects a bearer key. See Backend API key.

Connect multiple models

Each agent registers as an independent peer and represents one model. If one backend exposes multiple models, start one agent process for each model:

./bin/hivenet-agent \
  --engine vllm \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --backend-url http://localhost:8888 \
  --identity-path /var/lib/hivenet-router/agent/llama-8b.key \
  ...

Start the second agent with a different model and identity:

./bin/hivenet-agent \
  --engine vllm \
  --model meta-llama/Llama-3.1-70B-Instruct \
  --backend-url http://localhost:8888 \
  --identity-path /var/lib/hivenet-router/agent/llama-70b.key \
  ...

When several agents run on the same host, each one needs a separate --identity-path so every process keeps a distinct peer identity.

Set agent capacity

Set --capacity to the number of concurrent requests the backend can safely accept. For vLLM, start by matching it to --max-num-seqs:

vllm serve ... \
  --max-num-seqs 256

hivenet-agent ... \
  --capacity 256

You may set the Hivenet Router capacity slightly lower to preserve headroom:

--capacity 224

When an agent reaches its declared capacity, the router stops assigning new requests to it and considers other matching agents or configured fallback steps. Capacity does not change vLLM’s own scheduler limit. It tells Hivenet Router how much work the agent is prepared to accept.

For streaming responses, Hivenet Router holds the agent capacity slot until the stream fully drains, whether the stream ends normally or the client disconnects. --capacity therefore limits concurrent generations routed through Hivenet Router, including ongoing streams. Verify vLLM concurrency under sustained streaming load before raising it.

Verify the connection

On the router, check agent health:

curl http://192.168.1.100:8080/admin/health \
  | jq .

List the models available to clients:

curl http://192.168.1.100:8080/v1/models \
  | jq '.data[].id'

Inspect detailed routing state:

curl http://192.168.1.100:8080/admin/routing-table \
  | jq '.agents[] | {
      peer_id,
      model: .metadata.model,
      engine: .metadata.engine,
      capacity: .metadata.capacity,
      region: .metadata.region,
      healthy: .status.healthy,
      active_requests: .status.active_requests
    }'

Send an inference request:

curl -X POST http://192.168.1.100:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [
      {
        "role": "user",
        "content": "Hello"
      }
    ]
  }'

vLLM metrics

The agent scrapes vLLM’s /metrics endpoint every 500 milliseconds by default. It sends the resulting engine data to the router, which exposes it through the router’s Prometheus endpoint. The agent does not expose a separate Prometheus port.

Signal	vLLM source	Hivenet Router use
KV cache utilization	`vllm:kv_cache_usage_perc`	Routing gates and observability
Running requests	`vllm:num_requests_running`	Routing and load visibility
Waiting requests	`vllm:num_requests_waiting`	Scheduler saturation
Preemptions	`vllm:num_preemptions_total`	KV-cache pressure
Time to first token	`vllm:time_to_first_token_seconds`	Average, P90, and histogram metrics
Inter-token latency	`vllm:inter_token_latency_seconds`	Average, P90, and histogram metrics
Prompt-token distribution	`vllm:request_prompt_tokens`	Workload-shape analysis
Generated-token distribution	`vllm:request_generation_tokens`	Workload-shape analysis
Finish reason	`vllm:request_success_total`	Completion outcome analysis

View all engine metrics on the router:

curl http://192.168.1.100:2112/metrics \
  | grep hivenet_agent_engine

Useful exported metrics include:

hivenet_agent_engine_kv_cache_utilization
hivenet_agent_engine_running_requests
hivenet_agent_engine_waiting_requests
hivenet_agent_engine_preemptions_total
hivenet_agent_engine_avg_ttft_seconds
hivenet_agent_engine_p90_ttft_seconds
hivenet_agent_engine_avg_itl_seconds
hivenet_agent_engine_p90_itl_seconds

View live values through the routing table:

curl http://192.168.1.100:8080/admin/routing-table \
  | jq '.agents[]
      | select(.metadata.engine == "vllm")
      | {
          peer_id,
          model: .metadata.model,
          kv_cache_utilization: .engine.kv_cache_utilization,
          running_requests: .engine.running_requests,
          waiting_requests: .engine.waiting_requests
        }'

Use vLLM metrics in routing policies

You can exclude agents whose KV cache is under pressure:

exclude_if:
  kv_cache_utilization:
    gt: 0.85

Or exclude agents with queued work:

exclude_if:
  waiting_requests:
    gt: 0

When a metric is unavailable for an agent, Hivenet Router skips that gate for the agent rather than excluding it. See Policy YAML reference for the full policy schema.

Troubleshooting

The agent cannot discover a model

Check vLLM health:

curl -i http://localhost:8888/health

Check model discovery:

curl http://localhost:8888/v1/models \
  | jq '.data[].id'

The agent waits and retries if vLLM is healthy but no model is available yet.

The wrong model is registered

Automatic discovery selects the first model returned by /v1/models.Pin the correct one:

--model meta-llama/Llama-3.1-8B-Instruct

Restart the agent after changing the setting.

The agent does not register

Check the agent logs:

Docker
systemd

docker logs hivenet-agent

sudo journalctl -u hivenet-agent -n 100 --no-pager

Confirm connectivity:

nc -zv 192.168.1.100 50051
nc -zv 192.168.1.100 9000

Check that the router and agent use the same JWT secret.

KV cache utilization remains high

Reduce the amount of work Hivenet Router assigns to the agent:

--capacity 128

Or reduce vLLM’s scheduler limit:

--max-num-seqs 128

You can also use a routing gate to stop selecting the agent above a chosen threshold.

Preemptions keep increasing

Rising preemptions indicate that vLLM is evicting in-flight KV-cache data under memory pressure.Reduce:

--capacity on the Hivenet Router agent
--max-num-seqs on vLLM
model context or batch pressure, where appropriate for your workload

Backend requests time out

The agent’s backend HTTP timeout defaults to two minutes.Increase it for longer requests:

--http-timeout 10m

The peer ID changes after restart

Make sure --identity-path points to persistent storage.For Docker, confirm the volume is mounted:

docker inspect hivenet-agent \
  | jq '.[0].Mounts'

For bare metal:

ls -l /var/lib/hivenet-router/agent/identity.key

Metrics are missing

Check vLLM’s metrics endpoint:

curl http://localhost:8888/metrics \
  | head

Check the agent logs for scrape errors.Metrics are scraped in the background. A scrape failure does not stop the agent from forwarding requests.

Next steps

Ollama agent

Connect an Ollama backend for local or edge inference.

Routing concepts

Learn how Hivenet Router filters and ranks matching agents.

Engine metrics

Understand the metrics collected from vLLM and other engines.

Getting started

Deploy Hivenet Router

Use the API

Routing and policies

Security and auth

Observability

Integrations

Reference

Project

Before you start

Start vLLM

Prepare the Hivenet Router agent

Start the agent

Understand the agent settings

Select the model

Automatic discovery

Explicit model selection

Use a vLLM server that requires an API key

Connect multiple models

Set agent capacity

Verify the connection

vLLM metrics

Use vLLM metrics in routing policies

Troubleshooting

Next steps

Ollama agent

Routing concepts

Engine metrics

​Before you start

​Start vLLM

​Prepare the Hivenet Router agent

​Start the agent

​Understand the agent settings

​Select the model

​Automatic discovery

​Explicit model selection

​Use a vLLM server that requires an API key

​Connect multiple models

​Set agent capacity

​Verify the connection

​vLLM metrics

​Use vLLM metrics in routing policies

​Troubleshooting

​Next steps

Ollama agent

Routing concepts

Engine metrics

Before you start

Start vLLM

Prepare the Hivenet Router agent

Start the agent

Understand the agent settings

Select the model

Automatic discovery

Explicit model selection

Use a vLLM server that requires an API key

Connect multiple models

Set agent capacity

Verify the connection

vLLM metrics

Use vLLM metrics in routing policies

Troubleshooting

Next steps