> ## Documentation Index
> Fetch the complete documentation index at: https://routerdocs.hivenet.com/llms.txt
> Use this file to discover all available pages before exploring further.
# vLLM agent
> Connect a vLLM backend to Hivenet Router, configure model discovery and capacity, and expose vLLM metrics for routing and observability.
Connect a vLLM inference server to Hivenet Router by running an agent beside the backend.
The agent discovers the model served by vLLM, registers it with the router, forwards requests, and reports engine and hardware metrics that Hivenet Router can use for routing and observability.
Each Hivenet Router agent registers one model. If a backend exposes several models, run a separate agent for each model and use `--model` to select the model that agent represents.
## Before you start
You need:
* a running Hivenet Router router
* an NVIDIA GPU with CUDA support
* network access between the agent host and the router
* access to the model you want to serve
* either Docker or a Python environment supported by vLLM
* the Hivenet Router agent binary or Docker image
* the same JWT secret used by the router
The examples use:
```text theme={null}
Router IP: 192.168.1.100
Router gRPC port: 50051
Router libp2p port: 9000
Agent IP: 192.168.1.101
vLLM port: 8888
Model: meta-llama/Llama-3.1-8B-Instruct
```
Replace these values with addresses and model names from your deployment.
## Start vLLM
Start vLLM with host networking:
```bash theme={null}
docker run -d \
--name vllm \
--restart unless-stopped \
--gpus all \
--network host \
vllm/vllm-openai:latest \
--model meta-llama/Llama-3.1-8B-Instruct \
--host 0.0.0.0 \
--port 8888 \
--tensor-parallel-size 1 \
--max-num-seqs 256
```
If the model requires authentication or license acceptance, provide the required credentials through your normal secrets-management process.
Follow the startup logs:
```bash theme={null}
docker logs -f vllm
```
Install vLLM:
```bash theme={null}
pip install vllm
```
Start the server:
```bash theme={null}
vllm serve meta-llama/Llama-3.1-8B-Instruct \
--host 0.0.0.0 \
--port 8888 \
--tensor-parallel-size 1 \
--max-num-seqs 256
```
Wait for the model to load, then check the backend:
```bash theme={null}
curl -i http://localhost:8888/health
```
A ready vLLM server returns HTTP status `200`.
List the models reported by vLLM:
```bash theme={null}
curl http://localhost:8888/v1/models \
| jq '.data[].id'
```
The model ID returned here is the value clients must use when sending requests through Hivenet Router.
## Prepare the Hivenet Router agent
From the Hivenet Router repository, build the agent image if you have not already done so:
```bash theme={null}
docker build \
-f Dockerfile.agent \
-t hivenet-router/agent:latest \
.
```
Create a directory for the shared secret and persistent agent identity:
```bash theme={null}
sudo mkdir -p /opt/hivenet-router
sudo chown "$USER":"$USER" /opt/hivenet-router
```
Place the router’s JWT secret at:
```text theme={null}
/opt/hivenet-router/jwt.secret
```
Protect it:
```bash theme={null}
chmod 600 /opt/hivenet-router/jwt.secret
```
Build the agent from the Hivenet Router repository:
```bash theme={null}
go build -o bin/hivenet-agent ./cmd/agent/
```
Place the shared JWT secret somewhere the agent process can read, for example:
```text theme={null}
/opt/hivenet-router/jwt.secret
```
Create a writable directory for the persistent agent identity:
```bash theme={null}
sudo mkdir -p /var/lib/hivenet-router/agent
sudo chown "$USER":"$USER" /var/lib/hivenet-router/agent
```
## Start the agent
```bash theme={null}
docker run -d \
--name hivenet-agent \
--restart unless-stopped \
--network host \
--gpus all \
-v /opt/hivenet-router/jwt.secret:/jwt.secret:ro \
-v /opt/hivenet-router:/app/data \
hivenet-router/agent:latest \
--engine vllm \
--backend-url http://localhost:8888 \
--router-grpc 192.168.1.100:50051 \
--jwt-secret-file /jwt.secret \
--capacity 256 \
--region EU-France \
--organization ml-team \
--machine gpu-worker-1 \
--identity-path /app/data/agent-identity.key
```
The `/app/data` mount preserves the agent’s peer identity across container replacement and restarts.
The agent container receives GPU access so it can collect NVIDIA hardware metrics. Inference still runs in the separate vLLM process or container.
```bash theme={null}
./bin/hivenet-agent \
--engine vllm \
--backend-url http://localhost:8888 \
--router-grpc 192.168.1.100:50051 \
--jwt-secret-file /opt/hivenet-router/jwt.secret \
--capacity 256 \
--region EU-France \
--organization ml-team \
--machine gpu-worker-1 \
--identity-path /var/lib/hivenet-router/agent/identity.key
```
The agent waits for vLLM to become healthy before registering. If vLLM is still loading, the agent continues polling rather than exiting permanently.
## Understand the agent settings
| Setting | Purpose |
| ------------------- | -------------------------------------------------------------- |
| `--engine vllm` | Selects the vLLM backend integration |
| `--backend-url` | Base URL of the local vLLM server |
| `--router-grpc` | Router endpoint used for agent authentication |
| `--jwt-secret-file` | Shared secret used to authenticate the agent |
| `--capacity` | Maximum concurrent requests the router may assign to the agent |
| `--region` | Region metadata available to routing policies |
| `--organization` | Organization or infrastructure-provider metadata |
| `--machine` | Stable machine identifier |
| `--identity-path` | Persistent private key that keeps the peer ID stable |
You can also add comma-separated routing tags:
```bash theme={null}
--tags production,a100,high-memory
```
Tags can be matched in routing policies.
## Select the model
The agent can discover the model automatically or use an explicit model name.
### Automatic discovery
When `--model` is omitted, the agent requests:
```text theme={null}
GET /v1/models
```
It registers the first model in the response.
For a vLLM server exposing one model, automatic discovery is usually sufficient.
### Explicit model selection
Use `--model` when you want to pin the agent to a specific model:
```bash theme={null}
--model meta-llama/Llama-3.1-8B-Instruct
```
The explicit value overrides automatic discovery.
The model name registered by the agent must match the model name clients send to Hivenet Router.
## Connect multiple models
Each agent registers as an independent peer and represents one model.
If one backend exposes multiple models, start one agent process for each model:
```bash theme={null}
./bin/hivenet-agent \
--engine vllm \
--model meta-llama/Llama-3.1-8B-Instruct \
--backend-url http://localhost:8888 \
--identity-path /var/lib/hivenet-router/agent/llama-8b.key \
...
```
Start the second agent with a different model and identity:
```bash theme={null}
./bin/hivenet-agent \
--engine vllm \
--model meta-llama/Llama-3.1-70B-Instruct \
--backend-url http://localhost:8888 \
--identity-path /var/lib/hivenet-router/agent/llama-70b.key \
...
```
When several agents run on the same host, each one needs a separate `--identity-path` so every process keeps a distinct peer identity.
## Set agent capacity
Set `--capacity` to the number of concurrent requests the backend can safely accept.
For vLLM, start by matching it to `--max-num-seqs`:
```bash theme={null}
vllm serve ... \
--max-num-seqs 256
```
```bash theme={null}
hivenet-agent ... \
--capacity 256
```
You may set the Hivenet Router capacity slightly lower to preserve headroom:
```bash theme={null}
--capacity 224
```
When an agent reaches its declared capacity, the router stops assigning new requests to it and considers other matching agents or configured fallback steps.
Capacity does not change vLLM’s own scheduler limit. It tells Hivenet Router how much work the agent is prepared to accept.
For streaming responses, Hivenet Router releases the agent capacity slot when response headers arrive, while backend generation can continue. Treat `--capacity` as a routing-admission setting rather than a hard limit on ongoing streams, and verify vLLM concurrency under sustained streaming load.
## Verify the connection
On the router, check agent health:
```bash theme={null}
curl http://192.168.1.100:8080/admin/health \
| jq .
```
List the models available to clients:
```bash theme={null}
curl http://192.168.1.100:8080/v1/models \
| jq '.data[].id'
```
Inspect detailed routing state:
```bash theme={null}
curl http://192.168.1.100:8080/admin/routing-table \
| jq '.agents[] | {
peer_id,
model: .metadata.model,
engine: .metadata.engine,
capacity: .metadata.capacity,
region: .metadata.region,
healthy: .status.healthy,
active_requests: .status.active_requests
}'
```
Send an inference request:
```bash theme={null}
curl -X POST http://192.168.1.100:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "meta-llama/Llama-3.1-8B-Instruct",
"messages": [
{
"role": "user",
"content": "Hello"
}
]
}'
```
## vLLM metrics
The agent scrapes vLLM’s `/metrics` endpoint every 500 milliseconds by default.
It sends the resulting engine data to the router, which exposes it through the router’s Prometheus endpoint. The agent does not expose a separate Prometheus port.
| Signal | vLLM source | Hivenet Router use |
| ---------------------------- | ---------------------------------- | ----------------------------------- |
| KV cache utilization | `vllm:kv_cache_usage_perc` | Routing gates and observability |
| Running requests | `vllm:num_requests_running` | Routing and load visibility |
| Waiting requests | `vllm:num_requests_waiting` | Scheduler saturation |
| Preemptions | `vllm:num_preemptions_total` | KV-cache pressure |
| Time to first token | `vllm:time_to_first_token_seconds` | Average, P90, and histogram metrics |
| Inter-token latency | `vllm:inter_token_latency_seconds` | Average, P90, and histogram metrics |
| Prompt-token distribution | `vllm:request_prompt_tokens` | Workload-shape analysis |
| Generated-token distribution | `vllm:request_generation_tokens` | Workload-shape analysis |
| Finish reason | `vllm:request_success_total` | Completion outcome analysis |
View all engine metrics on the router:
```bash theme={null}
curl http://192.168.1.100:2112/metrics \
| grep hivenet_router_agent_engine
```
Useful exported metrics include:
```text theme={null}
hivenet_router_agent_engine_kv_cache_utilization
hivenet_router_agent_engine_running_requests
hivenet_router_agent_engine_waiting_requests
hivenet_router_agent_engine_preemptions_total
hivenet_router_agent_engine_avg_ttft_seconds
hivenet_router_agent_engine_p90_ttft_seconds
hivenet_router_agent_engine_avg_itl_seconds
hivenet_router_agent_engine_p90_itl_seconds
```
View live values through the routing table:
```bash theme={null}
curl http://192.168.1.100:8080/admin/routing-table \
| jq '.agents[]
| select(.metadata.engine == "vllm")
| {
peer_id,
model: .metadata.model,
kv_cache_utilization: .engine.kv_cache_utilization,
running_requests: .engine.running_requests,
waiting_requests: .engine.waiting_requests
}'
```
## Use vLLM metrics in routing policies
You can exclude agents whose KV cache is under pressure:
```yaml theme={null}
exclude_if:
kv_cache_utilization:
gt: 0.85
```
Or exclude agents with queued work:
```yaml theme={null}
exclude_if:
waiting_requests:
gt: 0
```
When a metric is unavailable for an agent, Hivenet Router skips that gate for the agent rather than excluding it.
See [Policy YAML reference](/routing/policy-yaml-reference) for the full policy schema.
## Troubleshooting
### The agent cannot discover a model
Check vLLM health:
```bash theme={null}
curl -i http://localhost:8888/health
```
Check model discovery:
```bash theme={null}
curl http://localhost:8888/v1/models \
| jq '.data[].id'
```
The agent waits and retries if vLLM is healthy but no model is available yet.
### The wrong model is registered
Automatic discovery selects the first model returned by `/v1/models`.
Pin the correct one:
```bash theme={null}
--model meta-llama/Llama-3.1-8B-Instruct
```
Restart the agent after changing the setting.
### The agent does not register
Check the agent logs:
```bash theme={null}
docker logs hivenet-agent
```
```bash theme={null}
sudo journalctl -u hivenet-agent -n 100 --no-pager
```
Confirm connectivity:
```bash theme={null}
nc -zv 192.168.1.100 50051
nc -zv 192.168.1.100 9000
```
Check that the router and agent use the same JWT secret.
### KV cache utilization remains high
Reduce the amount of work Hivenet Router assigns to the agent:
```bash theme={null}
--capacity 128
```
Or reduce vLLM’s scheduler limit:
```bash theme={null}
--max-num-seqs 128
```
You can also use a routing gate to stop selecting the agent above a chosen threshold.
### Preemptions keep increasing
Rising preemptions indicate that vLLM is evicting in-flight KV-cache data under memory pressure.
Reduce:
* `--capacity` on the Hivenet Router agent
* `--max-num-seqs` on vLLM
* model context or batch pressure, where appropriate for your workload
### Backend requests time out
The agent’s backend HTTP timeout defaults to two minutes.
Increase it for longer requests:
```bash theme={null}
--http-timeout 10m
```
### The peer ID changes after restart
Make sure `--identity-path` points to persistent storage.
For Docker, confirm the volume is mounted:
```bash theme={null}
docker inspect hivenet-agent \
| jq '.[0].Mounts'
```
For bare metal:
```bash theme={null}
ls -l /var/lib/hivenet-router/agent/identity.key
```
### Metrics are missing
Check vLLM’s metrics endpoint:
```bash theme={null}
curl http://localhost:8888/metrics \
| head
```
Check the agent logs for scrape errors.
Metrics are scraped in the background. A scrape failure does not stop the agent from forwarding requests.
## Next steps
Connect an Ollama backend for local or edge inference.
Learn how Hivenet Router filters and ranks matching agents.
Understand the metrics collected from vLLM and other engines.