> ## Documentation Index
> Fetch the complete documentation index at: https://routerdocs.hivenet.com/llms.txt
> Use this file to discover all available pages before exploring further.
# SGLang agent
> Connect an SGLang backend to Hivenet Router and expose cache, queue, and time-to-first-token metrics for routing.
Connect an SGLang inference server to Hivenet Router by running an agent beside the backend.
The agent discovers the model exposed by SGLang, registers it with the router, forwards OpenAI-compatible chat-completion requests, and reports engine and hardware metrics.
Each Hivenet Router agent registers one model. If an SGLang server exposes several models, run a separate agent for each model and use `--model` to select the model that agent represents.
## Before you start
You need:
* a running Hivenet Router router
* an NVIDIA GPU with CUDA support
* SGLang installed or available as a Docker image
* network access between the agent host and router
* the Hivenet Router agent binary or Docker image
* the same JWT secret used by the router
* access to the model you want to serve
The examples use:
```text theme={null}
Router IP: 192.168.1.100
Router gRPC port: 50051
Router libp2p port: 9000
Agent IP: 192.168.1.101
SGLang port: 3000
Model: meta-llama/Llama-3.1-8B-Instruct
```
Replace these values with addresses and model names from your deployment.
## Start SGLang
Start SGLang with host networking:
```bash theme={null}
docker run -d \
--name sglang \
--restart unless-stopped \
--gpus all \
--network host \
sglangai/sglang:latest \
python3 -m sglang.launch_server \
--model-path meta-llama/Llama-3.1-8B-Instruct \
--host 0.0.0.0 \
--port 3000 \
--enable-metrics
```
Follow the startup logs:
```bash theme={null}
docker logs -f sglang
```
Install the system dependency required by `sgl_kernel`:
```bash theme={null}
sudo apt-get update
sudo apt-get install -y libnuma-dev
```
Install SGLang:
```bash theme={null}
pip install "sglang[all]"
```
Start the server:
```bash theme={null}
python3 -m sglang.launch_server \
--model-path meta-llama/Llama-3.1-8B-Instruct \
--host 0.0.0.0 \
--port 3000 \
--enable-metrics
```
Start SGLang with `--enable-metrics`. Without it, Hivenet Router can still forward requests, but SGLang’s cache, queue, and time-to-first-token metrics will not be available.
For a multi-GPU deployment, set the tensor-parallel size:
```bash theme={null}
python3 -m sglang.launch_server \
--model-path meta-llama/Llama-3.1-8B-Instruct \
--host 0.0.0.0 \
--port 3000 \
--tp 2 \
--enable-metrics
```
Wait for the model to load, then check the health endpoint:
```bash theme={null}
curl -i http://localhost:3000/health
```
A ready SGLang server returns HTTP status `200`.
List the models reported by SGLang:
```bash theme={null}
curl http://localhost:3000/v1/models \
| jq '.data[].id'
```
The model ID returned here is the value clients must use when sending requests through Hivenet Router.
## Prepare the Hivenet Router agent
From the Hivenet Router repository, build the agent image if you have not already done so:
```bash theme={null}
docker build \
-f Dockerfile.agent \
-t hivenet-router/agent:latest \
.
```
Create a directory for the shared secret and persistent agent identity:
```bash theme={null}
sudo mkdir -p /opt/hivenet-router
sudo chown "$USER":"$USER" /opt/hivenet-router
```
Place the router’s JWT secret at:
```text theme={null}
/opt/hivenet-router/jwt.secret
```
Protect it:
```bash theme={null}
chmod 600 /opt/hivenet-router/jwt.secret
```
Build the agent from the Hivenet Router repository:
```bash theme={null}
go build -o bin/hivenet-agent ./cmd/agent/
```
Place the shared JWT secret somewhere the agent process can read, for example:
```text theme={null}
/opt/hivenet-router/jwt.secret
```
Create a writable directory for the persistent agent identity:
```bash theme={null}
sudo mkdir -p /var/lib/hivenet-router/agent
sudo chown "$USER":"$USER" /var/lib/hivenet-router/agent
```
## Start the agent
```bash theme={null}
docker run -d \
--name hivenet-agent \
--restart unless-stopped \
--network host \
--gpus all \
-v /opt/hivenet-router/jwt.secret:/jwt.secret:ro \
-v /opt/hivenet-router:/data \
hivenet-router/agent:latest \
--engine sglang \
--backend-url http://localhost:3000 \
--router-grpc 192.168.1.100:50051 \
--jwt-secret-file /jwt.secret \
--capacity 20 \
--region EU-France \
--organization ml-team \
--machine gpu-worker-1 \
--identity-path /data/agent-identity.key
```
The `/data` mount preserves the agent’s peer identity across container replacement and restarts.
The agent container receives GPU access so it can collect NVIDIA hardware metrics. SGLang continues to run as a separate process or container.
```bash theme={null}
./bin/hivenet-agent \
--engine sglang \
--backend-url http://localhost:3000 \
--router-grpc 192.168.1.100:50051 \
--jwt-secret-file /opt/hivenet-router/jwt.secret \
--capacity 20 \
--region EU-France \
--organization ml-team \
--machine gpu-worker-1 \
--identity-path /var/lib/hivenet-router/agent/identity.key
```
The agent waits for SGLang to become healthy and expose a model before registering with the router. If SGLang is still loading, the agent continues polling rather than exiting permanently.
## Understand the agent settings
| Setting | Purpose |
| ------------------- | ----------------------------------------------------- |
| `--engine sglang` | Selects the SGLang backend integration |
| `--backend-url` | Base URL of the local SGLang server |
| `--router-grpc` | Router endpoint used for agent authentication |
| `--jwt-secret-file` | Shared secret used to authenticate the agent |
| `--capacity` | Maximum concurrent requests Hivenet Router may assign |
| `--region` | Region metadata available to routing policies |
| `--organization` | Organization or infrastructure-provider metadata |
| `--machine` | Stable machine identifier |
| `--identity-path` | Persistent private key that keeps the peer ID stable |
You can also add comma-separated routing tags:
```bash theme={null}
--tags production,sglang,high-throughput
```
Policies can match these values when selecting agents.
## Select the model
The agent can discover the model automatically or use an explicit model name.
### Automatic discovery
When `--model` is omitted, the agent requests:
```text theme={null}
GET /v1/models
```
It registers the first model returned by SGLang.
For an SGLang server exposing one model, automatic discovery is usually sufficient.
### Explicit model selection
Use `--model` to pin the agent to a particular model:
```bash theme={null}
--model meta-llama/Llama-3.1-8B-Instruct
```
The explicit value overrides automatic discovery.
The model registered by the agent must match the model name clients send to Hivenet Router.
## Set agent capacity
`--capacity` controls how many concurrent requests Hivenet Router may assign to the agent.
It does not change SGLang’s own scheduler or memory configuration.
Start with a conservative value based on:
* available VRAM
* model size
* tensor-parallel configuration
* expected context length
* request concurrency
* acceptable latency
For example:
```bash theme={null}
--capacity 20
```
When the agent reaches its declared capacity, Hivenet Router stops assigning new requests to it and considers other matching agents or configured fallback steps.
Test the backend under representative load before increasing capacity.
For streaming responses, Hivenet Router releases the agent capacity slot when response headers arrive, while backend generation can continue. Treat `--capacity` as a routing-admission setting rather than a hard limit on ongoing streams, and verify SGLang concurrency under sustained streaming load.
## Verify the connection
On the router, check agent health:
```bash theme={null}
curl http://192.168.1.100:8080/admin/health \
| jq .
```
List the models available to clients:
```bash theme={null}
curl http://192.168.1.100:8080/v1/models \
| jq '.data[].id'
```
Inspect the SGLang agent:
```bash theme={null}
curl http://192.168.1.100:8080/admin/routing-table \
| jq '.agents[] | select(.metadata.engine == "sglang")'
```
Send a chat-completion request:
```bash theme={null}
curl -X POST http://192.168.1.100:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "meta-llama/Llama-3.1-8B-Instruct",
"messages": [
{
"role": "user",
"content": "Explain what an inference router does in one sentence."
}
]
}'
```
Applications call Hivenet Router’s OpenAI-compatible endpoint. The agent forwards the request to SGLang’s OpenAI-compatible API.
## SGLang metrics
The agent scrapes SGLang’s `/metrics` endpoint every 500 milliseconds by default.
It sends the resulting engine data to the router, which exposes it through the router’s Prometheus endpoint. The agent does not expose a separate Prometheus port.
| Signal | SGLang source | Hivenet Router field |
| --------------------------- | ------------------------------------ | ---------------------- |
| Token-cache utilization | `sglang:token_usage` | `kv_cache_utilization` |
| Running requests | `sglang:num_running_reqs` | `running_requests` |
| Queued requests | `sglang:num_queue_reqs` | `waiting_requests` |
| Average time to first token | `sglang:time_to_first_token_seconds` | Average TTFT |
| P90 time to first token | `sglang:time_to_first_token_seconds` | P90 TTFT |
SGLang does not expose an inter-token-latency histogram through this integration. Hivenet Router therefore does not populate average or P90 ITL values for SGLang agents.
SGLang also has no preemption metric equivalent to the one collected from vLLM.
View engine metrics on the router:
```bash theme={null}
curl http://192.168.1.100:2112/metrics \
| grep hivenet_router_agent_engine
```
Useful exported metrics include:
```text theme={null}
hivenet_router_agent_engine_kv_cache_utilization
hivenet_router_agent_engine_running_requests
hivenet_router_agent_engine_waiting_requests
hivenet_router_agent_engine_avg_ttft_seconds
hivenet_router_agent_engine_p90_ttft_seconds
```
View current values through the routing table:
```bash theme={null}
curl http://192.168.1.100:8080/admin/routing-table \
| jq '.agents[]
| select(.metadata.engine == "sglang")
| {
peer_id,
model: .metadata.model,
kv_cache_utilization: .engine.kv_cache_utilization,
running_requests: .engine.running_requests,
waiting_requests: .engine.waiting_requests
}'
```
## Use SGLang metrics in routing policies
Exclude agents whose token cache is under pressure:
```yaml theme={null}
exclude_if:
kv_cache_utilization:
gt: 0.85
```
Exclude agents with queued requests:
```yaml theme={null}
exclude_if:
waiting_requests:
gt: 0
```
Exclude agents whose time to first token is too high:
```yaml theme={null}
exclude_if:
avg_ttft_seconds:
gt: 2
```
When a metric is unavailable for an agent, Hivenet Router skips that gate for the agent rather than excluding it.
See [Policy YAML reference](/routing/policy-yaml-reference) for the full policy schema.
## Tune SGLang
### Use more than one GPU
Set the tensor-parallel size:
```bash theme={null}
--tp 2
```
Choose a value that matches the number of GPUs assigned to the SGLang process.
### Enable RadixAttention
Start SGLang with:
```bash theme={null}
--enable-radix-attn
```
RadixAttention can improve prefix reuse for workloads that share prompt prefixes.
### Increase the static memory fraction
Set:
```bash theme={null}
--mem-fraction-static 0.9
```
This allocates a larger fraction of GPU memory to SGLang’s static memory pool.
Higher memory allocation leaves less headroom for the runtime and other processes. Test the setting with your model and workload before using it in production.
## Troubleshooting
### SGLang is not ready
Check the health endpoint:
```bash theme={null}
curl -i http://localhost:3000/health
```
Check model discovery:
```bash theme={null}
curl http://localhost:3000/v1/models \
| jq .
```
Inspect the server logs:
```bash theme={null}
docker logs sglang
```
Review the terminal or service logs for the SGLang process.
The agent waits and retries while SGLang is unavailable or still loading.
### Metrics are missing
Confirm that SGLang was started with:
```bash theme={null}
--enable-metrics
```
Check the metrics endpoint directly:
```bash theme={null}
curl http://localhost:3000/metrics \
| head
```
Without metrics enabled, the endpoint may return HTTP `404`.
A metrics scrape failure does not stop the agent from forwarding requests.
### The wrong model is registered
Automatic discovery selects the first model returned by `/v1/models`.
Pin the correct model:
```bash theme={null}
--model meta-llama/Llama-3.1-8B-Instruct
```
Restart the agent after changing the value.
### The agent does not register
Check the agent logs:
```bash theme={null}
docker logs hivenet-agent
```
```bash theme={null}
sudo journalctl \
-u hivenet-agent \
-n 100 \
--no-pager
```
Test connectivity from the agent to the router:
```bash theme={null}
nc -zv 192.168.1.100 50051
nc -zv 192.168.1.100 9000
```
Check that:
* the router and agent use the same JWT secret
* the router’s libp2p interface is reachable
* the router advertises a libp2p address the agent can reach
* SGLang is healthy and exposes a model
### The peer ID changes after restart
Make sure `--identity-path` points to persistent storage.
For Docker:
```bash theme={null}
docker inspect hivenet-agent \
| jq '.[0].Mounts'
```
For bare metal:
```bash theme={null}
ls -l /var/lib/hivenet-router/agent/identity.key
```
### Requests time out
The agent’s backend HTTP timeout defaults to two minutes.
Increase it for long-running requests:
```bash theme={null}
--http-timeout 10m
```
Also check SGLang’s queue length, cache pressure, and the capacity declared by the agent.
### GPU metrics are missing
Check that NVIDIA tools work on the host:
```bash theme={null}
nvidia-smi
```
For Docker, confirm the agent received GPU access:
```bash theme={null}
docker inspect hivenet-agent \
| jq '.[0].HostConfig.DeviceRequests'
```
The agent continues to operate without NVML. It reports CPU and memory metrics but omits GPU measurements.
## Next steps
Connect a llama.cpp server and enable its Prometheus metrics.
Learn how Hivenet Router filters and ranks matching agents.
Understand the metrics collected from SGLang and other engines.