SGLang agent

Connect an SGLang inference server to Hivenet Router by running an agent beside the backend. The agent discovers the model exposed by SGLang, registers it with the router, forwards OpenAI-compatible chat-completion requests, and reports engine and hardware metrics.

Each Hivenet Router agent registers one model. If an SGLang server exposes several models, run a separate agent for each model and use --model to select the model that agent represents.

Before you start

You need:

a running Hivenet Router router
an NVIDIA GPU with CUDA support
SGLang installed or available as a Docker image
network access between the agent host and router
the Hivenet Router agent binary or Docker image
the same JWT secret used by the router
access to the model you want to serve

The examples use:

Router IP: 192.168.1.100
Router gRPC port: 50051
Router libp2p port: 9000
Agent IP: 192.168.1.101
SGLang port: 3000
Model: meta-llama/Llama-3.1-8B-Instruct

Replace these values with addresses and model names from your deployment.

Start SGLang

Docker
Bare metal

Start SGLang with host networking:

docker run -d \
  --name sglang \
  --restart unless-stopped \
  --gpus all \
  --network host \
  sglangai/sglang:latest \
  python3 -m sglang.launch_server \
    --model-path meta-llama/Llama-3.1-8B-Instruct \
    --host 0.0.0.0 \
    --port 3000 \
    --enable-metrics

Follow the startup logs:

docker logs -f sglang

Install the system dependency required by sgl_kernel:

sudo apt-get update
sudo apt-get install -y libnuma-dev

Install SGLang:

pip install "sglang[all]"

Start the server:

python3 -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 3000 \
  --enable-metrics

Start SGLang with --enable-metrics. Without it, Hivenet Router can still forward requests, but SGLang’s cache, queue, and time-to-first-token metrics will not be available.

For a multi-GPU deployment, set the tensor-parallel size:

python3 -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 3000 \
  --tp 2 \
  --enable-metrics

Wait for the model to load, then check the health endpoint:

curl -i http://localhost:3000/health

A ready SGLang server returns HTTP status 200. List the models reported by SGLang:

curl http://localhost:3000/v1/models \
  | jq '.data[].id'

The model ID returned here is the value clients must use when sending requests through Hivenet Router.

Prepare the Hivenet Router agent

Docker
Bare metal

From the Hivenet Router repository, build the agent image if you have not already done so:

docker build \
  -f Dockerfile.agent \
  -t hivenet-router/agent:latest \
  .

Create a directory for the shared secret and persistent agent identity:

sudo mkdir -p /opt/hivenet-router
sudo chown "$USER":"$USER" /opt/hivenet-router

Place the router’s JWT secret at:

/opt/hivenet-router/jwt.secret

Protect it:

chmod 600 /opt/hivenet-router/jwt.secret

Build the agent from the Hivenet Router repository:

go build -o bin/hivenet-agent ./cmd/agent/

Place the shared JWT secret somewhere the agent process can read, for example:

/opt/hivenet-router/jwt.secret

Create a writable directory for the persistent agent identity:

sudo mkdir -p /var/lib/hivenet-router/agent
sudo chown "$USER":"$USER" /var/lib/hivenet-router/agent

Start the agent

Docker
Bare metal

docker run -d \
  --name hivenet-agent \
  --restart unless-stopped \
  --network host \
  --gpus all \
  -v /opt/hivenet-router/jwt.secret:/jwt.secret:ro \
  -v /opt/hivenet-router:/data \
  hivenet-router/agent:latest \
  --engine sglang \
  --backend-url http://localhost:3000 \
  --router-grpc 192.168.1.100:50051 \
  --jwt-secret-file /jwt.secret \
  --capacity 20 \
  --region EU-France \
  --organization ml-team \
  --machine gpu-worker-1 \
  --identity-path /data/agent-identity.key

The /data mount preserves the agent’s peer identity across container replacement and restarts.The agent container receives GPU access so it can collect NVIDIA hardware metrics. SGLang continues to run as a separate process or container.

./bin/hivenet-agent \
  --engine sglang \
  --backend-url http://localhost:3000 \
  --router-grpc 192.168.1.100:50051 \
  --jwt-secret-file /opt/hivenet-router/jwt.secret \
  --capacity 20 \
  --region EU-France \
  --organization ml-team \
  --machine gpu-worker-1 \
  --identity-path /var/lib/hivenet-router/agent/identity.key

The agent waits for SGLang to become healthy and expose a model before registering with the router. If SGLang is still loading, the agent continues polling rather than exiting permanently.

Understand the agent settings

Setting	Purpose
`--engine sglang`	Selects the SGLang backend integration
`--backend-url`	Base URL of the local SGLang server
`--router-grpc`	Router endpoint used for agent authentication
`--jwt-secret-file`	Shared secret used to authenticate the agent
`--capacity`	Maximum concurrent requests Hivenet Router may assign
`--region`	Region metadata available to routing policies
`--organization`	Organization or infrastructure-provider metadata
`--machine`	Stable machine identifier
`--identity-path`	Persistent private key that keeps the peer ID stable

You can also add comma-separated routing tags:

--tags production,sglang,high-throughput

Policies can match these values when selecting agents.

Select the model

The agent can discover the model automatically or use an explicit model name.

Automatic discovery

When --model is omitted, the agent requests:

GET <backend-url>/v1/models

It registers the first model returned by SGLang. For an SGLang server exposing one model, automatic discovery is usually sufficient.

Explicit model selection

Use --model to pin the agent to a particular model:

--model meta-llama/Llama-3.1-8B-Instruct

The explicit value overrides automatic discovery.

The model registered by the agent must match the model name clients send to Hivenet Router.

Set agent capacity

--capacity controls how many concurrent requests Hivenet Router may assign to the agent. It does not change SGLang’s own scheduler or memory configuration. Start with a conservative value based on:

available VRAM
model size
tensor-parallel configuration
expected context length
request concurrency
acceptable latency

For example:

--capacity 20

When the agent reaches its declared capacity, Hivenet Router stops assigning new requests to it and considers other matching agents or configured fallback steps. Test the backend under representative load before increasing capacity.

For streaming responses, Hivenet Router releases the agent capacity slot when response headers arrive, while backend generation can continue. Treat --capacity as a routing-admission setting rather than a hard limit on ongoing streams, and verify SGLang concurrency under sustained streaming load.

Verify the connection

On the router, check agent health:

curl http://192.168.1.100:8080/admin/health \
  | jq .

List the models available to clients:

curl http://192.168.1.100:8080/v1/models \
  | jq '.data[].id'

Inspect the SGLang agent:

curl http://192.168.1.100:8080/admin/routing-table \
  | jq '.agents[] | select(.metadata.engine == "sglang")'

Send a chat-completion request:

curl -X POST http://192.168.1.100:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [
      {
        "role": "user",
        "content": "Explain what an inference router does in one sentence."
      }
    ]
  }'

Applications call Hivenet Router’s OpenAI-compatible endpoint. The agent forwards the request to SGLang’s OpenAI-compatible API.

SGLang metrics

The agent scrapes SGLang’s /metrics endpoint every 500 milliseconds by default. It sends the resulting engine data to the router, which exposes it through the router’s Prometheus endpoint. The agent does not expose a separate Prometheus port.

Signal	SGLang source	Hivenet Router field
Token-cache utilization	`sglang:token_usage`	`kv_cache_utilization`
Running requests	`sglang:num_running_reqs`	`running_requests`
Queued requests	`sglang:num_queue_reqs`	`waiting_requests`
Average time to first token	`sglang:time_to_first_token_seconds`	Average TTFT
P90 time to first token	`sglang:time_to_first_token_seconds`	P90 TTFT

SGLang does not expose an inter-token-latency histogram through this integration. Hivenet Router therefore does not populate average or P90 ITL values for SGLang agents. SGLang also has no preemption metric equivalent to the one collected from vLLM. View engine metrics on the router:

curl http://192.168.1.100:2112/metrics \
  | grep hivenet_agent_engine

Useful exported metrics include:

hivenet_agent_engine_kv_cache_utilization
hivenet_agent_engine_running_requests
hivenet_agent_engine_waiting_requests
hivenet_agent_engine_avg_ttft_seconds
hivenet_agent_engine_p90_ttft_seconds

View current values through the routing table:

curl http://192.168.1.100:8080/admin/routing-table \
  | jq '.agents[]
      | select(.metadata.engine == "sglang")
      | {
          peer_id,
          model: .metadata.model,
          kv_cache_utilization: .engine.kv_cache_utilization,
          running_requests: .engine.running_requests,
          waiting_requests: .engine.waiting_requests
        }'

Use SGLang metrics in routing policies

Exclude agents whose token cache is under pressure:

exclude_if:
  kv_cache_utilization:
    gt: 0.85

Exclude agents with queued requests:

exclude_if:
  waiting_requests:
    gt: 0

Exclude agents whose time to first token is too high:

exclude_if:
  avg_ttft_seconds:
    gt: 2

When a metric is unavailable for an agent, Hivenet Router skips that gate for the agent rather than excluding it. See Policy YAML reference for the full policy schema.

Tune SGLang

Use more than one GPU

Set the tensor-parallel size:

--tp 2

Choose a value that matches the number of GPUs assigned to the SGLang process.

Enable RadixAttention

Start SGLang with:

--enable-radix-attn

RadixAttention can improve prefix reuse for workloads that share prompt prefixes.

Increase the static memory fraction

Set:

--mem-fraction-static 0.9

This allocates a larger fraction of GPU memory to SGLang’s static memory pool.

Higher memory allocation leaves less headroom for the runtime and other processes. Test the setting with your model and workload before using it in production.

Troubleshooting

SGLang is not ready

Check the health endpoint:

curl -i http://localhost:3000/health

Check model discovery:

curl http://localhost:3000/v1/models \
  | jq .

Inspect the server logs:

Docker
Bare metal

docker logs sglang

The agent waits and retries while SGLang is unavailable or still loading.

Metrics are missing

Confirm that SGLang was started with:

--enable-metrics

Check the metrics endpoint directly:

curl http://localhost:3000/metrics \
  | head

Without metrics enabled, the endpoint may return HTTP 404.A metrics scrape failure does not stop the agent from forwarding requests.

The wrong model is registered

Automatic discovery selects the first model returned by /v1/models.Pin the correct model:

--model meta-llama/Llama-3.1-8B-Instruct

Restart the agent after changing the value.

The agent does not register

Check the agent logs:

Docker
systemd

docker logs hivenet-agent

sudo journalctl \
  -u hivenet-agent \
  -n 100 \
  --no-pager

Test connectivity from the agent to the router:

nc -zv 192.168.1.100 50051
nc -zv 192.168.1.100 9000

Check that:

the router and agent use the same JWT secret
the router’s libp2p interface is reachable
the router advertises a libp2p address the agent can reach
SGLang is healthy and exposes a model

The peer ID changes after restart

Make sure --identity-path points to persistent storage.For Docker:

docker inspect hivenet-agent \
  | jq '.[0].Mounts'

For bare metal:

ls -l /var/lib/hivenet-router/agent/identity.key

Requests time out

The agent’s backend HTTP timeout defaults to two minutes.Increase it for long-running requests:

--http-timeout 10m

Also check SGLang’s queue length, cache pressure, and the capacity declared by the agent.

GPU metrics are missing

Check that NVIDIA tools work on the host:

nvidia-smi

For Docker, confirm the agent received GPU access:

docker inspect hivenet-agent \
  | jq '.[0].HostConfig.DeviceRequests'

The agent continues to operate without NVML. It reports CPU and memory metrics but omits GPU measurements.

Next steps

llama.cpp agent

Connect a llama.cpp server and enable its Prometheus metrics.

Routing concepts

Learn how Hivenet Router filters and ranks matching agents.

Engine metrics

Understand the metrics collected from SGLang and other engines.

Getting started

Deploy Hivenet Router

Use the API

Routing and policies

Security and auth

Observability

Integrations

Reference

Project

Before you start

Start SGLang

Prepare the Hivenet Router agent

Start the agent

Understand the agent settings

Select the model

Automatic discovery

Explicit model selection

Set agent capacity

Verify the connection

SGLang metrics

Use SGLang metrics in routing policies

Tune SGLang

Use more than one GPU

Enable RadixAttention

Increase the static memory fraction

Troubleshooting

Next steps

llama.cpp agent

Routing concepts

Engine metrics

​Before you start

​Start SGLang

​Prepare the Hivenet Router agent

​Start the agent

​Understand the agent settings

​Select the model

​Automatic discovery

​Explicit model selection

​Set agent capacity

​Verify the connection

​SGLang metrics

​Use SGLang metrics in routing policies

​Tune SGLang

​Use more than one GPU

​Enable RadixAttention

​Increase the static memory fraction

​Troubleshooting

​Next steps

llama.cpp agent

Routing concepts

Engine metrics

Before you start

Start SGLang

Prepare the Hivenet Router agent

Start the agent

Understand the agent settings

Select the model

Automatic discovery

Explicit model selection

Set agent capacity

Verify the connection

SGLang metrics

Use SGLang metrics in routing policies

Tune SGLang

Use more than one GPU

Enable RadixAttention

Increase the static memory fraction

Troubleshooting

Next steps