Custom engine agent

Connect another OpenAI-compatible inference server to Hivenet Router with the custom engine. Use this integration when your backend is not covered by one of Hivenet Router’s built-in engine types but exposes a compatible chat-completions endpoint.

The custom engine requires both --model and --health-url. It does not discover either value from the backend.

Backend requirements

The backend must expose:

Endpoint	Requirement
Health endpoint	A `GET` endpoint that returns HTTP `200` when the backend is ready
`POST /v1/chat/completions`	An OpenAI-compatible chat-completions endpoint

The health endpoint can use any path. Pass its complete URL to --health-url. For example:

Backend URL: http://localhost:8000
Health URL: http://localhost:8000/health
Chat completions: http://localhost:8000/v1/chat/completions

Hivenet Router forwards the original chat-completion request body and HTTP headers to the backend. It does not translate a proprietary request format into the OpenAI schema.

This page covers language-model agents using the default llm capability. Use the dedicated Infinity integration for the documented embedding and reranking workflow.

Before you start

You need:

a running Hivenet Router router
an OpenAI-compatible inference backend
a health endpoint that returns HTTP 200
network access between the agent and router
the Hivenet Router agent binary or Docker image
the same JWT secret used by the router
a model name that clients and the backend can use

The examples use:

Router IP: 192.168.1.100
Router gRPC port: 50051
Router libp2p port: 9000
Agent IP: 192.168.1.101
Backend port: 8000
Model: meta-llama/Llama-3.1-8B-Instruct

Replace these values with addresses and model names from your deployment.

Start your backend

The command depends on the inference server you are using. A generic example might look like:

python -m my_inference_server \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --host 127.0.0.1 \
  --port 8000

Because the Hivenet Router agent runs beside the backend, the backend can remain bound to localhost unless another machine must reach it directly. Check the health endpoint:

curl -i http://localhost:8000/health

Hivenet Router considers the backend ready only when this endpoint returns HTTP 200. Test chat completions directly:

curl -X POST \
  http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [
      {
        "role": "user",
        "content": "Hello"
      }
    ]
  }'

Confirm this request works before starting the Hivenet Router agent.

Choose the model name

The custom engine does not call /v1/models. You must provide the model name explicitly:

--model meta-llama/Llama-3.1-8B-Instruct

Hivenet Router uses this value to:

register the agent
list the model through GET /v1/models
match incoming requests to the agent
label routing and observability data

The request body is forwarded without replacing its model field. For the cleanest setup, use the same model name:

in the agent’s --model value
in client requests
in the backend’s public API

If the backend uses another internal identifier, configure an alias or compatibility layer on the backend side.

Prepare the Hivenet Router agent

Docker
Bare metal

From the Hivenet Router repository, build the agent image if you have not already done so:

docker build \
  -f Dockerfile.agent \
  -t hivenet-router/agent:latest \
  .

Create a directory for the shared secret and persistent identity:

sudo mkdir -p /opt/hivenet-router
sudo chown "$USER":"$USER" /opt/hivenet-router

Place the router’s JWT secret at:

/opt/hivenet-router/jwt.secret

Protect it:

chmod 600 /opt/hivenet-router/jwt.secret

Build the agent from the Hivenet Router repository:

go build -o bin/hivenet-agent ./cmd/agent/

Place the shared JWT secret somewhere the agent can read, for example:

/opt/hivenet-router/jwt.secret

Create a writable directory for the persistent identity:

sudo mkdir -p /var/lib/hivenet-router/agent
sudo chown "$USER":"$USER" /var/lib/hivenet-router/agent

Start the agent

Docker
Bare metal

docker run -d \
  --name hivenet-agent \
  --restart unless-stopped \
  --network host \
  --gpus all \
  -v /opt/hivenet-router/jwt.secret:/jwt.secret:ro \
  -v /opt/hivenet-router:/data \
  hivenet-router/agent:latest \
  --engine custom \
  --backend-url http://localhost:8000 \
  --health-url http://localhost:8000/health \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --router-grpc 192.168.1.100:50051 \
  --jwt-secret-file /jwt.secret \
  --capacity 32 \
  --region EU-Custom \
  --organization ml-team \
  --machine custom-worker-1 \
  --identity-path /data/agent-identity.key

Omit --gpus all when the backend is CPU-only or the agent does not need NVIDIA hardware metrics.The /data mount preserves the agent’s peer identity across container replacement and restarts.

./bin/hivenet-agent \
  --engine custom \
  --backend-url http://localhost:8000 \
  --health-url http://localhost:8000/health \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --router-grpc 192.168.1.100:50051 \
  --jwt-secret-file /opt/hivenet-router/jwt.secret \
  --capacity 32 \
  --region EU-Custom \
  --organization ml-team \
  --machine custom-worker-1 \
  --identity-path /var/lib/hivenet-router/agent/identity.key

The agent waits until the configured health endpoint returns HTTP 200. If the backend is still starting, the agent continues polling rather than exiting permanently.

Understand the custom settings

Setting	Purpose
`--engine custom`	Selects the generic OpenAI-compatible integration
`--backend-url`	Base URL used for chat-completion requests
`--health-url`	Complete URL used to check backend readiness
`--model`	Explicit model name registered with Hivenet Router
`--router-grpc`	Router endpoint used for agent authentication
`--jwt-secret-file`	Shared secret used to authenticate the agent
`--capacity`	Maximum concurrent requests Hivenet Router may assign
`--region`	Region metadata available to routing policies
`--organization`	Organization or infrastructure-provider metadata
`--machine`	Stable machine identifier
`--identity-path`	Persistent private key that keeps the peer ID stable

You can also add routing tags:

--tags custom,production,high-memory

How requests are forwarded

For a non-streaming request, the agent sends:

POST <backend-url>/v1/chat/completions

It forwards the original JSON body and request headers to the backend. For example, a client request to:

http://<router>/v1/chat/completions

is eventually forwarded to:

http://localhost:8000/v1/chat/completions

Backend response headers and response bodies are returned through the agent and router to the client. Backend errors are also passed back through Hivenet Router. Where possible, Hivenet Router classifies common errors such as:

invalid parameters
context-length limits
backend unavailability
rate limits
general backend failures

Streaming responses

The custom engine supports streaming chat completions when the backend returns an SSE response compatible with the OpenAI API. Send:

curl -N -X POST \
  http://192.168.1.100:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "stream": true,
    "messages": [
      {
        "role": "user",
        "content": "Count from one to five."
      }
    ]
  }'

The agent forwards response chunks as they arrive rather than buffering the full completion. The backend must provide a valid streaming response. Hivenet Router does not convert a non-streaming backend response into SSE.

Set agent capacity

--capacity controls how many concurrent requests Hivenet Router may assign to the agent. It does not configure the backend’s own scheduler, worker count, queue, or batch size. Choose a value based on:

backend concurrency
model size
available CPU, RAM, or VRAM
context length
acceptable response time
whether other workloads share the machine

Start conservatively and test with representative traffic. When the agent reaches capacity, Hivenet Router considers another matching agent, waits if queueing is configured, or moves through the policy’s fallback chain.

For streaming responses, Hivenet Router releases the agent capacity slot when response headers arrive, while backend generation can continue. Treat --capacity as a routing-admission setting rather than a hard limit on ongoing streams, and verify the custom backend under sustained streaming load.

Verify the connection

On the router, check operational health:

curl http://192.168.1.100:8080/admin/health \
  | jq .

Confirm the model is visible:

curl http://192.168.1.100:8080/v1/models \
  | jq '.data[]
      | select(
          .id == "meta-llama/Llama-3.1-8B-Instruct"
        )'

Inspect custom agents:

curl http://192.168.1.100:8080/admin/routing-table \
  | jq '.agents[]
      | select(.metadata.engine == "custom")'

Send a request through Hivenet Router:

curl -X POST \
  http://192.168.1.100:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [
      {
        "role": "user",
        "content": "Explain what an inference router does in one sentence."
      }
    ]
  }'

Route to custom agents

Match the custom engine in a routing policy:

routing_policy:
  match:
    engine: custom
    region: EU-Custom
  exclude_if:
    success_rate:
      lt: 0.95
  strategy: least-loaded
  max_tries: 3

Add another engine as a fallback:

fallback_chain:
  - name: vllm-backup
    match:
      engine: vllm
    strategy: least-loaded

Custom agents participate in the same policy pipeline as other language-model agents. You can filter them using metadata such as:

model
engine
region
organization
machine
tags
GPU model

You can also use universal and hardware signals such as success rate, latency, capacity, GPU pressure, CPU use, and memory use.

Available metrics

The custom integration does not scrape engine-specific metrics from the backend. The following information is still available:

Routing and request metrics

routed and failed request counts
active requests and declared capacity
success and failure history
smoothed round-trip time
policy and fallback decisions
tenant and model labels where configured

Hardware metrics

When available on the agent host:

CPU use
system memory use
GPU utilization
VRAM use
GPU temperature
GPU power

Engine-specific fields such as KV cache utilization, backend queue size, time to first token, and inter-token latency are not populated by the custom integration. View the agent through the router:

curl http://192.168.1.100:8080/admin/routing-table \
  | jq '.agents[]
      | select(.metadata.engine == "custom")'

Inspect relevant Prometheus metrics:

curl http://192.168.1.100:2112/metrics \
  | grep 'engine="custom"'

Troubleshooting

The agent reports that `--model` is required

The custom engine cannot discover models. Start the agent with:

--model <model-name>

The agent reports that `--health-url` is required

Provide the complete health endpoint:

--health-url http://localhost:8000/health

This is separate from --backend-url.

The backend never becomes ready

Test the exact URL passed to --health-url:

curl -i http://localhost:8000/health

The endpoint must return HTTP 200. Redirects, authentication challenges, and other status codes are treated as unhealthy.

Direct requests work, but routed requests fail

Test the exact OpenAI-compatible path:

curl -X POST \
  http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [
      {
        "role": "user",
        "content": "Hello"
      }
    ]
  }'

Common causes include:

the backend uses another path
the backend expects a proprietary request schema
the model name differs
the backend requires credentials that are not present in the client request
the backend does not support the requested field or streaming mode

The model is visible but requests do not route

Check the model value in all three places:

Agent --model
Client request body
Backend model identifier

The agent’s registered model must match the client request. Inspect the routing table:

curl http://192.168.1.100:8080/admin/routing-table \
  | jq '.agents[]
      | select(.metadata.engine == "custom")
      | {
          model: .metadata.model,
          capability: .metadata.capability,
          healthy: .status.healthy,
          active_requests: .status.active_requests,
          capacity: .metadata.capacity
        }'

The agent does not register

Check the logs:

Docker
systemd

docker logs hivenet-agent

sudo journalctl \
  -u hivenet-agent \
  -n 100 \
  --no-pager

Test connectivity from the agent to the router:

nc -zv 192.168.1.100 50051
nc -zv 192.168.1.100 9000

Check that:

the router and agent use the same JWT secret
the backend health endpoint returns HTTP 200
the router’s libp2p interface is reachable
the router advertises a libp2p address the agent can reach

Streaming is buffered or incomplete

Test the backend directly with curl -N. Confirm that it returns a streaming content type and flushes chunks as they are generated. Hivenet Router can relay an SSE stream, but it cannot correct buffering introduced by the backend or an intermediate proxy.

Requests time out

The backend HTTP timeout defaults to two minutes. Increase it for longer requests:

--http-timeout 10m

For streaming responses, the rolling stream-write timeout is configured separately:

--stream-write-timeout 2m

Also check the backend’s own timeout, queue, and concurrency settings.

The peer ID changes after restart

Make sure --identity-path points to persistent storage. For Docker:

docker inspect hivenet-agent \
  | jq '.[0].Mounts'

For bare metal:

ls -l /var/lib/hivenet-router/agent/identity.key

Next steps

Chat completions

Review the supported request, response, and streaming behavior.

Routing concepts

Learn how custom agents participate in routing and fallback.

Hardware metrics

Understand the system and GPU signals collected from agent hosts.

Getting started

Deploy Hivenet Router

Use the API

Routing and policies

Security and auth

Observability

Integrations

Reference

Project

Backend requirements

Before you start

Start your backend

Choose the model name

Prepare the Hivenet Router agent

Start the agent

Understand the custom settings

How requests are forwarded

Streaming responses

Set agent capacity

Verify the connection

Route to custom agents

Available metrics

Routing and request metrics

Hardware metrics

Troubleshooting

The agent reports that `--model` is required

The agent reports that `--health-url` is required

The backend never becomes ready

Direct requests work, but routed requests fail

The model is visible but requests do not route

The agent does not register

Streaming is buffered or incomplete

Requests time out

The peer ID changes after restart

Next steps

Chat completions

Routing concepts

Hardware metrics

​Backend requirements

​Before you start

​Start your backend

​Choose the model name

​Prepare the Hivenet Router agent

​Start the agent

​Understand the custom settings

​How requests are forwarded

​Streaming responses

​Set agent capacity

​Verify the connection

​Route to custom agents

​Available metrics

​Routing and request metrics

​Hardware metrics

​Troubleshooting

​The agent reports that --model is required

​The agent reports that --health-url is required

​The backend never becomes ready

​Direct requests work, but routed requests fail

​The model is visible but requests do not route

​The agent does not register

​Streaming is buffered or incomplete

​Requests time out

​The peer ID changes after restart

​Next steps

Chat completions

Routing concepts

Hardware metrics

Backend requirements

Before you start

Start your backend

Choose the model name

Prepare the Hivenet Router agent

Start the agent

Understand the custom settings

How requests are forwarded

Streaming responses

Set agent capacity

Verify the connection

Route to custom agents

Available metrics

Routing and request metrics

Hardware metrics

Troubleshooting

The agent reports that `--model` is required

The agent reports that `--health-url` is required

The backend never becomes ready

Direct requests work, but routed requests fail

The model is visible but requests do not route

The agent does not register

Streaming is buffered or incomplete

Requests time out

The peer ID changes after restart

Next steps