Infinity agent

Connect an Infinity server to Hivenet Router for embedding and reranking workloads. Infinity can serve several models from one process. Hivenet Router connects to that process through separate agents, with one agent representing one model and one capability.

The Infinity integration does not support chat completions. Start each agent with either --capability embedding or --capability reranker.

Before you start

You need:

a running Hivenet Router router
Infinity installed or available as a Docker image
one or more embedding or reranking models
network access between the agent host and router
the Hivenet Router agent binary or Docker image
the same JWT secret used by the router
an NVIDIA GPU if you want CUDA inference and GPU metrics

The examples use one Infinity process serving two models:

Capability	Model ID	Served model name
Embeddings	`BAAI/bge-m3`	`bge-m3`
Reranking	`BAAI/bge-reranker-large`	`bge-reranker-large`

They also use:

Router IP: 192.168.1.100
Router gRPC port: 50051
Router libp2p port: 9000
Agent IP: 192.168.1.101
Infinity port: 7997

Replace these values with addresses and models from your deployment.

Start Infinity

Hivenet Router expects Infinity to expose:

GET /health
GET /v1/models
POST /v1/embeddings
POST /v1/rerank

Set INFINITY_URL_PREFIX=/v1 so the inference and model endpoints use the paths expected by Hivenet Router.

Docker
Bare metal

Start one Infinity process with an embedding model and a reranking model:

docker run -d \
  --name infinity \
  --restart unless-stopped \
  --gpus all \
  -p 7997:7997 \
  -v infinity-cache:/app/.cache \
  -e INFINITY_URL_PREFIX=/v1 \
  michaelf34/infinity:latest \
  v2 \
    --model-id BAAI/bge-m3 \
    --model-id BAAI/bge-reranker-large \
    --served-model-name bge-m3 \
    --served-model-name bge-reranker-large \
    --host 0.0.0.0 \
    --port 7997 \
    --device cuda \
    --dtype float16 \
    --batch-size 32

The named volume preserves downloaded model data between container replacements.Follow the startup logs:

docker logs -f infinity

For CPU inference, use an appropriate CPU image and remove:

--gpus all

Adjust --device and --dtype for the selected Infinity image and hardware.

Install Infinity:

pip install "infinity-emb[all]"

Set the API prefix:

export INFINITY_URL_PREFIX=/v1

Start Infinity:

nohup infinity_emb v2 \
  --model-id BAAI/bge-m3 \
  --model-id BAAI/bge-reranker-large \
  --served-model-name bge-m3 \
  --served-model-name bge-reranker-large \
  --host 0.0.0.0 \
  --port 7997 \
  --device cuda \
  --dtype float16 \
  --batch-size 32 \
  > ~/infinity.log 2>&1 &

Follow the logs:

tail -f ~/infinity.log

Wait until Infinity is ready:

curl -i http://localhost:7997/health

A ready server returns HTTP status 200. List the exposed models:

curl http://localhost:7997/v1/models \
  | jq '.data[] | {
      id,
      object
    }'

The response should include:

bge-m3
bge-reranker-large

Understand model names

--model-id tells Infinity which model to load. --served-model-name gives that model the public name exposed through Infinity’s API. For example:

--model-id BAAI/bge-m3 \
--served-model-name bge-m3

The Hivenet Router agent and API clients use:

bge-m3

They do not need to use the original Hugging Face model ID.

The order of repeated --served-model-name values must match the order of the corresponding --model-id values.

Prepare the Hivenet Router agents

You need two agents for this example:

one embedding agent for bge-m3
one reranking agent for bge-reranker-large

Both agents connect to the same Infinity process.

Docker
Bare metal

From the Hivenet Router repository, build the agent image if you have not already done so:

docker build \
  -f Dockerfile.agent \
  -t hivenet-router/agent:latest \
  .

Create a directory for the shared secret and persistent identities:

sudo mkdir -p /opt/hivenet-router
sudo chown "$USER":"$USER" /opt/hivenet-router

Place the router’s JWT secret at:

/opt/hivenet-router/jwt.secret

Protect it:

chmod 600 /opt/hivenet-router/jwt.secret

Build the agent from the Hivenet Router repository:

go build -o bin/hivenet-agent ./cmd/agent/

Place the shared JWT secret somewhere the agents can read, for example:

/opt/hivenet-router/jwt.secret

Create a writable directory for the persistent identities:

sudo mkdir -p /var/lib/hivenet-router/agents
sudo chown "$USER":"$USER" /var/lib/hivenet-router/agents

Start the embedding agent

The embedding agent represents bge-m3 and accepts requests sent to:

POST /v1/embeddings

Docker
Bare metal

docker run -d \
  --name hivenet-router-embedding-agent \
  --restart unless-stopped \
  --network host \
  --gpus all \
  -v /opt/hivenet-router/jwt.secret:/jwt.secret:ro \
  -v /opt/hivenet-router:/data \
  hivenet-router/agent:latest \
  --engine infinity \
  --capability embedding \
  --model bge-m3 \
  --backend-url http://localhost:7997 \
  --router-grpc 192.168.1.100:50051 \
  --jwt-secret-file /jwt.secret \
  --capacity 64 \
  --region EU-France \
  --tags infinity,embedding \
  --identity-path /data/embedding-agent.key

For CPU-only inference, omit:

--gpus all

./bin/hivenet-agent \
  --engine infinity \
  --capability embedding \
  --model bge-m3 \
  --backend-url http://localhost:7997 \
  --router-grpc 192.168.1.100:50051 \
  --jwt-secret-file /opt/hivenet-router/jwt.secret \
  --capacity 64 \
  --region EU-France \
  --tags infinity,embedding \
  --identity-path /var/lib/hivenet-router/agents/embedding.key

Start the reranking agent

The reranking agent represents bge-reranker-large and accepts requests sent to:

POST /v1/rerank

Docker
Bare metal

docker run -d \
  --name hivenet-router-reranking-agent \
  --restart unless-stopped \
  --network host \
  --gpus all \
  -v /opt/hivenet-router/jwt.secret:/jwt.secret:ro \
  -v /opt/hivenet-router:/data \
  hivenet-router/agent:latest \
  --engine infinity \
  --capability reranker \
  --model bge-reranker-large \
  --backend-url http://localhost:7997 \
  --router-grpc 192.168.1.100:50051 \
  --jwt-secret-file /jwt.secret \
  --capacity 32 \
  --region EU-France \
  --tags infinity,reranking \
  --identity-path /data/reranking-agent.key

./bin/hivenet-agent \
  --engine infinity \
  --capability reranker \
  --model bge-reranker-large \
  --backend-url http://localhost:7997 \
  --router-grpc 192.168.1.100:50051 \
  --jwt-secret-file /opt/hivenet-router/jwt.secret \
  --capacity 32 \
  --region EU-France \
  --tags infinity,reranking \
  --identity-path /var/lib/hivenet-router/agents/reranking.key

When several agents run on the same host, each one needs:

a separate --identity-path
a unique process or container name

The agents can share the same --backend-url. They initiate their own connections to the router, so you do not need a separate inbound agent port for each process.

Understand capabilities

Hivenet Router uses the agent capability to separate different kinds of inference traffic.

Capability	Hivenet Router endpoint	Infinity endpoint
`embedding`	`POST /v1/embeddings`	`POST /v1/embeddings`
`reranker`	`POST /v1/rerank`	`POST /v1/rerank`

The router filters agents by capability before applying the rest of the routing policy. An embedding request cannot be sent to a reranking agent, even if both agents use Infinity or share the same backend process. Do not leave the default capability of llm on an Infinity agent. Infinity rejects chat-completion requests, and Hivenet Router reports a structured invalid-request error.

Set agent capacity

--capacity controls how many concurrent requests Hivenet Router may assign to an agent. It does not change Infinity’s batching or scheduler settings. The original deployment examples use:

Agent	Example capacity
Embeddings	`64`
Reranking	`32`

Treat these as starting points. The appropriate value depends on:

model size
accelerator and available memory
input length
batch size
number of models sharing the Infinity process
acceptable latency

Because both agents share one Infinity process and one set of hardware resources, their capacities are not independent physical limits. Test the combined workload before raising either value.

For streaming-capable workloads, Hivenet Router releases an agent capacity slot when response headers arrive, while backend work can continue. Infinity embeddings and reranking are normally non-streaming, but the same --capacity value remains a routing-admission setting rather than a direct backend scheduler limit.

Verify both agents

On the router, check health:

curl http://192.168.1.100:8080/admin/health \
  | jq .

Inspect the Infinity agents:

curl http://192.168.1.100:8080/admin/routing-table \
  | jq '.agents[]
      | select(.metadata.engine == "infinity")
      | {
          peer_id,
          model: .metadata.model,
          capability: .metadata.capability,
          capacity: .metadata.capacity,
          region: .metadata.region,
          healthy: .status.healthy,
          active_requests: .status.active_requests
        }'

List the models exposed to clients:

curl http://192.168.1.100:8080/v1/models \
  | jq '.data[] | {
      id,
      capability
    }'

You should see one embedding model and one reranking model.

Test embeddings

Send a batch embedding request:

curl -X POST \
  http://192.168.1.100:8080/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{
    "model": "bge-m3",
    "input": [
      "Hello world",
      "How are you?"
    ]
  }'

A successful response contains one vector for each input:

{
  "object": "list",
  "data": [
    {
      "object": "embedding",
      "embedding": [0.012, -0.031, 0.044],
      "index": 0
    }
  ],
  "model": "bge-m3",
  "usage": {
    "prompt_tokens": 5,
    "total_tokens": 5
  }
}

The vector above is shortened for readability.

Test reranking

Send a reranking request:

curl -X POST \
  http://192.168.1.100:8080/v1/rerank \
  -H "Content-Type: application/json" \
  -d '{
    "model": "bge-reranker-large",
    "query": "What is the capital of France?",
    "documents": [
      "Paris is the capital of France.",
      "London is the capital of the United Kingdom.",
      "Berlin is the capital of Germany."
    ],
    "top_n": 2
  }'

The response orders results by relevance score:

{
  "object": "rerank",
  "results": [
    {
      "relevance_score": 0.99,
      "index": 0,
      "document": null
    }
  ],
  "model": "bge-reranker-large"
}

The score above is illustrative. Actual values depend on the model and input. The index value points to the document’s position in the original input array.

Routing behavior

Hivenet Router routes embedding and reranking requests through the same policy pipeline used for language-model requests:

Filter by model.
Filter by capability.
Apply static policy matches.
Apply health, capacity, and dynamic gates.
Rank the remaining agents.
Continue through configured fallback steps if needed.

A policy can target Infinity agents:

routing_policy:
  match:
    engine: infinity
    region: EU-France
  exclude_if:
    success_rate:
      lt: 0.95
  strategy: least-loaded

Capability filtering is automatic. You do not need a policy rule to prevent embedding traffic from reaching reranking agents.

Observability

The Infinity integration does not currently scrape engine-specific metrics from the Infinity backend. Hivenet Router still exposes:

registered agent state
capability and model metadata
routed and failed request counters
active requests and capacity utilization
request success rate
smoothed round-trip time
CPU and memory metrics
NVIDIA GPU metrics when NVML is available
audit records when audit logging is enabled

Embedding request rate:

rate(
  hivenet_router_routing_requests_routed_total{
    engine="infinity",
    model="bge-m3"
  }[5m]
)

Reranking request rate:

rate(
  hivenet_router_routing_requests_routed_total{
    engine="infinity",
    model="bge-reranker-large"
  }[5m]
)

Smoothed round-trip time:

hivenet_router_agent_srtt_ms{
  engine="infinity"
}

Inspect all Infinity-related metrics:

curl http://192.168.1.100:2112/metrics \
  | grep 'engine="infinity"'

Troubleshooting

`/v1/models` returns `404`

Confirm that Infinity was started with:

INFINITY_URL_PREFIX=/v1

For Docker, inspect the container environment:

docker inspect infinity \
  | jq '.[0].Config.Env'

Without the prefix, Infinity serves its API at different paths from those expected by Hivenet Router. Restart Infinity after changing the setting.

An agent cannot discover its model

Check Infinity:

curl http://localhost:7997/health

List model IDs:

curl http://localhost:7997/v1/models \
  | jq '.data[].id'

The returned ID must match the agent’s --model value exactly.

The wrong model name is exposed

Check the order of:

--model-id
--served-model-name

Each served name corresponds to the model ID in the same position. Restart Infinity after changing the mapping.

Embedding requests are routed incorrectly

Confirm that the agent uses:

--capability embedding

Inspect the routing table:

curl http://192.168.1.100:8080/admin/routing-table \
  | jq '.agents[]
      | select(.metadata.model == "bge-m3")
      | {
          model: .metadata.model,
          capability: .metadata.capability,
          engine: .metadata.engine
        }'

Reranking requests are routed incorrectly

Confirm that the agent uses:

--capability reranker

The accepted capability value is reranker, while the HTTP endpoint is:

/v1/rerank

An agent does not register

Check its logs:

Docker
systemd

docker logs hivenet-router-embedding-agent
docker logs hivenet-router-reranking-agent

sudo journalctl \
  -u hivenet-agent \
  -n 100 \
  --no-pager

Test connectivity from the agent host:

nc -zv 192.168.1.100 50051
nc -zv 192.168.1.100 9000

Check that:

the router and agents use the same JWT secret
the agent host can reach the router’s gRPC and libp2p ports
each agent has a unique identity path
Infinity is healthy
the selected model is listed by /v1/models

A peer ID changes after restart

Make sure each --identity-path points to persistent storage. For Docker:

docker inspect hivenet-router-embedding-agent \
  | jq '.[0].Mounts'

For bare metal:

ls -l /var/lib/hivenet-router/agents/

Requests time out

The agent’s backend HTTP timeout defaults to two minutes. Increase it when large batches need more time:

--http-timeout 10m

Also review:

embedding or document batch size
input length
declared agent capacity
Infinity batch size
whether both models compete for the same GPU memory

GPU metrics are missing

Check the host:

nvidia-smi

For Docker, confirm that each agent received GPU access:

docker inspect hivenet-router-embedding-agent \
  | jq '.[0].HostConfig.DeviceRequests'

The agents continue to operate without NVML. They report CPU and memory metrics but omit GPU measurements.

Next steps

Custom engine agent

Connect another OpenAI-compatible backend to Hivenet Router.

Embeddings

Review the embedding request and response format.

Reranking

Review reranking fields, examples, and response ordering.

Getting started

Deploy Hivenet Router

Use the API

Routing and policies

Security and auth

Observability

Integrations

Reference

Project

Before you start

Start Infinity

Understand model names

Prepare the Hivenet Router agents

Start the embedding agent

Start the reranking agent

Understand capabilities

Set agent capacity

Verify both agents

Test embeddings

Test reranking

Routing behavior

Observability

Troubleshooting

`/v1/models` returns `404`

An agent cannot discover its model

The wrong model name is exposed

Embedding requests are routed incorrectly

Reranking requests are routed incorrectly

An agent does not register

A peer ID changes after restart

Requests time out

GPU metrics are missing

Next steps

Custom engine agent

Embeddings

Reranking

​Before you start

​Start Infinity

​Understand model names

​Prepare the Hivenet Router agents

​Start the embedding agent

​Start the reranking agent

​Understand capabilities

​Set agent capacity

​Verify both agents

​Test embeddings

​Test reranking

​Routing behavior

​Observability

​Troubleshooting

​/v1/models returns 404

​An agent cannot discover its model

​The wrong model name is exposed

​Embedding requests are routed incorrectly

​Reranking requests are routed incorrectly

​An agent does not register

​A peer ID changes after restart

​Requests time out

​GPU metrics are missing

​Next steps

Custom engine agent

Embeddings

Reranking

Before you start

Start Infinity

Understand model names

Prepare the Hivenet Router agents

Start the embedding agent

Start the reranking agent

Understand capabilities

Set agent capacity

Verify both agents

Test embeddings

Test reranking

Routing behavior

Observability

Troubleshooting

`/v1/models` returns `404`

An agent cannot discover its model

The wrong model name is exposed

Embedding requests are routed incorrectly

Reranking requests are routed incorrectly

An agent does not register

A peer ID changes after restart

Requests time out

GPU metrics are missing

Next steps