Ollama agent

Connect an Ollama server to Hivenet Router by running an agent beside the backend. The agent checks Ollama through its /api/tags endpoint, registers one available model with the router, and forwards chat-completion requests to Ollama’s OpenAI-compatible API.

One Hivenet Router agent represents one model. If your Ollama server contains several models, pin each agent to a model with --model and run a separate agent process for each one.

Ollama does not provide the engine metrics collected from vLLM, SGLang, or metrics-enabled llama.cpp servers. Hivenet Router still records routing, request, latency, health, and available hardware metrics for the agent.

Before you start

You need:

a running Hivenet Router router
Ollama on the agent host
at least one model pulled in Ollama
network access between the agent and router
the Hivenet Router agent binary or Docker image
the same JWT secret used by the router

The examples use:

Router IP: 192.168.1.100
Router gRPC port: 50051
Router libp2p port: 9000
Agent IP: 192.168.1.101
Ollama port: 11434
Model: llama3.1:8b

Replace these values with addresses and model names from your deployment.

Start Ollama

Linux
Docker

Install Ollama:

curl -fsSL https://ollama.com/install.sh | sh

Start the server:

export OLLAMA_HOST=127.0.0.1:11434
export OLLAMA_KEEP_ALIVE=-1

nohup ollama serve > ~/ollama.log 2>&1 &

OLLAMA_KEEP_ALIVE=-1 keeps loaded models in memory instead of unloading them after an idle period.To store models in another directory, set OLLAMA_MODELS before starting the server:

export OLLAMA_MODELS=/path/to/models

The agent runs on the same host, so Ollama does not need to listen on a public interface. Bind it to 0.0.0.0 only when another machine must connect directly and your network controls restrict access appropriately.

Start Ollama with persistent model storage:

docker run -d \
  --name ollama \
  --restart unless-stopped \
  --network host \
  --gpus all \
  -e OLLAMA_KEEP_ALIVE=-1 \
  -v ollama:/root/.ollama \
  ollama/ollama

For a CPU-only host, omit:

--gpus all

Follow the logs:

docker logs -f ollama

Pull a model

Linux
Docker

ollama pull llama3.1:8b

List the available models:

ollama list

docker exec ollama \
  ollama pull llama3.1:8b

List the available models:

docker exec ollama \
  ollama list

Check that the Ollama server is ready:

curl http://localhost:11434/api/tags \
  | jq .

The response should contain the model you pulled.

Prepare the Hivenet Router agent

Docker
Bare metal

From the Hivenet Router repository, build the agent image if you have not already done so:

docker build \
  -f Dockerfile.agent \
  -t hivenet-router/agent:latest \
  .

Create a directory for the shared secret and persistent agent identity:

sudo mkdir -p /opt/hivenet-router
sudo chown "$USER":"$USER" /opt/hivenet-router

Place the router’s JWT secret at:

/opt/hivenet-router/jwt.secret

Protect it:

chmod 600 /opt/hivenet-router/jwt.secret

Build the agent from the Hivenet Router repository:

go build -o bin/hivenet-agent ./cmd/agent/

Place the router’s JWT secret somewhere the agent can read, for example:

/opt/hivenet-router/jwt.secret

Create a writable directory for the persistent agent identity:

sudo mkdir -p /var/lib/hivenet-router/agent
sudo chown "$USER":"$USER" /var/lib/hivenet-router/agent

Start the agent

The examples pin the agent to llama3.1:8b. This avoids ambiguity when more than one model is available in Ollama.

Docker
Bare metal

docker run -d \
  --name hivenet-agent \
  --restart unless-stopped \
  --network host \
  --gpus all \
  -v /opt/hivenet-router/jwt.secret:/jwt.secret:ro \
  -v /opt/hivenet-router:/data \
  hivenet-router/agent:latest \
  --engine ollama \
  --model llama3.1:8b \
  --backend-url http://localhost:11434 \
  --router-grpc 192.168.1.100:50051 \
  --jwt-secret-file /jwt.secret \
  --capacity 5 \
  --region EU-France \
  --identity-path /data/agent-identity.key

The agent container receives GPU access so it can collect NVIDIA hardware metrics. For a CPU-only host, omit:

--gpus all

The /data mount preserves the agent’s peer identity across container replacement and restarts.

./bin/hivenet-agent \
  --engine ollama \
  --model llama3.1:8b \
  --backend-url http://localhost:11434 \
  --router-grpc 192.168.1.100:50051 \
  --jwt-secret-file /opt/hivenet-router/jwt.secret \
  --capacity 5 \
  --region EU-France \
  --identity-path /var/lib/hivenet-router/agent/identity.key

The agent waits until Ollama responds and the selected model is available. If Ollama is still starting or loading its model list, the agent continues polling instead of exiting permanently.

Understand the agent settings

Setting	Purpose
`--engine ollama`	Selects the Ollama integration
`--model`	Pins the agent to one Ollama model
`--backend-url`	Base URL of the local Ollama server
`--router-grpc`	Router endpoint used for agent authentication
`--jwt-secret-file`	Shared secret used to authenticate the agent
`--capacity`	Maximum concurrent requests Hivenet Router may assign
`--region`	Region metadata available to routing policies
`--identity-path`	Persistent private key that keeps the peer ID stable

You can provide more routing metadata:

--organization edge-team \
--machine office-gpu-1 \
--tags local,ollama,production

Policies can match these values when selecting agents.

Model discovery and naming

If you omit --model, the agent requests:

GET <backend-url>/api/tags

It registers the first model returned by Ollama. Automatic discovery is convenient when the Ollama server contains one model. Pin the model explicitly when more than one is available.

The `:latest` suffix

Ollama adds :latest when a model has no explicit tag. Hivenet Router removes that implicit suffix during discovery.

Ollama model name	Registered with Hivenet Router
`llama3.1:latest`	`llama3.1`
`llama3.1:8b`	`llama3.1:8b`
`mistral:7b`	`mistral:7b`

Explicit tags remain unchanged. When using --model, provide the model name expected by Ollama:

--model llama3.1:8b

The model name clients send to Hivenet Router must match the model registered by the agent.

Run several Ollama models

Run one agent for each model you want Hivenet Router to expose. For the first model:

./bin/hivenet-agent \
  --engine ollama \
  --model llama3.1:8b \
  --backend-url http://localhost:11434 \
  --identity-path /var/lib/hivenet-router/agent/llama-8b.key \
  ...

For another model:

./bin/hivenet-agent \
  --engine ollama \
  --model mistral:7b \
  --backend-url http://localhost:11434 \
  --identity-path /var/lib/hivenet-router/agent/mistral-7b.key \
  ...

Each agent on the same host needs a separate --identity-path so every process keeps a distinct peer identity. The agents can share the same Ollama backend URL.

Set agent capacity

--capacity controls how many concurrent requests Hivenet Router may assign to the agent. It does not change Ollama’s own parallelism or memory settings. A conservative starting point is:

Host type	Starting capacity
GPU inference	`3` to `5`
CPU-only inference	`1` to `2`

These values are starting points, not fixed limits. The right setting depends on:

model size
available RAM or VRAM
context length
Ollama’s parallelism configuration
acceptable response time
whether several models share the same hardware

Test the backend under representative load before raising capacity. When the agent reaches capacity, Hivenet Router stops assigning new requests to it. Depending on your router configuration, requests may wait in the per-model queue, move through a fallback chain, or fail when no capacity is available.

For streaming responses, Hivenet Router releases the agent capacity slot when response headers arrive, while backend generation can continue. Treat --capacity as a routing-admission setting rather than a hard limit on ongoing streams, and verify Ollama concurrency under sustained streaming load.

Verify the connection

On the router, check agent health:

curl http://192.168.1.100:8080/admin/health \
  | jq .

List the models exposed to clients:

curl http://192.168.1.100:8080/v1/models \
  | jq '.data[].id'

Inspect the Ollama agent:

curl http://192.168.1.100:8080/admin/routing-table \
  | jq '.agents[] | select(.metadata.engine == "ollama")'

Send a chat-completion request:

curl -X POST http://192.168.1.100:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3.1:8b",
    "messages": [
      {
        "role": "user",
        "content": "Explain what an inference router does in one sentence."
      }
    ]
  }'

Applications call Hivenet Router’s OpenAI-compatible endpoint. The agent forwards the request to the Ollama backend.

Available observability

The Ollama integration does not scrape an engine-specific Prometheus endpoint. Hivenet Router can still expose:

agent health
routed request counts
failed request counts
active request and capacity state
smoothed round-trip time
success and failure history
CPU and memory metrics
NVIDIA GPU utilization, memory, temperature, and power when NVML is available

View metrics through the router:

curl http://192.168.1.100:2112/metrics \
  | grep hivenet-router

View current agent state:

curl http://192.168.1.100:8080/admin/routing-table \
  | jq '.agents[] | select(.metadata.engine == "ollama")'

Engine-specific values such as KV cache utilization, waiting requests, time to first token, and inter-token latency are not available from the Ollama integration.

Troubleshooting

No model is found

List the pulled models:

Linux
Docker

ollama list

docker exec ollama \
  ollama list

Pull the missing model:

ollama pull llama3.1:8b

Check the endpoint used by the agent:

curl http://localhost:11434/api/tags \
  | jq .

The wrong model is registered

When --model is omitted, the agent registers the first model returned by /api/tags. Restart the agent with an explicit model:

--model llama3.1:8b

A model appears without `:latest`

This is expected. Hivenet Router converts:

llama3.1:latest

to:

llama3.1

Explicit tags such as :8b or :7b are preserved.

Ollama is unreachable

Check the server:

curl http://localhost:11434/api/tags

For a Linux process, inspect its log:

tail -f ~/ollama.log

For Docker:

docker logs ollama

Confirm that the agent’s --backend-url matches the address where Ollama is listening.

The first response is slow

Ollama may need to load the model into memory before serving the first request. Check loaded models:

Linux
Docker

ollama ps

docker exec ollama \
  ollama ps

Using:

OLLAMA_KEEP_ALIVE=-1

keeps the model loaded after it has been used.

The agent does not register

Check the agent logs:

Docker
systemd

docker logs hivenet-agent

sudo journalctl \
  -u hivenet-agent \
  -n 100 \
  --no-pager

Test connectivity from the agent to the router:

nc -zv 192.168.1.100 50051
nc -zv 192.168.1.100 9000

Check that:

the router and agent use the same JWT secret
the router’s libp2p interface is reachable
the router advertises a libp2p address the agent can reach
Ollama contains the selected model

The agent gets a new peer ID after restart

Make sure --identity-path points to persistent storage. For Docker:

docker inspect hivenet-agent \
  | jq '.[0].Mounts'

For bare metal:

ls -l /var/lib/hivenet-router/agent/identity.key

GPU metrics are missing

Check that NVIDIA tools work on the host:

nvidia-smi

For Docker, confirm the agent received GPU access:

docker inspect hivenet-agent \
  | jq '.[0].HostConfig.DeviceRequests'

The agent continues to operate without NVML. It reports CPU and memory metrics but omits GPU measurements.

Requests time out

The agent’s backend HTTP timeout defaults to two minutes. Increase it for long-running requests:

--http-timeout 10m

Also check whether the agent capacity is too high for the model and hardware.

Next steps

SGLang agent

Connect an SGLang backend and collect engine metrics.

Routing concepts

Learn how metadata, capacity, and policies affect agent selection.

Hardware metrics

Understand the CPU, memory, and GPU signals reported by agents.

Getting started

Deploy Hivenet Router

Use the API

Routing and policies

Security and auth

Observability

Integrations

Reference

Project

Before you start

Start Ollama

Pull a model

Prepare the Hivenet Router agent

Start the agent

Understand the agent settings

Model discovery and naming

The `:latest` suffix

Run several Ollama models

Set agent capacity

Verify the connection

Available observability

Troubleshooting

No model is found

The wrong model is registered

A model appears without `:latest`

Ollama is unreachable

The first response is slow

The agent does not register

The agent gets a new peer ID after restart

GPU metrics are missing

Requests time out

Next steps

SGLang agent

Routing concepts

Hardware metrics

​Before you start

​Start Ollama

​Pull a model

​Prepare the Hivenet Router agent

​Start the agent

​Understand the agent settings

​Model discovery and naming

​The :latest suffix

​Run several Ollama models

​Set agent capacity

​Verify the connection

​Available observability

​Troubleshooting

​No model is found

​The wrong model is registered

​A model appears without :latest

​Ollama is unreachable

​The first response is slow

​The agent does not register

​The agent gets a new peer ID after restart

​GPU metrics are missing

​Requests time out

​Next steps

SGLang agent

Routing concepts

Hardware metrics

Before you start

Start Ollama

Pull a model

Prepare the Hivenet Router agent

Start the agent

Understand the agent settings

Model discovery and naming

The `:latest` suffix

Run several Ollama models

Set agent capacity

Verify the connection

Available observability

Troubleshooting

No model is found

The wrong model is registered

A model appears without `:latest`

Ollama is unreachable

The first response is slow

The agent does not register

The agent gets a new peer ID after restart

GPU metrics are missing

Requests time out

Next steps