> ## Documentation Index
> Fetch the complete documentation index at: https://routerdocs.hivenet.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Ollama agent
> Connect an Ollama backend to Hivenet Router, select the model an agent represents, and configure capacity for local or edge inference.
Connect an Ollama server to Hivenet Router by running an agent beside the backend.
The agent checks Ollama through its `/api/tags` endpoint, registers one available model with the router, and forwards chat-completion requests to Ollama’s OpenAI-compatible API.
One Hivenet Router agent represents one model. If your Ollama server contains several models, pin each agent to a model with `--model` and run a separate agent process for each one.
Ollama does not provide the engine metrics collected from vLLM, SGLang, or metrics-enabled llama.cpp servers. Hivenet Router still records routing, request, latency, health, and available hardware metrics for the agent.
## Before you start
You need:
* a running Hivenet Router router
* Ollama on the agent host
* at least one model pulled in Ollama
* network access between the agent and router
* the Hivenet Router agent binary or Docker image
* the same JWT secret used by the router
The examples use:
```text theme={null}
Router IP: 192.168.1.100
Router gRPC port: 50051
Router libp2p port: 9000
Agent IP: 192.168.1.101
Ollama port: 11434
Model: llama3.1:8b
```
Replace these values with addresses and model names from your deployment.
## Start Ollama
Install Ollama:
```bash theme={null}
curl -fsSL https://ollama.com/install.sh | sh
```
Start the server:
```bash theme={null}
export OLLAMA_HOST=127.0.0.1:11434
export OLLAMA_KEEP_ALIVE=-1
nohup ollama serve > ~/ollama.log 2>&1 &
```
`OLLAMA_KEEP_ALIVE=-1` keeps loaded models in memory instead of unloading them after an idle period.
To store models in another directory, set `OLLAMA_MODELS` before starting the server:
```bash theme={null}
export OLLAMA_MODELS=/path/to/models
```
The agent runs on the same host, so Ollama does not need to listen on a public interface. Bind it to `0.0.0.0` only when another machine must connect directly and your network controls restrict access appropriately.
Start Ollama with persistent model storage:
```bash theme={null}
docker run -d \
--name ollama \
--restart unless-stopped \
--network host \
--gpus all \
-e OLLAMA_KEEP_ALIVE=-1 \
-v ollama:/root/.ollama \
ollama/ollama
```
For a CPU-only host, omit:
```text theme={null}
--gpus all
```
Follow the logs:
```bash theme={null}
docker logs -f ollama
```
## Pull a model
```bash theme={null}
ollama pull llama3.1:8b
```
List the available models:
```bash theme={null}
ollama list
```
```bash theme={null}
docker exec ollama \
ollama pull llama3.1:8b
```
List the available models:
```bash theme={null}
docker exec ollama \
ollama list
```
Check that the Ollama server is ready:
```bash theme={null}
curl http://localhost:11434/api/tags \
| jq .
```
The response should contain the model you pulled.
## Prepare the Hivenet Router agent
From the Hivenet Router repository, build the agent image if you have not already done so:
```bash theme={null}
docker build \
-f Dockerfile.agent \
-t hivenet-router/agent:latest \
.
```
Create a directory for the shared secret and persistent agent identity:
```bash theme={null}
sudo mkdir -p /opt/hivenet-router
sudo chown "$USER":"$USER" /opt/hivenet-router
```
Place the router’s JWT secret at:
```text theme={null}
/opt/hivenet-router/jwt.secret
```
Protect it:
```bash theme={null}
chmod 600 /opt/hivenet-router/jwt.secret
```
Build the agent from the Hivenet Router repository:
```bash theme={null}
go build -o bin/hivenet-agent ./cmd/agent/
```
Place the router’s JWT secret somewhere the agent can read, for example:
```text theme={null}
/opt/hivenet-router/jwt.secret
```
Create a writable directory for the persistent agent identity:
```bash theme={null}
sudo mkdir -p /var/lib/hivenet-router/agent
sudo chown "$USER":"$USER" /var/lib/hivenet-router/agent
```
## Start the agent
The examples pin the agent to `llama3.1:8b`. This avoids ambiguity when more than one model is available in Ollama.
```bash theme={null}
docker run -d \
--name hivenet-agent \
--restart unless-stopped \
--network host \
--gpus all \
-v /opt/hivenet-router/jwt.secret:/jwt.secret:ro \
-v /opt/hivenet-router:/data \
hivenet-router/agent:latest \
--engine ollama \
--model llama3.1:8b \
--backend-url http://localhost:11434 \
--router-grpc 192.168.1.100:50051 \
--jwt-secret-file /jwt.secret \
--capacity 5 \
--region EU-France \
--identity-path /data/agent-identity.key
```
The agent container receives GPU access so it can collect NVIDIA hardware metrics. For a CPU-only host, omit:
```text theme={null}
--gpus all
```
The `/data` mount preserves the agent’s peer identity across container replacement and restarts.
```bash theme={null}
./bin/hivenet-agent \
--engine ollama \
--model llama3.1:8b \
--backend-url http://localhost:11434 \
--router-grpc 192.168.1.100:50051 \
--jwt-secret-file /opt/hivenet-router/jwt.secret \
--capacity 5 \
--region EU-France \
--identity-path /var/lib/hivenet-router/agent/identity.key
```
The agent waits until Ollama responds and the selected model is available. If Ollama is still starting or loading its model list, the agent continues polling instead of exiting permanently.
## Understand the agent settings
| Setting | Purpose |
| ------------------- | ----------------------------------------------------- |
| `--engine ollama` | Selects the Ollama integration |
| `--model` | Pins the agent to one Ollama model |
| `--backend-url` | Base URL of the local Ollama server |
| `--router-grpc` | Router endpoint used for agent authentication |
| `--jwt-secret-file` | Shared secret used to authenticate the agent |
| `--capacity` | Maximum concurrent requests Hivenet Router may assign |
| `--region` | Region metadata available to routing policies |
| `--identity-path` | Persistent private key that keeps the peer ID stable |
You can provide more routing metadata:
```bash theme={null}
--organization edge-team \
--machine office-gpu-1 \
--tags local,ollama,production
```
Policies can match these values when selecting agents.
## Model discovery and naming
If you omit `--model`, the agent requests:
```text theme={null}
GET /api/tags
```
It registers the first model returned by Ollama.
Automatic discovery is convenient when the Ollama server contains one model. Pin the model explicitly when more than one is available.
### The `:latest` suffix
Ollama adds `:latest` when a model has no explicit tag. Hivenet Router removes that implicit suffix during discovery.
| Ollama model name | Registered with Hivenet Router |
| ----------------- | ------------------------------ |
| `llama3.1:latest` | `llama3.1` |
| `llama3.1:8b` | `llama3.1:8b` |
| `mistral:7b` | `mistral:7b` |
Explicit tags remain unchanged.
When using `--model`, provide the model name expected by Ollama:
```bash theme={null}
--model llama3.1:8b
```
The model name clients send to Hivenet Router must match the model registered by the agent.
## Run several Ollama models
Run one agent for each model you want Hivenet Router to expose.
For the first model:
```bash theme={null}
./bin/hivenet-agent \
--engine ollama \
--model llama3.1:8b \
--backend-url http://localhost:11434 \
--identity-path /var/lib/hivenet-router/agent/llama-8b.key \
...
```
For another model:
```bash theme={null}
./bin/hivenet-agent \
--engine ollama \
--model mistral:7b \
--backend-url http://localhost:11434 \
--identity-path /var/lib/hivenet-router/agent/mistral-7b.key \
...
```
Each agent on the same host needs a separate `--identity-path` so every process keeps a distinct peer identity.
The agents can share the same Ollama backend URL.
## Set agent capacity
`--capacity` controls how many concurrent requests Hivenet Router may assign to the agent.
It does not change Ollama’s own parallelism or memory settings.
A conservative starting point is:
| Host type | Starting capacity |
| ------------------ | ----------------- |
| GPU inference | `3` to `5` |
| CPU-only inference | `1` to `2` |
These values are starting points, not fixed limits. The right setting depends on:
* model size
* available RAM or VRAM
* context length
* Ollama’s parallelism configuration
* acceptable response time
* whether several models share the same hardware
Test the backend under representative load before raising capacity.
When the agent reaches capacity, Hivenet Router stops assigning new requests to it. Depending on your router configuration, requests may wait in the per-model queue, move through a fallback chain, or fail when no capacity is available.
For streaming responses, Hivenet Router releases the agent capacity slot when response headers arrive, while backend generation can continue. Treat `--capacity` as a routing-admission setting rather than a hard limit on ongoing streams, and verify Ollama concurrency under sustained streaming load.
## Verify the connection
On the router, check agent health:
```bash theme={null}
curl http://192.168.1.100:8080/admin/health \
| jq .
```
List the models exposed to clients:
```bash theme={null}
curl http://192.168.1.100:8080/v1/models \
| jq '.data[].id'
```
Inspect the Ollama agent:
```bash theme={null}
curl http://192.168.1.100:8080/admin/routing-table \
| jq '.agents[] | select(.metadata.engine == "ollama")'
```
Send a chat-completion request:
```bash theme={null}
curl -X POST http://192.168.1.100:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "llama3.1:8b",
"messages": [
{
"role": "user",
"content": "Explain what an inference router does in one sentence."
}
]
}'
```
Applications call Hivenet Router’s OpenAI-compatible endpoint. The agent forwards the request to the Ollama backend.
## Available observability
The Ollama integration does not scrape an engine-specific Prometheus endpoint.
Hivenet Router can still expose:
* agent health
* routed request counts
* failed request counts
* active request and capacity state
* smoothed round-trip time
* success and failure history
* CPU and memory metrics
* NVIDIA GPU utilization, memory, temperature, and power when NVML is available
View metrics through the router:
```bash theme={null}
curl http://192.168.1.100:2112/metrics \
| grep hivenet-router
```
View current agent state:
```bash theme={null}
curl http://192.168.1.100:8080/admin/routing-table \
| jq '.agents[] | select(.metadata.engine == "ollama")'
```
Engine-specific values such as KV cache utilization, waiting requests, time to first token, and inter-token latency are not available from the Ollama integration.
## Troubleshooting
### No model is found
List the pulled models:
```bash theme={null}
ollama list
```
```bash theme={null}
docker exec ollama \
ollama list
```
Pull the missing model:
```bash theme={null}
ollama pull llama3.1:8b
```
Check the endpoint used by the agent:
```bash theme={null}
curl http://localhost:11434/api/tags \
| jq .
```
### The wrong model is registered
When `--model` is omitted, the agent registers the first model returned by `/api/tags`.
Restart the agent with an explicit model:
```bash theme={null}
--model llama3.1:8b
```
### A model appears without `:latest`
This is expected.
Hivenet Router converts:
```text theme={null}
llama3.1:latest
```
to:
```text theme={null}
llama3.1
```
Explicit tags such as `:8b` or `:7b` are preserved.
### Ollama is unreachable
Check the server:
```bash theme={null}
curl http://localhost:11434/api/tags
```
For a Linux process, inspect its log:
```bash theme={null}
tail -f ~/ollama.log
```
For Docker:
```bash theme={null}
docker logs ollama
```
Confirm that the agent’s `--backend-url` matches the address where Ollama is listening.
### The first response is slow
Ollama may need to load the model into memory before serving the first request.
Check loaded models:
```bash theme={null}
ollama ps
```
```bash theme={null}
docker exec ollama \
ollama ps
```
Using:
```bash theme={null}
OLLAMA_KEEP_ALIVE=-1
```
keeps the model loaded after it has been used.
### The agent does not register
Check the agent logs:
```bash theme={null}
docker logs hivenet-agent
```
```bash theme={null}
sudo journalctl \
-u hivenet-agent \
-n 100 \
--no-pager
```
Test connectivity from the agent to the router:
```bash theme={null}
nc -zv 192.168.1.100 50051
nc -zv 192.168.1.100 9000
```
Check that:
* the router and agent use the same JWT secret
* the router’s libp2p interface is reachable
* the router advertises a libp2p address the agent can reach
* Ollama contains the selected model
### The agent gets a new peer ID after restart
Make sure `--identity-path` points to persistent storage.
For Docker:
```bash theme={null}
docker inspect hivenet-agent \
| jq '.[0].Mounts'
```
For bare metal:
```bash theme={null}
ls -l /var/lib/hivenet-router/agent/identity.key
```
### GPU metrics are missing
Check that NVIDIA tools work on the host:
```bash theme={null}
nvidia-smi
```
For Docker, confirm the agent received GPU access:
```bash theme={null}
docker inspect hivenet-agent \
| jq '.[0].HostConfig.DeviceRequests'
```
The agent continues to operate without NVML. It reports CPU and memory metrics but omits GPU measurements.
### Requests time out
The agent’s backend HTTP timeout defaults to two minutes.
Increase it for long-running requests:
```bash theme={null}
--http-timeout 10m
```
Also check whether the agent capacity is too high for the model and hardware.
## Next steps
Connect an SGLang backend and collect engine metrics.
Learn how metadata, capacity, and policies affect agent selection.
Understand the CPU, memory, and GPU signals reported by agents.