> ## Documentation Index
> Fetch the complete documentation index at: https://routerdocs.hivenet.com/llms.txt
> Use this file to discover all available pages before exploring further.
# llama.cpp agent
> Connect a llama.cpp server to Hivenet Router, set a stable model name, and expose cache, latency, and throughput metrics.
Connect a llama.cpp server to Hivenet Router by running an agent beside the backend.
The agent discovers the model served by llama.cpp, registers it with the router, forwards OpenAI-compatible chat-completion requests, and reports engine and hardware metrics.
A llama.cpp server runs one model per process, and each Hivenet Router agent registers one model. Run a separate server and agent pair for each model you want to expose.
## Before you start
You need:
* a running Hivenet Router router
* a GGUF model file
* llama.cpp built locally or available as a Docker image
* network access between the agent host and router
* the Hivenet Router agent binary or Docker image
* the same JWT secret used by the router
* an NVIDIA GPU if you want CUDA acceleration and GPU metrics
The examples use:
```text theme={null}
Router IP: 192.168.1.100
Router gRPC port: 50051
Router libp2p port: 9000
Agent IP: 192.168.1.101
llama.cpp port: 8888
Model alias: llama-3.1-8b
```
Replace these values with addresses and model names from your deployment.
## Start llama.cpp
Use the `-a` option to give the model a stable alias. The alias becomes the model name registered with Hivenet Router and used by API clients.
Start llama.cpp with `--metrics` if you want Hivenet Router to collect cache, queue, latency, and throughput metrics.
Place the GGUF file in a local model directory, such as:
```text theme={null}
/data/models
```
Start the server:
```bash theme={null}
docker run -d \
--name llama-cpp \
--restart unless-stopped \
--gpus all \
--network host \
-v /data/models:/models:ro \
ghcr.io/ggerganov/llama.cpp:server \
-m /models/llama-3.1-8b-instruct.Q4_K_M.gguf \
-a "llama-3.1-8b" \
--host 0.0.0.0 \
--port 8888 \
--metrics
```
Follow the startup logs:
```bash theme={null}
docker logs -f llama-cpp
```
For CPU-only inference, omit:
```text theme={null}
--gpus all
```
Clone llama.cpp:
```bash theme={null}
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
```
Build with CUDA support:
```bash theme={null}
cmake -B build \
-DGGML_CUDA=ON
cmake --build build \
--config Release \
-j
```
Start the server:
```bash theme={null}
./build/bin/llama-server \
-m /path/to/llama-3.1-8b-instruct.Q4_K_M.gguf \
-a "llama-3.1-8b" \
--host 0.0.0.0 \
--port 8888 \
-ngl 99 \
--flash-attn \
--metrics
```
For CPU-only inference, build without `-DGGML_CUDA=ON` and omit the GPU-specific launch options.
Wait for the model to load, then check the server:
```bash theme={null}
curl -i http://localhost:8888/health
```
A ready llama.cpp server returns HTTP status `200`.
List the model exposed by its OpenAI-compatible API:
```bash theme={null}
curl http://localhost:8888/v1/models \
| jq '.data[].id'
```
## Choose a stable model name
The alias passed with `-a` determines the model name returned by llama.cpp.
| llama.cpp launch options | Model registered with Hivenet Router |
| ------------------------------------------ | ------------------------------------ |
| `-m model.gguf -a "my-model"` | `my-model` |
| `-m llama-3.1-8b.Q4_K_M.gguf` without `-a` | `llama-3.1-8b.Q4_K_M` |
Using an explicit alias makes client configuration and routing policies less dependent on the model filename.
Changing the alias changes the model name seen by Hivenet Router. Clients and routing policies must use the new value.
## Prepare the Hivenet Router agent
From the Hivenet Router repository, build the agent image if you have not already done so:
```bash theme={null}
docker build \
-f Dockerfile.agent \
-t hivenet-router/agent:latest \
.
```
Create a directory for the shared secret and persistent identity:
```bash theme={null}
sudo mkdir -p /opt/hivenet-router
sudo chown "$USER":"$USER" /opt/hivenet-router
```
Place the router’s JWT secret at:
```text theme={null}
/opt/hivenet-router/jwt.secret
```
Protect it:
```bash theme={null}
chmod 600 /opt/hivenet-router/jwt.secret
```
Build the agent from the Hivenet Router repository:
```bash theme={null}
go build -o bin/hivenet-agent ./cmd/agent/
```
Place the shared JWT secret somewhere the agent can read, for example:
```text theme={null}
/opt/hivenet-router/jwt.secret
```
Create a writable directory for the agent identity:
```bash theme={null}
sudo mkdir -p /var/lib/hivenet-router/agent
sudo chown "$USER":"$USER" /var/lib/hivenet-router/agent
```
## Start the agent
```bash theme={null}
docker run -d \
--name hivenet-agent \
--restart unless-stopped \
--network host \
--gpus all \
-v /opt/hivenet-router/jwt.secret:/jwt.secret:ro \
-v /opt/hivenet-router:/data \
hivenet-router/agent:latest \
--engine llamacpp \
--backend-url http://localhost:8888 \
--router-grpc 192.168.1.100:50051 \
--jwt-secret-file /jwt.secret \
--capacity 5 \
--region EU-France \
--organization ml-team \
--machine gpu-worker-1 \
--identity-path /data/agent-identity.key
```
For CPU-only inference, omit:
```text theme={null}
--gpus all
```
The `/data` mount preserves the agent’s peer identity across container replacement and restarts.
```bash theme={null}
./bin/hivenet-agent \
--engine llamacpp \
--backend-url http://localhost:8888 \
--router-grpc 192.168.1.100:50051 \
--jwt-secret-file /opt/hivenet-router/jwt.secret \
--capacity 5 \
--region EU-France \
--organization ml-team \
--machine gpu-worker-1 \
--identity-path /var/lib/hivenet-router/agent/identity.key
```
The agent waits until llama.cpp is healthy and exposes a model before registering with the router. If the backend is still loading, the agent continues polling rather than exiting permanently.
## Understand the agent settings
| Setting | Purpose |
| ------------------- | ----------------------------------------------------- |
| `--engine llamacpp` | Selects the llama.cpp backend integration |
| `--backend-url` | Base URL of the local llama.cpp server |
| `--router-grpc` | Router endpoint used for agent authentication |
| `--jwt-secret-file` | Shared secret used to authenticate the agent |
| `--capacity` | Maximum concurrent requests Hivenet Router may assign |
| `--region` | Region metadata available to routing policies |
| `--organization` | Organization or infrastructure-provider metadata |
| `--machine` | Stable machine identifier |
| `--identity-path` | Persistent private key that keeps the peer ID stable |
You can also add routing tags:
```bash theme={null}
--tags production,llama-cpp,gguf
```
## Model discovery
When `--model` is omitted, the agent requests:
```text theme={null}
GET /v1/models
```
It registers the first model returned by llama.cpp.
Because a llama.cpp server normally serves one model, automatic discovery is usually sufficient.
You can also pin the model explicitly:
```bash theme={null}
--model llama-3.1-8b
```
The value must match the model alias returned by llama.cpp.
## Set agent capacity
`--capacity` limits how many concurrent requests Hivenet Router may assign to the agent.
It does not change llama.cpp’s own parallelism, slot count, context allocation, or batch configuration.
Conservative starting values from the original deployment guidance are:
| Workload | Starting capacity |
| ------------------ | ----------------- |
| GPU inference | `5` to `10` |
| CPU-only inference | `2` to `5` |
Treat these as starting points rather than universal limits. Test with your model, quantization, context size, and hardware.
When the agent reaches its declared capacity, Hivenet Router considers another matching agent or a configured fallback step.
For streaming responses, Hivenet Router releases the agent capacity slot when response headers arrive, while backend generation can continue. Treat `--capacity` as a routing-admission setting rather than a hard limit on ongoing streams, and verify llama.cpp concurrency under sustained streaming load.
## Verify the connection
On the router, check agent health:
```bash theme={null}
curl http://192.168.1.100:8080/admin/health \
| jq .
```
List the models available to clients:
```bash theme={null}
curl http://192.168.1.100:8080/v1/models \
| jq '.data[].id'
```
Inspect llama.cpp agents:
```bash theme={null}
curl http://192.168.1.100:8080/admin/routing-table \
| jq '.agents[] | select(.metadata.engine == "llamacpp")'
```
Send a chat-completion request:
```bash theme={null}
curl -X POST http://192.168.1.100:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "llama-3.1-8b",
"messages": [
{
"role": "user",
"content": "Explain what a GGUF model is in one sentence."
}
]
}'
```
The model in the request must match the alias registered by the agent.
## llama.cpp metrics
The agent scrapes llama.cpp’s `/metrics` endpoint every 500 milliseconds by default.
Start llama.cpp with `--metrics` to make this data available. Metrics are sent to the router and re-exported through its Prometheus endpoint. The agent does not expose a separate Prometheus port.
| Signal | llama.cpp source | Hivenet Router field |
| --------------------- | ---------------------------------------- | --------------------------- |
| KV cache utilization | `llamacpp:kv_cache_usage_ratio` | `kv_cache_utilization` |
| Running requests | `llamacpp:requests_processing` | `running_requests` |
| Waiting requests | `llamacpp:requests_deferred` | `waiting_requests` |
| Average and P90 TTFT | `llamacpp:time_to_first_token_seconds` | TTFT metrics |
| Average and P90 ITL | `llamacpp:time_per_output_token_seconds` | ITL metrics |
| Generation throughput | `llamacpp:predicted_tokens_seconds` | Predicted tokens per second |
| Prompt throughput | `llamacpp:prompt_tokens_seconds` | Prompt tokens per second |
llama.cpp does not use the preemption mechanism measured for vLLM, so Hivenet Router does not populate a preemption metric for this engine.
View all engine metrics through the router:
```bash theme={null}
curl http://192.168.1.100:2112/metrics \
| grep hivenet_router_agent_engine
```
Useful exported metrics include:
```text theme={null}
hivenet_router_agent_engine_kv_cache_utilization
hivenet_router_agent_engine_running_requests
hivenet_router_agent_engine_waiting_requests
hivenet_router_agent_engine_avg_ttft_seconds
hivenet_router_agent_engine_p90_ttft_seconds
hivenet_router_agent_engine_avg_itl_seconds
hivenet_router_agent_engine_p90_itl_seconds
hivenet_router_agent_engine_predicted_tps
hivenet_router_agent_engine_prompt_tps
```
View current values through the routing table:
```bash theme={null}
curl http://192.168.1.100:8080/admin/routing-table \
| jq '.agents[]
| select(.metadata.engine == "llamacpp")
| {
peer_id,
model: .metadata.model,
kv_cache_utilization: .engine.kv_cache_utilization,
running_requests: .engine.running_requests,
waiting_requests: .engine.waiting_requests
}'
```
The routing table exposes scalar cache, queue, TTFT, and ITL values. llama.cpp prompt and generation throughput are available through Prometheus, not the current routing-table response.
## Use llama.cpp metrics in routing policies
Exclude agents whose KV cache is under pressure:
```yaml theme={null}
exclude_if:
kv_cache_utilization:
gt: 0.85
```
Exclude agents with queued requests:
```yaml theme={null}
exclude_if:
waiting_requests:
gt: 0
```
Exclude agents whose average time to first token is too high:
```yaml theme={null}
exclude_if:
avg_ttft_seconds:
gt: 2
```
When a metric is unavailable, Hivenet Router skips that gate for the agent rather than excluding it.
See [Policy YAML reference](/routing/policy-yaml-reference) for the full schema.
## Tune llama.cpp
### Reduce the context size
A smaller context window reduces KV-cache memory use:
```bash theme={null}
llama-server \
... \
--ctx-size 4096
```
### Change the quantization
Quantization affects model size, memory requirements, speed, and output quality.
The source deployment guide uses:
```text theme={null}
Q4_K_M
```
as a practical balance for its example. Choose a quantization appropriate for your model and hardware.
### Load layers on the GPU
For a CUDA build:
```bash theme={null}
-ngl 99
```
requests that llama.cpp offload model layers to the GPU.
The number of layers that fit depends on the model, quantization, and available VRAM.
### Enable flash attention
The example uses:
```bash theme={null}
--flash-attn
```
Support depends on the build and hardware. Remove it if the server fails to start with the option enabled.
## Troubleshooting
### llama.cpp is not ready
Check the health endpoint:
```bash theme={null}
curl -i http://localhost:8888/health
```
List the exposed model:
```bash theme={null}
curl http://localhost:8888/v1/models \
| jq .
```
Inspect the logs:
```bash theme={null}
docker logs llama-cpp
```
Review the terminal or service logs for `llama-server`.
The agent waits and retries while the backend is unavailable or still loading.
### Metrics are missing
Confirm that llama.cpp was started with:
```bash theme={null}
--metrics
```
Check the endpoint directly:
```bash theme={null}
curl http://localhost:8888/metrics \
| head
```
Without `--metrics`, the endpoint returns HTTP `404` and Hivenet Router cannot collect engine-specific metrics.
A metrics scrape failure does not stop the agent from forwarding requests.
### The model name does not match
Check the name returned by llama.cpp:
```bash theme={null}
curl http://localhost:8888/v1/models \
| jq '.data[].id'
```
Use `-a` to set a stable alias:
```bash theme={null}
-a "llama-3.1-8b"
```
Clients must send the same name to Hivenet Router.
### Responses are slow
Possible adjustments include:
* reduce `--capacity` on the Hivenet Router agent
* reduce llama.cpp’s context size
* use a smaller model or quantization
* increase GPU offloading
* reduce parallel request pressure
Measure with representative prompts before changing several settings at once.
### The agent does not register
Check the agent logs:
```bash theme={null}
docker logs hivenet-agent
```
```bash theme={null}
sudo journalctl \
-u hivenet-agent \
-n 100 \
--no-pager
```
Test connectivity from the agent to the router:
```bash theme={null}
nc -zv 192.168.1.100 50051
nc -zv 192.168.1.100 9000
```
Check that:
* the router and agent use the same JWT secret
* the router’s libp2p interface is reachable
* the router advertises a libp2p address the agent can reach
* llama.cpp is healthy and exposes a model
### The peer ID changes after restart
Make sure `--identity-path` points to persistent storage.
For Docker:
```bash theme={null}
docker inspect hivenet-agent \
| jq '.[0].Mounts'
```
For bare metal:
```bash theme={null}
ls -l /var/lib/hivenet-router/agent/identity.key
```
### Requests time out
The agent’s backend HTTP timeout defaults to two minutes.
Increase it for long-running requests:
```bash theme={null}
--http-timeout 10m
```
Also check the declared agent capacity and llama.cpp queue state.
### GPU metrics are missing
Check the host:
```bash theme={null}
nvidia-smi
```
For Docker, confirm the agent received GPU access:
```bash theme={null}
docker inspect hivenet-agent \
| jq '.[0].HostConfig.DeviceRequests'
```
The agent continues to operate without NVML. It reports CPU and memory metrics but omits GPU measurements.
## Next steps
Connect embedding and reranking models through Infinity.
Learn how Hivenet Router filters and ranks matching agents.
Understand the metrics collected from llama.cpp and other engines.