llama.cpp agent

Connect a llama.cpp server to Hivenet Router by running an agent beside the backend. The agent discovers the model served by llama.cpp, registers it with the router, forwards OpenAI-compatible chat-completion requests, and reports engine and hardware metrics.

A llama.cpp server runs one model per process, and each Hivenet Router agent registers one model. Run a separate server and agent pair for each model you want to expose.

Before you start

You need:

a running Hivenet Router router
a GGUF model file
llama.cpp built locally or available as a Docker image
network access between the agent host and router
the Hivenet Router agent binary or Docker image
the same JWT secret used by the router
an NVIDIA GPU if you want CUDA acceleration and GPU metrics

The examples use:

Router IP: 192.168.1.100
Router gRPC port: 50051
Router libp2p port: 9000
Agent IP: 192.168.1.101
llama.cpp port: 8888
Model alias: llama-3.1-8b

Replace these values with addresses and model names from your deployment.

Start llama.cpp

Use the -a option to give the model a stable alias. The alias becomes the model name registered with Hivenet Router and used by API clients. Start llama.cpp with --metrics if you want Hivenet Router to collect cache, queue, latency, and throughput metrics.

Docker
Bare metal

Place the GGUF file in a local model directory, such as:

/data/models

Start the server:

docker run -d \
  --name llama-cpp \
  --restart unless-stopped \
  --gpus all \
  --network host \
  -v /data/models:/models:ro \
  ghcr.io/ggerganov/llama.cpp:server \
  -m /models/llama-3.1-8b-instruct.Q4_K_M.gguf \
  -a "llama-3.1-8b" \
  --host 0.0.0.0 \
  --port 8888 \
  --metrics

Follow the startup logs:

docker logs -f llama-cpp

For CPU-only inference, omit:

--gpus all

Clone llama.cpp:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp

Build with CUDA support:

cmake -B build \
  -DGGML_CUDA=ON

cmake --build build \
  --config Release \
  -j

Start the server:

./build/bin/llama-server \
  -m /path/to/llama-3.1-8b-instruct.Q4_K_M.gguf \
  -a "llama-3.1-8b" \
  --host 0.0.0.0 \
  --port 8888 \
  -ngl 99 \
  --flash-attn \
  --metrics

For CPU-only inference, build without -DGGML_CUDA=ON and omit the GPU-specific launch options.

Wait for the model to load, then check the server:

curl -i http://localhost:8888/health

A ready llama.cpp server returns HTTP status 200. List the model exposed by its OpenAI-compatible API:

curl http://localhost:8888/v1/models \
  | jq '.data[].id'

Choose a stable model name

The alias passed with -a determines the model name returned by llama.cpp.

llama.cpp launch options	Model registered with Hivenet Router
`-m model.gguf -a "my-model"`	`my-model`
`-m llama-3.1-8b.Q4_K_M.gguf` without `-a`	`llama-3.1-8b.Q4_K_M`

Using an explicit alias makes client configuration and routing policies less dependent on the model filename.

Changing the alias changes the model name seen by Hivenet Router. Clients and routing policies must use the new value.

Prepare the Hivenet Router agent

Docker
Bare metal

From the Hivenet Router repository, build the agent image if you have not already done so:

docker build \
  -f Dockerfile.agent \
  -t hivenet-router/agent:latest \
  .

Create a directory for the shared secret and persistent identity:

sudo mkdir -p /opt/hivenet-router
sudo chown "$USER":"$USER" /opt/hivenet-router

Place the router’s JWT secret at:

/opt/hivenet-router/jwt.secret

Protect it:

chmod 600 /opt/hivenet-router/jwt.secret

Build the agent from the Hivenet Router repository:

go build -o bin/hivenet-agent ./cmd/agent/

Place the shared JWT secret somewhere the agent can read, for example:

/opt/hivenet-router/jwt.secret

Create a writable directory for the agent identity:

sudo mkdir -p /var/lib/hivenet-router/agent
sudo chown "$USER":"$USER" /var/lib/hivenet-router/agent

Start the agent

Docker
Bare metal

docker run -d \
  --name hivenet-agent \
  --restart unless-stopped \
  --network host \
  --gpus all \
  -v /opt/hivenet-router/jwt.secret:/jwt.secret:ro \
  -v /opt/hivenet-router:/data \
  hivenet-router/agent:latest \
  --engine llamacpp \
  --backend-url http://localhost:8888 \
  --router-grpc 192.168.1.100:50051 \
  --jwt-secret-file /jwt.secret \
  --capacity 5 \
  --region EU-France \
  --organization ml-team \
  --machine gpu-worker-1 \
  --identity-path /data/agent-identity.key

For CPU-only inference, omit:

--gpus all

The /data mount preserves the agent’s peer identity across container replacement and restarts.

./bin/hivenet-agent \
  --engine llamacpp \
  --backend-url http://localhost:8888 \
  --router-grpc 192.168.1.100:50051 \
  --jwt-secret-file /opt/hivenet-router/jwt.secret \
  --capacity 5 \
  --region EU-France \
  --organization ml-team \
  --machine gpu-worker-1 \
  --identity-path /var/lib/hivenet-router/agent/identity.key

The agent waits until llama.cpp is healthy and exposes a model before registering with the router. If the backend is still loading, the agent continues polling rather than exiting permanently.

Understand the agent settings

Setting	Purpose
`--engine llamacpp`	Selects the llama.cpp backend integration
`--backend-url`	Base URL of the local llama.cpp server
`--router-grpc`	Router endpoint used for agent authentication
`--jwt-secret-file`	Shared secret used to authenticate the agent
`--capacity`	Maximum concurrent requests Hivenet Router may assign
`--region`	Region metadata available to routing policies
`--organization`	Organization or infrastructure-provider metadata
`--machine`	Stable machine identifier
`--identity-path`	Persistent private key that keeps the peer ID stable

You can also add routing tags:

--tags production,llama-cpp,gguf

Model discovery

When --model is omitted, the agent requests:

GET <backend-url>/v1/models

It registers the first model returned by llama.cpp. Because a llama.cpp server normally serves one model, automatic discovery is usually sufficient. You can also pin the model explicitly:

--model llama-3.1-8b

The value must match the model alias returned by llama.cpp.

Set agent capacity

--capacity limits how many concurrent requests Hivenet Router may assign to the agent. It does not change llama.cpp’s own parallelism, slot count, context allocation, or batch configuration. Conservative starting values from the original deployment guidance are:

Workload	Starting capacity
GPU inference	`5` to `10`
CPU-only inference	`2` to `5`

Treat these as starting points rather than universal limits. Test with your model, quantization, context size, and hardware. When the agent reaches its declared capacity, Hivenet Router considers another matching agent or a configured fallback step.

For streaming responses, Hivenet Router releases the agent capacity slot when response headers arrive, while backend generation can continue. Treat --capacity as a routing-admission setting rather than a hard limit on ongoing streams, and verify llama.cpp concurrency under sustained streaming load.

Verify the connection

On the router, check agent health:

curl http://192.168.1.100:8080/admin/health \
  | jq .

List the models available to clients:

curl http://192.168.1.100:8080/v1/models \
  | jq '.data[].id'

Inspect llama.cpp agents:

curl http://192.168.1.100:8080/admin/routing-table \
  | jq '.agents[] | select(.metadata.engine == "llamacpp")'

Send a chat-completion request:

curl -X POST http://192.168.1.100:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-3.1-8b",
    "messages": [
      {
        "role": "user",
        "content": "Explain what a GGUF model is in one sentence."
      }
    ]
  }'

The model in the request must match the alias registered by the agent.

llama.cpp metrics

The agent scrapes llama.cpp’s /metrics endpoint every 500 milliseconds by default. Start llama.cpp with --metrics to make this data available. Metrics are sent to the router and re-exported through its Prometheus endpoint. The agent does not expose a separate Prometheus port.

Signal	llama.cpp source	Hivenet Router field
KV cache utilization	`llamacpp:kv_cache_usage_ratio`	`kv_cache_utilization`
Running requests	`llamacpp:requests_processing`	`running_requests`
Waiting requests	`llamacpp:requests_deferred`	`waiting_requests`
Average and P90 TTFT	`llamacpp:time_to_first_token_seconds`	TTFT metrics
Average and P90 ITL	`llamacpp:time_per_output_token_seconds`	ITL metrics
Generation throughput	`llamacpp:predicted_tokens_seconds`	Predicted tokens per second
Prompt throughput	`llamacpp:prompt_tokens_seconds`	Prompt tokens per second

llama.cpp does not use the preemption mechanism measured for vLLM, so Hivenet Router does not populate a preemption metric for this engine. View all engine metrics through the router:

curl http://192.168.1.100:2112/metrics \
  | grep hivenet_router_agent_engine

Useful exported metrics include:

hivenet_router_agent_engine_kv_cache_utilization
hivenet_router_agent_engine_running_requests
hivenet_router_agent_engine_waiting_requests
hivenet_router_agent_engine_avg_ttft_seconds
hivenet_router_agent_engine_p90_ttft_seconds
hivenet_router_agent_engine_avg_itl_seconds
hivenet_router_agent_engine_p90_itl_seconds
hivenet_router_agent_engine_predicted_tps
hivenet_router_agent_engine_prompt_tps

View current values through the routing table:

curl http://192.168.1.100:8080/admin/routing-table \
  | jq '.agents[]
      | select(.metadata.engine == "llamacpp")
      | {
          peer_id,
          model: .metadata.model,
          kv_cache_utilization: .engine.kv_cache_utilization,
          running_requests: .engine.running_requests,
          waiting_requests: .engine.waiting_requests
        }'

The routing table exposes scalar cache, queue, TTFT, and ITL values. llama.cpp prompt and generation throughput are available through Prometheus, not the current routing-table response.

Use llama.cpp metrics in routing policies

Exclude agents whose KV cache is under pressure:

exclude_if:
  kv_cache_utilization:
    gt: 0.85

Exclude agents with queued requests:

exclude_if:
  waiting_requests:
    gt: 0

Exclude agents whose average time to first token is too high:

exclude_if:
  avg_ttft_seconds:
    gt: 2

When a metric is unavailable, Hivenet Router skips that gate for the agent rather than excluding it. See Policy YAML reference for the full schema.

Tune llama.cpp

Reduce the context size

A smaller context window reduces KV-cache memory use:

llama-server \
  ... \
  --ctx-size 4096

Change the quantization

Quantization affects model size, memory requirements, speed, and output quality. The source deployment guide uses:

Q4_K_M

as a practical balance for its example. Choose a quantization appropriate for your model and hardware.

Load layers on the GPU

For a CUDA build:

-ngl 99

requests that llama.cpp offload model layers to the GPU. The number of layers that fit depends on the model, quantization, and available VRAM.

Enable flash attention

The example uses:

--flash-attn

Support depends on the build and hardware. Remove it if the server fails to start with the option enabled.

Troubleshooting

llama.cpp is not ready

Check the health endpoint:

curl -i http://localhost:8888/health

List the exposed model:

curl http://localhost:8888/v1/models \
  | jq .

Inspect the logs:

Docker
Bare metal

docker logs llama-cpp

Review the terminal or service logs for llama-server.

The agent waits and retries while the backend is unavailable or still loading.

Metrics are missing

Confirm that llama.cpp was started with:

--metrics

Check the endpoint directly:

curl http://localhost:8888/metrics \
  | head

Without --metrics, the endpoint returns HTTP 404 and Hivenet Router cannot collect engine-specific metrics. A metrics scrape failure does not stop the agent from forwarding requests.

The model name does not match

Check the name returned by llama.cpp:

curl http://localhost:8888/v1/models \
  | jq '.data[].id'

Use -a to set a stable alias:

-a "llama-3.1-8b"

Clients must send the same name to Hivenet Router.

Responses are slow

Possible adjustments include:

reduce --capacity on the Hivenet Router agent
reduce llama.cpp’s context size
use a smaller model or quantization
increase GPU offloading
reduce parallel request pressure

Measure with representative prompts before changing several settings at once.

The agent does not register

Check the agent logs:

Docker
systemd

docker logs hivenet-agent

sudo journalctl \
  -u hivenet-agent \
  -n 100 \
  --no-pager

Test connectivity from the agent to the router:

nc -zv 192.168.1.100 50051
nc -zv 192.168.1.100 9000

Check that:

the router and agent use the same JWT secret
the router’s libp2p interface is reachable
the router advertises a libp2p address the agent can reach
llama.cpp is healthy and exposes a model

The peer ID changes after restart

Make sure --identity-path points to persistent storage. For Docker:

docker inspect hivenet-agent \
  | jq '.[0].Mounts'

For bare metal:

ls -l /var/lib/hivenet-router/agent/identity.key

Requests time out

The agent’s backend HTTP timeout defaults to two minutes. Increase it for long-running requests:

--http-timeout 10m

Also check the declared agent capacity and llama.cpp queue state.

GPU metrics are missing

Check the host:

nvidia-smi

For Docker, confirm the agent received GPU access:

docker inspect hivenet-agent \
  | jq '.[0].HostConfig.DeviceRequests'

The agent continues to operate without NVML. It reports CPU and memory metrics but omits GPU measurements.

Next steps

Infinity agent

Connect embedding and reranking models through Infinity.

Routing concepts

Learn how Hivenet Router filters and ranks matching agents.

Engine metrics

Understand the metrics collected from llama.cpp and other engines.

Getting started

Deploy Hivenet Router

Use the API

Routing and policies

Security and auth

Observability

Integrations

Reference

Project

Before you start

Start llama.cpp

Choose a stable model name

Prepare the Hivenet Router agent

Start the agent

Understand the agent settings

Model discovery

Set agent capacity

Verify the connection

llama.cpp metrics

Use llama.cpp metrics in routing policies

Tune llama.cpp

Reduce the context size

Change the quantization

Load layers on the GPU

Enable flash attention

Troubleshooting

llama.cpp is not ready

Metrics are missing

The model name does not match

Responses are slow

The agent does not register

The peer ID changes after restart

Requests time out

GPU metrics are missing

Next steps

Infinity agent

Routing concepts

Engine metrics

​Before you start

​Start llama.cpp

​Choose a stable model name

​Prepare the Hivenet Router agent

​Start the agent

​Understand the agent settings

​Model discovery

​Set agent capacity

​Verify the connection

​llama.cpp metrics

​Use llama.cpp metrics in routing policies

​Tune llama.cpp

​Reduce the context size

​Change the quantization

​Load layers on the GPU

​Enable flash attention

​Troubleshooting

​llama.cpp is not ready

​Metrics are missing

​The model name does not match

​Responses are slow

​The agent does not register

​The peer ID changes after restart

​Requests time out

​GPU metrics are missing

​Next steps

Infinity agent

Routing concepts

Engine metrics

Before you start

Start llama.cpp

Choose a stable model name

Prepare the Hivenet Router agent

Start the agent

Understand the agent settings

Model discovery

Set agent capacity

Verify the connection

llama.cpp metrics

Use llama.cpp metrics in routing policies

Tune llama.cpp

Reduce the context size

Change the quantization

Load layers on the GPU

Enable flash attention

Troubleshooting

llama.cpp is not ready

Metrics are missing

The model name does not match

Responses are slow

The agent does not register

The peer ID changes after restart

Requests time out

GPU metrics are missing

Next steps