Architecture overview

Hivenet Router uses a hub-and-spoke architecture. A central router exposes the API used by applications and coordinates a fleet of agents. Each agent connects one inference backend to the router and reports the information needed to make routing decisions. Applications only need to know the router endpoint. They do not need to know which machine, engine, or agent handles each request.

​

Main components

Hivenet Router has two runtime components: the router and one or more agents.

​

Router

The router is the central entry point and coordinator. It:

• exposes OpenAI-compatible and Anthropic-compatible inference endpoints
• authenticates clients when API authentication is enabled
• authenticates and registers agents
• tracks agent health, models, capabilities, and metadata
• evaluates routing policies
• selects an available agent for each request
• enforces quotas and model restrictions
• maintains operational and historical state
• exports Prometheus metrics

The router exposes four interfaces by default:

​

Agent

An agent connects one inference backend to the router. It:

• authenticates with the router
• registers its engine, model, capability, capacity, region, and tags
• forwards inference requests to its local backend
• checks backend health
• collects CPU, memory, and GPU metrics
• scrapes engine metrics when the backend exposes them
• sends heartbeats and routing signals to the router
• maintains a persistent libp2p identity when an identity path is configured

Each agent registers one model and one capability. Supported capabilities are:

• llm
• embedding
• reranker

​

Supported backends

A deployment can mix different backend types. Applications continue to use the router’s API rather than connecting to each backend separately.

​

Request lifecycle

A typical inference request moves through Hivenet Router like this:

1. A client sends a request to the router.
2. The router authenticates the client if API authentication is enabled.
3. The router identifies the requested model and capability.
4. Static policy filters remove agents that do not match the request.
5. Dynamic policy gates remove agents that fail configured health or metric thresholds.
6. The routing strategy ranks the remaining agents.
7. The router acquires capacity on the selected agent.
8. The request is forwarded to the agent over the encrypted libp2p data plane.
9. The agent sends the request to its local inference backend.
10. The response returns through the agent and router to the client.
11. Hivenet Router records the outcome and updates routing and latency metrics.

If the initial routing step cannot serve the request, Hivenet Router can continue through a configured fallback chain. An optional provider fallback can send the request to OpenAI or Anthropic after local options are exhausted.

​

Routing pipeline

Routing policies use three stages. least-loaded is currently the only implemented ranking strategy. It ranks agents by the proportion of their declared capacity that is in use. If you do not configure a policy file, Hivenet Router uses its built-in least-loaded policy without static filters or dynamic gates.

​

Control and data paths

Hivenet Router separates authentication from inference traffic.

​

Agent authentication

1. The agent signs a JWT with the shared secret.
2. It connects to the router’s gRPC authentication endpoint.
3. The router validates the token and agent metadata.
4. The router returns a session token and its libp2p address.
5. The agent registers over libp2p using the session token.

The gRPC connection uses TLS 1.3. The certificate and client trust material are derived from the shared JWT secret, so the deployment does not require a separate public-key infrastructure for router-agent authentication. Agent sessions last one hour by default. Agents reauthenticate before their current session expires.

​

Request forwarding

Router-agent traffic uses libp2p with Noise encryption. The same connection carries registration, heartbeats, routing signals, and forwarded inference requests. Agents initiate the persistent connection to the router, which then uses that established connection for bidirectional signaling and request forwarding.

​

Client traffic

The router’s built-in API server listens over HTTP. For production deployments, place the API behind a reverse proxy or load balancer that provides TLS and any network controls your environment requires.

​

Health and metrics

Agents collect backend and system state at different intervals. The router uses these signals to maintain agent health and make routing decisions. Routing signals carry recent engine and hardware data without updating the agent’s heartbeat timestamp. Heartbeats act as the health keepalive and include fallback metric information. This separation lets Hivenet Router receive fresh routing data without treating every metrics update as proof that the agent is healthy.

​

Storage

Hivenet Router uses two BadgerDB stores. The in-memory database is rebuilt as agents connect and report state. The disk database preserves information such as smoothed round-trip time and lifetime counters across router restarts. You can change its path and retention period through router configuration.

​

Authentication and access control

Hivenet Router handles agent and client authentication separately.

​

Agents

Agents authenticate with a shared HMAC-SHA256 secret. The router validates their JWTs over the gRPC authentication connection.

​

API clients

The inference API supports three authentication modes: API keys can also define:

• allowed models
• requests-per-minute limits
• daily token limits
• ownership and descriptive metadata
• expiration dates

The administration API has its own authentication configuration and should not be exposed without protection in production.

​

Deployment shape

The router and agents do not need to run in the same place. Common deployments include:

• one router and several agents on a private network
• agents distributed across multiple sites
• inference backends running on bare-metal GPU hosts
• router and agents running in Docker
• a mix of local infrastructure and hosted compute instances

Hivenet Router does not require Kubernetes. The current repository does not include a supported Helm deployment.

​

What to read next