Clustering Overview

A Stratum cluster is a group of nodes that share network state and run a VXLAN overlay mesh so that every managed network is present on every host. Shared state is replicated across the cluster for high availability: one node is the elected leader and accepts writes, the others replicate it. The VXLAN overlay stretches L2 segments across physical hosts, so a workload's endpoint keeps its IP and MAC when it moves between nodes.

Every node in a cluster is the same kind of node: each hosts workloads and each can handle north-south traffic.

Which node do you talk to?

This is the first question when building anything against a cluster, and the answer is not "any of them".

Reads work anywhere. Every member holds the replicated state, so listing networks, endpoints, rules or leases gives the same answer on any node.

Changes belong to the node that owns the thing you are changing. Two kinds of state exist and they behave differently:

What you are changing	Where to send it
Something on one machine — its interfaces, its local services, its own attachments	That node
Shared cluster state	The leader

A change to shared state sent to a follower is not silently swallowed, but it does not replicate either — the leader is what writes. Ask cluster status which node is the leader before making one, and expect the answer to change over time, because leadership moves when a node restarts or becomes unreachable.

For anything automated, look up the leader rather than hard-coding a node. A cluster that has failed over once will otherwise start silently dropping your changes while still answering every read normally.

Changes are not instant across the cluster. Replication is quick but not simultaneous, so a write on the leader followed immediately by a read on another node can miss it. If you write then verify, read back from the same node you wrote to.

Ports used by clustering

Cluster communication happens only on the management bridge. Open these ports between all cluster members on the management network:

Port	Protocol	Purpose
7071	TCP	Node-to-node control plane
7073	TCP	Cluster state replication
7074	UDP	Gateway HA

Do not expose these ports to untrusted networks. The management bridge is separate from the workload bridge for exactly this reason — see Networking Overview.

High-availability cluster state

Stratum replicates cluster state across every member: the list of networks, endpoint bindings, firewall rules, load balancer VIPs, and cluster membership. State replication is persisted on each node so a restarted node rejoins with the latest state.

Clustering runs only when it is enabled (cluster.enabled: true) and requires mutual TLS between members — the transport refuses to run unauthenticated and there is no silent plaintext downgrade. Provision the cluster member certificates before enabling clustering.

Key properties:

A cluster of N nodes tolerates (N-1)/2 simultaneous failures and still makes progress.
A 3-node cluster (2 Compute + 1 Gateway, for example) tolerates 1 failure.
A 5-node cluster tolerates 2 simultaneous failures.
A cluster of 2 nodes has no fault tolerance — losing one node stalls writes.

A 1-node deployment runs in single-node mode: the agent is always the leader and there is no replication. This is fine for development and testing.

VXLAN overlay mesh

Each network you create is stretched across all cluster members via a VXLAN overlay, anchored on the management interface. When an endpoint on node A sends a frame to an endpoint on node B, the overlay encapsulates the frame and sends it to node B's management IP. The receiving node decapsulates it and delivers it to the destination endpoint — transparently, as if both endpoints were on the same physical switch.

Each Stratum network gets its own isolated overlay segment, so networks remain isolated even though they share the same underlay. MAC-to-node mappings are maintained in the cluster's shared state and updated in real time as endpoints attach, detach, and move between nodes. Validate cross-host forwarding against your own topology before production rollout.

Forming a cluster

Clustering is provisioned centrally, not with an ad-hoc CLI join. Enable it through the panel-delivered signed configuration (the cluster_enabled / cluster_bind_addr / cluster_bootstrap fields and the mutual-TLS material under cluster_cert_dir), read by each agent at boot. The first node bootstraps as leader; each additional node joins once its config is applied and its member certificate is trusted.

A node can also be added over the node's REST API — POST /api/v1/cluster/join with { "node_id": "<id>", "address": "<mgmt-ip:7073>" } — pointing the joining node at an existing member. The joining node syncs the full cluster state and applies it before participating in elections.

Check the cluster from any member:

cenvero-str-ctl cluster status

ROLE      NODE-ID   PEER              STATE
leader    cmp-01    10.0.0.11:7073    healthy
follower  cmp-02    10.0.0.12:7073    healthy
follower  gw-01     10.0.0.13:7073    healthy

Leader election

If the current leader becomes unreachable, the remaining members start an election after a short timeout elapses with no signal from the leader. The node with the most up-to-date state and a majority of votes becomes the new leader. During the election window, writes are paused — existing traffic continues uninterrupted: forwarding does not depend on the leader being available, so an election is felt only by whoever is trying to make a change at that moment.

You can see the current leader at any time — cluster status reports the leader's id along with an is_leader flag for the node you ran it on:

cenvero-str-ctl cluster status

Joining and leaving

Add a new node at any time by provisioning it into the cluster — through the panel-delivered config or POST /api/v1/cluster/join. The cluster rebalances: if the new node is a Gateway, it begins participating in HA once it has caught up with the current cluster state.

To remove a node gracefully — for maintenance or decommission — have it leave over the node's REST API:

curl -k -X POST "$NODE/api/v1/cluster/leave" -H "Authorization: Bearer $TOKEN"

This notifies the leader, which records the membership change across the cluster and adjusts the quorum size. The leaving node shuts down its cluster participation cleanly. Workloads on a node being removed should be moved to another node first — see Moving Workloads Between Nodes.

Do not hard-power-off a node without letting it leave the cluster first. The cluster will continue to function (assuming quorum remains), but it will count the node as a failed member until it is explicitly removed.