Exclusive Access · Invitation Only

Cluster Networks

A cluster network is one network stretched over the members that carry it. You create it once, for the cluster, and a machine on one member can reach a machine of the same network on another member. Create a network on a single node when its machines live on that node only; create a cluster network when they live on several members.

Part of Networking and Clustering. Creating and changing a cluster network needs clustering in the licence of the node you call; a machine on one needs compute. Reads are never refused.

When a member takes machines on a network

A member takes machines on a cluster network once it holds two checks, and it answers for itself:

  • Only the network's own members are accepted. The member accepts the network's traffic only from the members that carry it, and drops a frame that comes from any other address.
  • Only the network's own tenant is reached. The member makes the network's connection a port of its tenant's machines, isolated as that tenant's port, so the network's traffic reaches that tenant's machines on the member.

Until a member holds both checks, a machine that asks for the network is refused with "This network cannot take machines yet." A member that holds both but cannot isolate the network's connection refuses the network with **"This server cannot keep the network apart from the tenants' machines, so it cannot take this network."**

A member cannot check the network's traffic on every path when its overlay address is on a loopback or dummy device, is an IPv6 address, or is missing, or when the nft tool is not installed. The installer installs that tool, so it is missing only on a server where someone removed it, or one updated without running the installer. Such a member does not take the network at all. It refuses the network with **"This member cannot check its overlay traffic, so it cannot take a cluster network. Its overlay address must be an IPv4 address on a network interface, and the nft tool must be installed."** The member's own operator networks on that address are kept, and run without the check; the networking overview says what that means.

The answers say which case applies. The list of cluster networks, and each network's own answer, carry "released": true on a member that can take machines on them, and "released": false on one that cannot. A network's state is ready once it can take machines, and awaiting_check until then; the console shows the same state as ready or awaiting check.

Creating a cluster network

sudo cenvero-str-ctl cluster network create --name web-tier --cidr 10.20.0.0/24 \
  --gateway 10.20.0.1 --tenant t-acme --host-gateway --dhcp

Over the API, POST /api/v1/cluster/networks takes {"name", "cidr", "gateway", "tenant_id", "mtu", "host_gateway", "dhcp"} and answers 202 with the network. Only these fields are read: a field the route does not take is refused. If the network is created and its host gateway cannot be set, the answer is still 202 with the network, which then has no host gateway, and host_gateway_error says so; set the host gateway on the network to finish, rather than creating it again. A tenant that was removed is refused as one this server does not know.

The rules are the ones the answer gives when a value breaks them:

  • The name is 1 to 64 printable characters.
  • The address range is an IPv4 block in its canonical form, between /20 and /30. "The network's address range must be an IPv4 block in canonical form." and "The network's address range must be between /20 and /30."
  • The range may not overlap another cluster network's: "This network's address range overlaps another cluster network's, so it cannot be created."
  • The gateway is a host address inside the range: "The network's gateway must be a host address inside its range."
  • Every cluster network belongs to one tenant. Only that tenant's machines attach to it. A network with no tenant is refused with "The network names no tenant it can belong to." A tenant this server does not know is refused with "This tenant is not known to this server."
  • The MTU is between 576 and 9000 (see below).
  • DHCP needs the host gateway. "A network's DHCP is served by its host gateway, so DHCP needs a host gateway."
  • A host gateway cannot turn DHCP off. "A cluster network's host gateway serves DHCP wherever this member's DHCP service is on, so DHCP cannot be set to false with the host gateway." An omitted dhcp is accepted, and so is true.

The cluster gives each network an identifier, and a number on the overlay (VNI) that no other cluster network has. The number appears in the network's answer for operators who can read the network.

Where a network runs

A member takes part in a network when its first machine attaches to the network there. It leaves the network **ten minutes after its last running machine on it stops**, so a machine that stops and starts again within that time keeps the member in place.

The host gateway is the one member that answers as the network's gateway and, when DHCP is on, serves its addresses. A network has at most one host gateway, so a single segment never has two gateways answering for one address. A member whose DHCP service is off still answers as the gateway, and the network's DHCP follows once that service is turned on; until then the member's view of the network says why DHCP is not served. Set or move it:

sudo cenvero-str-ctl cluster network host-gateway <network-id> --member edge-1
sudo cenvero-str-ctl cluster network host-gateway <network-id> --clear

Over the API: PUT /api/v1/cluster/networks/{id}/host-gateway with {"node": "<member>", "enabled": true} answers 202. It may also name "dhcp", and false is refused as above. Setting a host gateway on a network whose gateway address is not set is refused: **"A host gateway needs the network's gateway address, so give the network a gateway first."** A member that runs the host gateway keeps its place in the network until the host gateway is moved or cleared: "The member runs this network's host gateway. Clear the host gateway first."

There is no failover for the host gateway. If its member is down, the network has no gateway until you move the host gateway to another member. The cluster does not do that for you.

What reaches what

On each member, the network's traffic reaches the machines of its own tenant. Another tenant's machines receive none of it, and another tenant's traffic never enters the network. Machines with no tenant are not separated at their port, as on any node (see Current limits).

Traffic between members is not metered per tenant.

Overlay traffic between members is not encrypted. Carry it on an underlay that reaches only the members.

Addresses and the MTU

The guest MTU of a cluster network defaults to the cluster's default, 1500, but each member lowers it to what its own underlay can carry: the underlay's MTU less 50 bytes of overlay overhead. On a standard 1500-byte underlay, the default is 1450. A machine on a network whose MTU is below 1500 receives that MTU in its first-boot network settings, so a cloud-image guest uses it.

A member whose underlay cannot carry the MTU you ask for refuses the network with "This member's network cannot carry the MTU this network needs." The MTU must be between 576 and 9000; otherwise **"The network's MTU must be between 576 and 9000."**

Attaching machines

A machine attaches to a cluster network on the member that runs it, and its tenant must be the network's tenant: **"This machine belongs to a different tenant from the network, so it cannot attach to it."** On a member that does not yet take machines on the network, the attach is refused with "This network cannot take machines yet."

Deleting a cluster network

sudo cenvero-str-ctl cluster network delete <network-id>

Over the API: DELETE /api/v1/cluster/networks/{id} answers 200 when the network goes. It is refused with 409 and the sentence **"Machines are attached to this network. Move or remove them first."** while any of these holds: a machine uses the network; a member still holds it (members leave ten minutes after their last machine stops); or the network has a host gateway. Clear the host gateway, and wait out the ten minutes, before you delete.

Listing and reading

cluster network list and cluster network show <id> give each network's name, range, gateway, tenant, members, host gateway, machine count, state and the released flag.

In the console, Datacenter › Networks lists the networks with their members, host gateway, machine count and state. Its actions are Create a cluster network, **Save the host gateway and Delete the network**, each offered only where the server carries it out.

Who can do what

ActionPrivilegeLicence
List and read a cluster networkNetwork.Auditnone (reads are never refused)
Create or delete a cluster networkNetwork.Allocatecluster
Set the host gatewayNetwork.Modifycluster

A tenant's key cannot use these routes; they answer 403.

See also

↓ This page as JSON ↓ All documentation as JSON