Exclusive Access · Invitation Only

Load Balancer

Stratum ships an L4 load balancer that works in the node's packet path: it distributes TCP and UDP flows across a set of backends using a per-VIP algorithm, maintains a stateful connection table so established flows stay pinned to their backend, and performs health checks to take unhealthy backends out of rotation automatically.

Status: the L4 load balancer's commands (VIP definitions, backend pools, health checks) and its packet handling are both in place. Validate the L4 balancer against your own workload profile before production rollout. The Layer-7 HTTP proxy described below is a separate reverse proxy and does not depend on it.

A VIP can serve traffic from outside the fabric or stay internal for service-mesh style balancing between endpoints — that is a matter of how you route and announce it, not of what kind of node you are on.

Two load balancers, and which one you want

Stratum has two, and they solve different problems. Choosing the wrong one is the most common mistake, so start here.

L4 (lb)L7 HTTP (l7lb)
Works onTCP and UDP — any protocolHTTP and HTTPS only
Decides usingAddresses and portsHostname, URL path, headers
Handles TLSNo — traffic passes through untouchedYes — terminates it for you
Use it forDatabases, game servers, SMTP, gRPC, anything not HTTP, or HTTPS you want passed straight throughWebsites and APIs where you want one address serving several apps, or TLS handled centrally

They are independent. You can run both — an L4 VIP in front of a database and an L7 frontend for your web tier — and neither depends on the other.

What the L4 balancer requires: its own interface

This is the requirement people actually hit.

The L4 balancer must be given a dedicated interface to receive VIP traffic on — one it does not share with your workload bridge or with the node's WAN or LAN forwarding interfaces. An interface can do only one of these jobs, so they cannot be stacked on the same NIC. See Nodes and Interfaces.

Until that interface is configured, VIPs are configuration only. Every lb command works, your VIPs and backends are stored and listed back correctly, and health checks report — but no traffic is balanced, because nothing is attached to receive it. If your VIP looks perfectly configured and simply does not pass traffic, this is the first thing to check.

The interface is named in the node's configuration and applied at startup — see Configuration. The L7 balancer has no such requirement: it is an ordinary listener, so it needs a free port and nothing else.

Every node can host workloads and forward traffic — one node doing both is a normal setup. What cannot be shared is an interface, not a role.

How the L4 balancer moves a packet

Understanding this matters because one deployment detail depends on it.

  1. A client connects to the VIP. As far as the client is concerned it is talking to one address, and it will keep talking to that address for the lifetime of the connection.
  2. A backend is chosen for the connection, using the VIP's algorithm.
  3. The destination address is rewritten to that backend, and the packet is forwarded on. The backend can be anywhere the node can route to — it does not have to be on the same subnet or the same machine.
  4. The connection is remembered, so later packets in that conversation go to the same backend while it stays in rotation and the node still remembers the connection — see How long a connection keeps its backend.
  5. The reply comes back through the node, which puts the VIP address back on it before it reaches the client — so the client only ever sees the address it connected to.

Two consequences follow, and they surprise people:

Your backends see the real client address. Only the destination is rewritten, never the source. Your application logs show actual client IPs with no extra configuration and no X-Forwarded-For to parse.
Replies must come back through the balancing node. Step 5 is where the VIP address is restored. If a backend answers the client directly — because it has its own route out — the client receives a reply from an address it never contacted and discards it. The usual symptom is connections that hang instead of failing. Make sure your backends route back through the node holding the VIP.

How long a connection keeps its backend

The node remembers up to 65,536 balanced connections at a time, shared by all of its VIPs. When it needs room for more, it forgets the connections that have been quiet longest. A forgotten connection that sends again has a backend chosen for it afresh:

  • with source-hash, maglev or consistent-hash, that is the same backend as before, as long as the VIP's backends in rotation have not changed;
  • with maglev or consistent-hash, it usually is even when they have: a backend leaving or joining changes the placement of few connections besides its own;
  • with round-robin or weighted, it can be a different backend, which breaks a TCP connection.

A connection whose backend leaves rotation is moved to another backend at once — see Adding and removing backends live.

One VIP is one address, one port, one protocol

A VIP carries a single frontend_port and a single protocol. To serve a service on two ports, create two VIPs:

# HTTP and HTTPS for the same service = two VIPs, same backends
sudo cenvero-str-ctl lb create '{"id":"web-80","frontend_ip":"10.0.0.100","frontend_port":80,"protocol":"tcp","algorithm":"round-robin"}'
sudo cenvero-str-ctl lb create '{"id":"web-443","frontend_ip":"10.0.0.100","frontend_port":443,"protocol":"tcp","algorithm":"round-robin"}'

Each VIP has its own backends, its own algorithm and its own health checks, so they can differ where that is useful — for example checking a different health path per port.

TLS at layer 4

The L4 balancer does not read or terminate TLS. Encrypted traffic passes through exactly as it arrives, and each backend must present its own certificate. That is the right choice when you want end-to-end encryption with no middle box, or when the protocol is not HTTP at all.

If you want certificates handled in one place instead, that is the L7 balancer — see Terminating TLS below.

Direct server return

Direct server return is not available: replies always pass back through the node, as described above, so plan capacity on that basis. A new VIP with dsr_enabled set to true is refused. A VIP created with it before this release keeps working, with replies through the node, and lb show lists dsr_enabled in its not_enforced entries.

Creating a VIP

lb create takes one argument: a JSON object describing the virtual IP. A VIP and its backends are created separately — first the VIP, then each backend with lb add-backend.

FieldTypeDescription
idstringRequired. The VIP's name, used by every other lb command.
frontend_ipstringRequired. The virtual IP address clients connect to, an IPv4 address.
algorithmstringRequired. One of the algorithms below.
frontend_portnumberRequired. The port clients connect to, from 1 to 65535.
protocolstringRequired. tcp or udp.
dsr_enabledbooleanRefused when true — see Direct server return.
health_checkobjectOptional active health check — see Health checks below.

The L4 balancer is IPv4 only. An address that is not IPv4, for a VIP or a backend, is refused, and so is a VIP on port 0 or a backend port outside 1 to

  1. A VIP or backend created with one of these before this release takes no
traffic, and lb show lists it in the VIP's not_enforced entries.

sudo cenvero-str-ctl lb create '{"id":"web-lb","frontend_ip":"10.20.0.10","frontend_port":80,"protocol":"tcp","algorithm":"round-robin"}'

List your VIPs:

cenvero-str-ctl lb list

Show detail for one VIP, including each backend's health:

cenvero-str-ctl lb show web-lb
{
  "data": {
    "load_balancer": {
      "id": "web-lb",
      "frontend_ip": "10.20.0.10",
      "frontend_port": 80,
      "protocol": "tcp",
      "algorithm": "round-robin",
      "dsr_enabled": false,
      "backends": [
        {
          "id": "web-1",
          "ip": "10.20.0.50",
          "port": 80,
          "weight": 1,
          "healthy": true,
          "admin_state": "in"
        }
      ]
    }
  },
  "status": "ok"
}

admin_state is in, or out while you have the backend out of rotation — see Retiring a backend.

Algorithms

Set one of these as the algorithm field. An unrecognized value is rejected when the VIP is created.

AlgorithmValueBehaviour
Round-robinround-robinDistributes new connections evenly in turn. Good default for stateless services.
Weighted round-robinweightedLike round-robin but each backend has a relative weight (see below). Use when backends have unequal capacity.
Source hashsource-hashChooses the backend from the client's address and source port, so a connection that has to be placed again lands on the same backend while the backends in rotation stay the same. When they change, a connection placed again can land on a different backend even though its own is still in rotation. It does not keep a client on one backend: a client's separate connections normally come from different ports and can reach different backends.
MaglevmaglevChooses the backend from the client's address and source port through a Maglev table: when a backend leaves or joins rotation, few connections besides its own are placed on a different backend. So a connection the node has to place again (see How long a connection keeps its backend) usually keeps its backend even after the backends have changed.
Consistent hashconsistent-hashAnother name for maglev: the same table and the same behaviour.

The three hashing algorithms spread clients by their whole address and source port: clients of one network, and one client's connections from different ports, are spread across the backends in rotation. A VIP's connections are divided into 61 shares, and each backend in rotation takes the same number of them, give or take one. With many backends that difference shows: with 40, some backends take two shares and others one, so some receive twice the new connections of others.

least-conn is not available for a layer-4 VIP and is refused: the balancer keeps no count of each backend's open connections. For HTTP, a layer-7 pool with least-connections sends each request to the backend with the fewest in progress. A VIP created with least-conn before this release keeps working, round-robin, and lb show lists it with a not_enforced entry saying so.

Setting weights

Weight is a property of each backend, so choose the weighted algorithm on the VIP and give each backend its relative weight as you add it:

sudo cenvero-str-ctl lb create '{"id":"api-lb","frontend_ip":"10.20.0.20","frontend_port":443,"protocol":"tcp","algorithm":"weighted"}'

sudo cenvero-str-ctl lb add-backend '{"vip_id":"api-lb","id":"api-1","ip":"10.20.0.50","port":443,"weight":3}'
sudo cenvero-str-ctl lb add-backend '{"vip_id":"api-lb","id":"api-2","ip":"10.20.0.51","port":443,"weight":1}'

The weights 3 and 1 are relative — api-1 receives three times the new connections of api-2. A weighted VIP has 64 shares to divide: when the weights add up to more than 64 they are scaled down to fit, and every backend keeps at least one. A backend's weight counts only with weighted: every other algorithm ignores it.

Health checks

The agent probes each backend of a VIP that has a health check about every interval_sec seconds, or every 5 seconds if you leave it out. A backend that fails enough consecutive checks is marked unhealthy and taken out of rotation, and the connections on it move to the other backends. It goes back into rotation automatically when it passes the same number of consecutive checks.

A health check is configured as part of the VIP, in the optional health_check object passed to lb create:

sudo cenvero-str-ctl lb create '{"id":"app-lb","frontend_ip":"10.20.0.30","frontend_port":80,"protocol":"tcp","algorithm":"round-robin","health_check":{"type":"http","interval_sec":10,"timeout_sec":2,"threshold":3,"http_path":"/healthz"}}'
FieldDescriptionNotes
typetcp (the backend accepts a connection), http (a GET of http_path answers with a status below 400) or https (the same request over HTTPS)Omit the whole object, or leave type empty, for no active check. Any other type is refused
interval_secSeconds between checks, from 1 to 36005 if left out
timeout_secProbe timeout in seconds, from 1 to 603 if left out
thresholdConsecutive checks before flipping a backend's stateOne value covers both directions; 1 if left out
http_pathPath to request for an http or https check, e.g. /healthz/ if left out; refused for a tcp check

An http check speaks plain HTTP to the backend's own address and port, and an https check speaks HTTPS (TLS 1.2 or later) to it. A redirect counts as an answer and is not followed. An https check does not verify the backend's certificate: it asks only whether the backend answers. For backends that serve HTTPS, use an https check: an http check against them fails, and takes them out of rotation.

Overriding a backend's health manually

lb set-health is a manual override, not the check configurator — it forces one backend up or down. It takes three positional arguments: the VIP id, the backend id, and up or down. Marking a backend down sends the connections already on it to the other backends at once, which breaks TCP connections.

# Mark a backend down: it takes no connections
sudo cenvero-str-ctl lb set-health web-lb web-1 down

# Mark it up again
sudo cenvero-str-ctl lb set-health web-lb web-1 up

A backend keeps its state when the node restarts, whether you set it or a health check did: one you marked down stays down until you mark it up.

If an active health check is configured for the VIP, the check overrides a manual setting: a backend you marked down is put back once it has passed threshold checks in a row, and one you marked up is taken out once it has failed that many — which can be at the very next check. To keep a backend out whatever its checks find, take it out of rotation instead — see Retiring a backend.

Adding and removing backends live

Adding a backend takes effect immediately and leaves established connections where they are. Removing a backend, or marking it down, sends the connections already on it to the other backends at once, which breaks TCP connections.

lb add-backend takes a JSON object identifying the VIP (vip_id) plus the backend's own id, ip, port, and weight. vip_id, id, ip and port are required.

# Add a backend
sudo cenvero-str-ctl lb add-backend '{"vip_id":"web-lb","id":"web-3","ip":"10.20.0.52","port":80,"weight":1}'

A backend's id need only be unique within its VIP: two VIPs can each have a backend called web-1.

A VIP can have at most 64 backends, or 61 with source-hash, maglev or consistent-hash. Adding one more is refused. A VIP given more before this release lists backends in its not_enforced entries: past that number, a backend takes no connections.

Retiring a backend

There is no graceful drain yet. lb drain is refused and changes nothing, because the balancer cannot yet let a backend's existing connections finish while new ones go elsewhere. To retire a backend, remove it or mark it down at a quiet time.

To take a backend out for maintenance and keep it out, use lb out-of-rotation. The backend takes no connections: new ones go to the other backends, and so do the ones already on it, at once, which breaks TCP connections. It stays out whatever its health checks find, and when the node restarts, until you put it back with lb back-in-rotation. Then it takes new connections again once it is healthy, and connections already on the other backends stay where they are.

sudo cenvero-str-ctl lb out-of-rotation web-lb web-1
sudo cenvero-str-ctl lb back-in-rotation web-lb web-1

Each command answers with the backend's state and a note saying what happened to connections. Taking out the last backend in rotation leaves the VIP with none: its connections reach no backend until one is back.

lb remove-backend takes two positional arguments — the VIP id and the backend id:

# Remove a backend
sudo cenvero-str-ctl lb remove-backend web-lb web-1

A removal is saved before it takes effect: if the node cannot save it, removing a backend or a VIP is refused and changes nothing.

List a VIP's backends at any time:

cenvero-str-ctl lb backends web-lb

Reaching a VIP from outside

A VIP is reachable from outside only if traffic for its address is routed to the node. Nothing is announced on your behalf: a VIP address is not advertised just because it exists. Give the VIP an address your upstream already routes to the node, or assign it from a prefix you announce yourself with bgp announce:

sudo cenvero-str-ctl lb create '{"id":"external-web","frontend_ip":"203.0.113.10","frontend_port":443,"protocol":"tcp","algorithm":"round-robin"}'

sudo cenvero-str-ctl lb add-backend '{"vip_id":"external-web","id":"ext-1","ip":"10.20.0.50","port":443,"weight":1}'
sudo cenvero-str-ctl lb add-backend '{"vip_id":"external-web","id":"ext-2","ip":"10.20.0.51","port":443,"weight":1}'
sudo cenvero-str-ctl bgp announce 203.0.113.0/24

See BGP Edge Routing for how prefixes are advertised.

Deleting a VIP

sudo cenvero-str-ctl lb delete web-lb

Connections through it stop working immediately.

Layer-7 HTTP load balancer

Separately from the L4 balancer above, Stratum ships a Layer-7 (HTTP/HTTPS) reverse proxy, run by the agent itself. Where the L4 balancer distributes raw TCP/UDP flows, the L7 proxy understands HTTP: it terminates TLS, routes by Host header and URL path prefix, and load-balances across backend pools. The two are orthogonal — reach for the L4 balancer for raw flow distribution and the L7 proxy when you need HTTP-aware routing or TLS termination. The L7 proxy stays off until you configure a frontend.

Its two building blocks are:

  • Pools — a named set of HTTP backends with a selection algorithm (round-robin or least-connections) and health checks.
  • Frontends — a listener, optionally terminating TLS, that routes each request to a pool by Host and path prefix, with a default pool as fallback.

How a request travels

  1. A request arrives at a frontend — a listener on one address and port.
  2. If the frontend has TLS, it is decrypted here. From this point on the proxy is working with a plain HTTP request.
  3. The routes are checked in order and the first match wins. A route matches on hostname, on a path prefix, or both. Anything matching no route goes to the frontend's default_pool; a frontend with no default and no match returns an error rather than guessing.
  4. The pool picks a backend — round-robin, or the one with the fewest active requests.
  5. The request is forwarded over plain HTTP and the response passed back.

Because the proxy is the one talking to the client, it adds X-Forwarded-For (the real client address), X-Forwarded-Proto and X-Forwarded-Host so your application can still see who asked and how. WebSocket and other upgrade connections pass through unchanged.

Routing is per request, not per connection — two requests on the same keep-alive connection can land on different backends, and on different pools.

Terminating TLS

This is where most questions come up, so plainly:

The certificate goes on the frontend, once. Your backends do not need one.

A frontend with a tls block accepts HTTPS from clients, decrypts, and then speaks plain HTTP to the backends. So:

  • You need one certificate per frontend, not one per backend and not one per port. Ten backends behind an HTTPS frontend need zero certificates between them.
  • The backend link is unencrypted. That is fine when the backends are on a network you control — the usual case, since they are your own endpoints on your own fabric. If you need encryption all the way to the application, do not terminate here: use an L4 VIP and let each backend hold its own certificate.
  • A frontend with no tls block serves plain HTTP. Mixing is normal — run a plain frontend on :80 and a TLS one on :443.

The certificate itself is either a PEM certificate and key you supply, or the node's own managed certificate. The minimum accepted version is TLS 1.2. On the node's command line you name the PEM files on the node, or choose the node's own certificate with "tls":{"use_managed":true}; over the API and in the web console you send a certificate's content instead, and the node's own certificate cannot be chosen there — see Over the API.

Serving several ports

Unlike an L4 VIP, one frontend is one listener — but you can add as many frontends as you need, and they can share pools:

# Plain HTTP on 80 and TLS on 443, both serving the same backends
sudo cenvero-str-ctl l7lb frontend add '{"name":"http","listen":":80","default_pool":"web"}'
sudo cenvero-str-ctl l7lb frontend add '{"name":"https","listen":":443","tls":{"use_managed":true},"default_pool":"web"}'

One frontend can also serve many sites on a single port by routing on hostname — that is the usual reason to choose L7 over L4:

sudo cenvero-str-ctl l7lb frontend add '{"name":"edge","listen":":443","tls":{"use_managed":true},
  "routes":[
    {"host":"api.example.com","pool":"api"},
    {"host":"www.example.com","path_prefix":"/static","pool":"assets"},
    {"host":"www.example.com","pool":"web"}
  ],"default_pool":"web"}'

Order matters: the /static rule sits before the general www rule, because the first match wins.

Backend pools

A pool is created from a single JSON object:

sudo cenvero-str-ctl l7lb pool add '{"name":"web","algorithm":"round-robin","backends":["10.20.0.50:8080","10.20.0.51:8080"]}'

For a pool that already exists, l7lb pool add changes its algorithm and health check and keeps its backends: naming different backends is refused. Add and remove them one at a time instead.

List pools, and add or remove a pool's backends live:

cenvero-str-ctl l7lb pool list
cenvero-str-ctl l7lb backend list web
sudo cenvero-str-ctl l7lb backend add web 10.20.0.52:8080
sudo cenvero-str-ctl l7lb backend remove web 10.20.0.50:8080

Each backend has an id within its pool: the first free one of b1, b2, … unless you give one over the API. l7lb backend remove takes either the backend's address or its id.

The proxy runs active health checks (periodic HTTP probes with configurable healthy and unhealthy thresholds) and also ejects a backend passively when it returns transport errors while proxying, re-admitting it once it recovers.

TLS-terminating frontends

A frontend binds a listener and routes to pools. This one terminates TLS with an operator-provided certificate and routes one host and path prefix to the web pool, falling back to it by default:

sudo cenvero-str-ctl l7lb frontend add '{"name":"https","listen":":8443","tls":{"cert_file":"/etc/cenvero-str/tls/lb.crt","key_file":"/etc/cenvero-str/tls/lb.key"},"default_pool":"web","routes":[{"host":"api.example.com","path_prefix":"/v1","pool":"web"}]}'

TLS termination accepts either an operator-supplied PEM certificate and key (as above) or, from the node's own command line only, the node's own managed certificate; the minimum TLS version is 1.2. A frontend with no tls block serves plain HTTP. The proxy sets the X-Forwarded-For, X-Forwarded-Proto, and X-Forwarded-Host headers on proxied requests and passes WebSocket and other upgrade connections through unchanged.

A frontend listens on the port you give it, from 1 to 65535: port 0 is refused. The ports the server's own services use are refused too, on any address: 7070 to 7074 (the APIs, the cluster and the gateway pair's heartbeat, even before the server uses them) and the ports the server's configuration gives its REST, gRPC and WebSocket APIs. A frontend saved on one of them by an earlier version is kept and does not listen; l7lb status gives the reason "the port is one this server's own services use". Remove it and add it again on another port.

List or remove frontends, and show the full picture — every frontend and pool with per-backend health and the requests each backend is serving (requests_in_progress in the output; active_connections is the same count under its old name), and whether each frontend is listening:

cenvero-str-ctl l7lb frontend list
sudo cenvero-str-ctl l7lb frontend remove https
cenvero-str-ctl l7lb status

A frontend that cannot start when the node starts (its port taken, its certificate unreadable) is kept. l7lb status shows it with listening: false and the reason, and the node tries it again every 30 seconds, so it starts once the port is free or the certificate is back. Adding a frontend whose port is taken is refused at once. The reason is a short fixed phrase, such as "the address is in use"; on the node's own command line, detail adds the system's full message, which can name a file.

A saved pool that cannot be loaded when the node starts — one saved by a newer version with a setting this one does not accept, for example — is kept as saved, and serves nothing. l7lb status shows it with loaded: false and the reason, and frontends that route to it are not listening. Add the pool again with l7lb pool add, which replaces it, or remove it with l7lb pool remove once no frontend routes to it. Frontends waiting for it start within 30 seconds of the pool being added again.

When a pool has no healthy backend the proxy returns 503; when a chosen backend fails mid-request it returns 502 without leaking internal error detail.

The proxy does not open tunnels: a CONNECT request is answered 405 Method Not Allowed, never reaches a backend and does not count against the backend's health.

Over the API

Pools, their backends and frontends can also be created, listed and removed over the REST API — see the API reference. The rules are the command line's, with these differences:

  • A frontend's certificate is sent as its PEM content (certificate_pem and key_pem). A path to a file on the node is refused, and so is use_managed (400): the node's own certificate goes on a frontend only on the node's own command line, run as root.
  • A frontend that uses the node's own certificate cannot be changed over the API, and neither can the pools it routes to: creating such a pool, changing its algorithm or health check, and adding or removing a backend are refused with 409. Change them with cenvero-str-ctl on that node, as root: the same commands sent from another member of a cluster with --node are refused too. The frontend can still be removed over the API. A pool lists such frontends in managed_frontends.
  • Once no frontend with the node's own certificate routes to a pool, the pool can be changed over the API again. So before you make such a frontend on the command line — again after one was removed, too — check the backends of every pool it routes to (cenvero-str-ctl l7lb backend list <pool>): the frontend hands what its clients send, believing they talk to this node, to those backends.
  • The node keeps the certificate and key itself and never returns them: a frontend shows only whether it has a certificate (has_certificate). They are deleted when the frontend is removed.
  • Why a frontend is not listening, or a pool is not loaded, is given as the fixed phrase only, never the system's message.
  • A pool's or frontend's name is 1 to 64 letters, digits, ., _ or -, starting with a letter or digit.

See also

↓ This page as JSON ↓ All documentation as JSON