Upgrades
Stratum upgrades are pull-based and cryptographically verified. The management panel publishes releases; each agent polls a signed manifest, and when a newer, complete release is available it downloads and verifies it. A server on its own then installs it; a member of a cluster only keeps it ready, and the cluster's rolling update decides when each member changes (see Updating the members of a cluster). The panel never pushes an update to a node.
Releases
A release bundles the agent (and the CLI) at a pinned version, one download per supported architecture. There is nothing else to fetch: everything the agent needs on the node comes with it.
A release becomes visible to agents only when it is published and complete — every required (component, architecture) artifact is present and carries a publisher signature. A half-published or unsigned release is never advertised, so an agent never pulls an incomplete set. Once published, a release is immutable: the files of a release that has already gone out are never replaced.
Release channels (stable, beta, RC)
Every release ships on one of three channels: stable, beta, or rc (beta and
RC together form the pre-release track). The channel comes from the release
tag — vX.Y.Z is stable, vX.Y.Z-beta.N is beta, vX.Y.Z-rc.N is rc.
Pre-release builds require a pre-release license (see Licensing), and the gate is strict in both directions:
- a stable license is served — and can run — only stable builds;
- a pre-release license is served — and can run — only beta/RC builds, never stable.
This is enforced at every layer: the manifest only advertises releases on the
channel your license is entitled to, the /download route returns 404 for any
out-of-channel build (so a stable key cannot even fetch a beta binary), and the
channel is written into the signed license, so the agent refuses to run a
mismatched build even offline.
The update manifest
Agents read /update/manifest.json from the panel. The manifest is
license-gated (the same valid-license check as the installer), and it
advertises the highest complete published release on the channel your license
is entitled to (see Release channels above), by semantic version — not simply
the most recently published one. Each artifact entry carries:
- a license-gated download URL on the panel,
- the artifact's sha256 checksum, and
- a publisher signature that proves the binary was released by Cenvero and has not been altered.
How a node updates
Each agent checks every six hours (every check is a no-op unless a newer, fully verified build exists):
- The agent fetches
/update/manifest.jsonand compares the advertised version against what it is running and against its monotonic version floor (see below). - If a newer version applies, it downloads the artifact for its component and architecture over HTTPS.
- It checks the download's sha256 against the manifest, then verifies the binary's publisher signature, then checks that the version the signed build itself names is the version it was offered as. Anything that fails verification is discarded — nothing unsigned, altered or offered under another version is ever installed, and a build that does not say which version it is (one from before 1.0.0-rc.83) is refused too.
- A server that is not in a cluster installs it, as a single step — there is never a half-written agent on disk. It runs from the agent's next restart, and
update statusshows a restart is pending until then; runupdate applyto restart into it now. - A member of a cluster downloads and checks the newer version but installs nothing on its own:
update statusshows it as staged, and the rolling update decides when each member changes.
update apply installs the version the server is offered and restarts the agent
into it; with --wait the command follows it through the restart until the new
version has passed its checks. A version already installed and waiting for a
restart is not downloaded again. If the agent cannot restart itself, the update's
task says so, and the version runs from the agent's next restart.
What pauses while the agent restarts
Packet forwarding does not depend on the agent process, and the interfaces stay in place across the restart, so traffic through the node keeps flowing and existing flows are not torn down. What the agent itself serves pauses for the moment it takes to start again:
- BGP sessions close and come back up; how long a peer keeps the routes it learned from this node depends on the peer (see BGP Edge Routing);
- the layer-7 HTTP load balancer, and DHCP and DNS, answer again once the agent is back;
- signed-in web console sessions end: sign in again once the server is back.
Routes left by an earlier version
After upgrading from an earlier version without rebooting the server, each static route that has a metric is on the server twice until the next reboot: once with its metric and once with metric 0 (1024 for IPv6), which the earlier version gave it; the copy with the lower metric is the one in use. Routes learned over BGP before the upgrade also stay until the next reboot, even once withdrawn. Reboot the server once after upgrading, or remove the old copy by hand:
sudo ip route del <destination> via <gateway> dev <interface> metric 0
# add "table <id>" for a route in another table; for IPv6:
sudo ip -6 route del <destination> via <gateway> dev <interface> metric 1024
Configuration an update migrates for you
An update can carry a settings change the agent applies to itself, so a node does not need an operator to keep working. There is one of those in this version.
Compute mode has been removed, and nodes on it migrate automatically. Older versions asked a node to be either a Compute node or a Gateway node; every node now routes. A node updating from an older version has its configuration rewritten for it as part of the update — no reinstall, nothing to set, and no interruption beyond the usual restart. Its networks, endpoints, firewall rules, tenants and addressing carry over untouched. See Nodes and Interfaces.
Downgrade protection
The agent keeps a version floor on disk. Once a new version runs and reports itself as the version it was installed as, the agent raises the floor to it, and it refuses any update at or below the floor; a version installed and waiting for its restart is never replaced by an older one. An offer whose version is not a well-formed release version is refused. A panel that is tampered with (or rolled back) to advertise an older version cannot walk an agent backwards, and the panel enforces its own minimum version as well, so both sides agree on the lowest acceptable version.
The floor applies to updates: to those pulled from the panel, and to those installed by
hand. A rollback is not an update, so the version floor does not stop update rollback,
and a binary put in place by hand is not checked against the floor at all.
A node that stores its machines' first-boot settings sealed (their SSH keys and
user-data) keeps a second floor: the oldest version that can read them. The node records
it when it first stores such a setting, and it does not change afterwards. While that
floor is recorded, the node installs no version older than it, and update rollback
puts back the previous copy only when that copy is known to be at or above the floor.
The refusal says that the node's machine settings are stored in a form that older
versions cannot read. A manual swap of the agent binary is not refused, and then those
settings stay unreadable until they are entered again.
Rolling back
The agent keeps a copy of the version it last ran. If a new version fails to
start twice in a row, the previous version is restored automatically, and the
update's task says so: version <new> did not start twice in a row; <previous>
was restored.
A new version that starts but reports itself as another version than the one
offered is a failed update too: the previous version is restored, the agent
restarts into it, and the task says the build installed as version <new> reports
itself as version <other>, so it is not the version offered; <previous> was
restored.
A version that fails in either of these ways, is rolled back by hand before it
ran (with update rollback, the agent running or stopped), or fails its checks
after an update and is then rolled back is remembered
on the server with the reason, which update status shows: the server never
installs it on its own again (a newer version installs as usual), and
update apply tries it again, forgetting the failure once it runs and passes
its checks.
To go back by hand:
# Return to the version this server last ran, and restart into it
sudo cenvero-str-ctl update rollback --wait
# With the agent stopped, for a version that does not start at all
sudo systemctl stop cenvero-stratum
sudo cenvero-str-ctl update rollback --offline
sudo systemctl start cenvero-stratum
update rollback returns to the version the server last ran; with a version
installed and not yet run, it only removes that install. update rollback
--offline works with the agent stopped, and refuses while it runs. It skips both
floors (see Downgrade protection): it puts the previous copy
back whatever its version. On a node that stores its machines' first-boot settings
sealed, those settings can then be unreadable until they are entered again.
Updates and rollbacks run as tasks, one at a time on each server: a second one is refused while one runs. Neither an update check, an apply nor a rollback is stopped by a frozen license, nor is cancelling an update or a rolling update.
Checking versions
# What runs, what is installed, staged and offered (contacts nothing)
cenvero-str-ctl update status
# Ask the update server what it offers (installs nothing)
cenvero-str-ctl update check
# Update to the version on offer and restart into it, following it through
sudo cenvero-str-ctl update apply --wait
Updating the members of a cluster
A member of a cluster does not install a new version on its own. Update the whole cluster with one command, run on any member:
# What the update would do, server by server; changes nothing
cenvero-str-ctl cluster update --plan
# Update every server, one at a time, and follow it
sudo cenvero-str-ctl cluster update --wait
- One server at a time: servers that do not vote, then voters, then the leader, then the server you run it on.
- Each server must come back on the new version, healthy, within 10 minutes before the next one starts.
- It stops at the first server that does not. The servers after it are not updated, and the ones before stay on the new version. If the server that stopped it runs the new version but failed its checks, it is left so, and its own rollback is offered; it is never rolled back on its own. Start the update again with the same version to resume: a server already on it is only checked. A rolling update to a version that failed on one of the servers is refused, naming the server and why: offer a newer build, or run
update applyon that server to try it again. - Every server must be online and offered the same version, and you need permission to update every server: each server is asked before anything starts.
- A voting server is never restarted while another voter is offline, so the cluster keeps its majority throughout.
- A server that does not answer for a moment — just after the previous one's restart, say — is asked again for up to 75 seconds before the update stops there; a server whose update had already begun is followed, never started twice.
- A cancel takes effect between servers: the server being updated finishes first. Cancelling needs the same permission as starting: to update every server.
- One rolling update at a time in the cluster: one started on any server while another runs, or starts, is refused, naming the server it runs from.
- A server's moves come first. A server that is taking part in a machine move, as the source or as the destination, is not restarted until the move has ended. The update waits for at most 30 minutes. After that it stops. Its task names the move, says that it "has not ended in 30 minutes, so the update stopped", and asks you to "Mend or remove the move's destination, then start the update again." The server it was waiting for is taken out of maintenance again.
- Maintenance around each restart. Once every member runs a version that supports moves, the update puts a server into maintenance before it waits for that server's moves, so no machine moves onto it while it restarts. It takes maintenance off once the server runs the new version. A server that was already in maintenance stays in it. If the update stops after it has asked a server to update, that server stays in maintenance, and the task says so: turn maintenance off once it runs the new version.
Over the API, GET /api/v1/cluster/update is the plan and
POST /api/v1/cluster/update starts it (see API Reference). While a
member restarts for an update the others list it as not answering for a moment,
and the cluster keeps working. A member running a release the others cannot
manage — much older or newer — is listed as such, with a link to its own console,
until the members are on releases that work together again.
Forming or joining a cluster needs agent 1.0.0-rc.81 or later on the node.
Accounts after an update
The release that brings accounts and access control
changes nothing you rely on: the node's API token, operator API keys and tenant
keys keep working exactly as before, and each operator key becomes an API token
of the built-in account apikeys@local, with the same full control, the same
secret and the same key id.
- A node on its own has accounts as soon as it runs the new release. Make the first administrator with
cenvero-str-ctl user add(see Installation). - In a cluster, accounts start once every member runs it. Until then the access settings answer
access control is available once every node of this cluster runs version <v>; <n> nodes still need the update, and every key works as before. Update the members one at a time, as above; nothing needs doing when the last one is updated. - Afterwards, a node on an earlier release cannot join the cluster, and a member that talks to one on an earlier release cannot pass it requests made with an account or API token (
node <name> runs a version that cannot check this account; …): keep every member on the new release or later.
Relationship to licensing
The update manifest and the artifact downloads require a valid license, just like the initial install, so a node whose license has expired or been revoked gets no new updates from the panel until the license is renewed — see Licensing. The node itself never refuses an update check, an apply or a rollback because its license is frozen.
Next steps
- Installation — the initial install flow.
- Plugins Overview — extending a node beyond the core.