Exclusive Access · Invitation Only

Operations & Troubleshooting

Day-to-day operation of a Stratum node: managing its systemd units, finding the logs, turning individual network services on and off, changing settings, applying updates, and a runbook for the issues you are most likely to hit.

Everything here runs on the node itself as root. The operator tool, cenvero-str-ctl, talks to the running agent over a local socket — see the CLI Reference for the full command surface.

Systemd units

A node runs three systemd units, installed and enabled by the installer:

UnitWhat it does
cenvero-stratumThe agent itself — configuration, APIs, and the node's packet processing.
cenvero-str-watchdogA small separate watchdog that monitors the agent and restarts it if it stops responding. It is bound to the agent (BindsTo), so it follows the agent's lifecycle.
cenvero-str-networkA one-shot boot unit that brings up the host uplinks and the two bridges (cnv-mgmt-br0, cnv-user-br0) before the agent starts, so the interfaces are present when the agent attaches to them.

Manage them with the usual systemctl verbs:

# Status and recent log lines for the agent
systemctl status cenvero-stratum

# Start / stop / restart the agent (the watchdog follows it)
sudo systemctl restart cenvero-stratum
sudo systemctl stop cenvero-stratum
sudo systemctl start cenvero-stratum

# The watchdog and the boot-time network unit
systemctl status cenvero-str-watchdog
systemctl status cenvero-str-network

Restarting the agent is safe: packet forwarding does not depend on the agent process, and the interfaces stay in place across the restart, so the brief restart does not tear down existing traffic. Stopping the agent leaves the forwarding rules and the bridges in place.

Do not disable cenvero-str-network while the node is in service — it owns the host networking the agent attaches to.

Logs

The agent logs structured JSON to standard out, which under systemd is captured by the journal. That is the first place to look:

# Follow the agent's live log
journalctl -u cenvero-stratum -f

# Last 200 lines
journalctl -u cenvero-stratum -n 200

# Since a time, or only errors and worse
journalctl -u cenvero-stratum --since "1 hour ago"
journalctl -u cenvero-stratum -p err

# The watchdog has its own unit log
journalctl -u cenvero-str-watchdog -n 100

On-disk files live under /var/log/cenvero-str/ — most notably the installer log written during install. The agent's self-healing service prunes old rotated log files (older than 7 days) from this directory when disk space runs low, so it is also where any rotated logs accumulate.

Turn up the detail when you need it (see Changing settings below):

sudo cenvero-str-ctl config set log_level debug
sudo systemctl restart cenvero-stratum

Valid levels are debug, info, warn, and error. Set it back to info when you are done — debug is noisy.

Reading the log from the web console or the API

The same lines can be read without a shell on the server: the server's System log in the web console, or GET /api/v1/syslog (see API Reference), newest first, 100 at a time and up to 1,000, the agent's or the watchdog's, filtered by level. What these never show:

  • lines the audit log keeps — read those in the audit log;
  • the value of anything named for who a line is about (an account, a user, an actor, a subject, an email address) or for a client, such as the address a sign-in came from: the audit log has those;
  • the value of anything named as a secret (a token, a password, a key, a ticket, a cookie, an authorization, a licence, a credential), and anything in PEM form, such as a key;
  • the query, fragment and user part of any address in a line;
  • the directories of a path on the host: a path keeps only its last element;
  • more than 4 KiB of any one line.

One read runs at a time, for at most 10 seconds. journalctl on the server itself shows everything.

Backups and the configuration export

# A full backup of the node's state, or a config backup (the default)
sudo cenvero-str-ctl backup create full
cenvero-str-ctl backup list

# Back up every night at 03:00 UTC, keeping the last 7
sudo cenvero-str-ctl backup schedule add daily@03:00 full 7

# The node's settings as one document, secrets left out
cenvero-str-ctl config export
  • Schedules run in UTC, whatever the host's own time zone; each schedule says so ("timezone": "UTC"). A schedule's retention (1 to 1000) removes only that schedule's own backups, never one made by hand.
  • A backup is named by its id and its file's name, never by where it is on the server: backup create answers both, backup list shows them, and backup restore takes either, as the API does. backup create and backup list answer the same when you run them for another server of your cluster with --node.
  • A backup is listed only once it has been read back intact. One backup or restore runs at a time on a node; another is refused while it runs. Each runs as a task, so a scheduled backup that fails shows in task list.
  • A config backup is a record and is not restored: it holds the firewall rules, the firewall's default action, and the networks and address pools. A full backup is staged beside the current data: backup restore unpacks and checks it, and the rollback is completed by stopping the agent, putting the staged database in place and starting it again. A damaged archive, one without a database or one whose database fails its check is refused, and nothing is staged.
  • config export prints the firewall rules, the default action, the networks and the address pools as one document, leaving out anything named as a secret. A part that cannot be read fails the whole export rather than being left out.

Network services: on / off

The agent exposes several network services you can independently enable or disable. View them and their live state with service status, and flip each one with service <name> on|off:

# See every service: its switch, its address:port, and whether it is listening
cenvero-str-ctl service status

# Turn services off / on
sudo cenvero-str-ctl service rest off
sudo cenvero-str-ctl service metrics on

The toggleable services are:

ServiceNameWhat it is
REST APIrestThe HTTPS management & operator/billing API.
gRPC portgrpcAnswers the standard gRPC health check, so a load balancer can probe the node. Nothing else is served on this port.
WebSocket APIwebsocketThe live-events / streaming API.
MetricsmetricsThe Prometheus /metrics endpoint.
DNSdnsThe built-in DNS server.
DHCPdhcpThe built-in DHCP server.

The verb-first form works too — service off rest, service on dns — and -h on the group shows the help.

A few notes:

  • A change takes effect on the next agent restart. After toggling a service, apply it with sudo systemctl restart cenvero-stratum, then re-check with service status.
  • The local control socket can never be disabled — it is how cenvero-str-ctl reaches the agent (including the command you just ran). Trying to turn it off is refused with an explanation.
  • Disabling the REST API cuts off remote callers. The REST API also serves the operator/billing API (tenant suspend/resume/limit). Turning it off is allowed — it is your call — but the command warns you first; the local socket and the panel are unaffected.

Under the hood, service <name> on|off writes the matching off-switch into the same operator local-overrides file that config set uses (see below), so a panel re-sync never clobbers your choice.

Changing settings

config set adjusts a node's operational settings on a running node. It records your change in an operator local-override file that takes precedence over the panel-delivered config and survives a re-sync — it never edits the signed node config itself.

# Inspect the active configuration (paths, ports, API settings, services)
cenvero-str-ctl config show

# Change a setting, then restart to apply it
sudo cenvero-str-ctl config set api_rate_limit 2000
sudo systemctl restart cenvero-stratum

The key names match what you see in config show, so the same name round-trips between the two. Commonly-set keys include:

KeyMeaning
log_levelAgent log level: debug, info, warn, error
api_bind_addressIP the REST/gRPC/WebSocket APIs listen on (e.g. 0.0.0.0 or a management IP)
api_rate_limit / api_rate_burstAPI requests per minute from each source IP address, and the burst allowance (0 rate disables the limiter)
api_allowed_ipsComma-separated IPs/CIDRs allowed to reach the API (empty = allow all)
port_rest, port_grpc, port_websocketAPI listen ports
metrics_bind_addrPrometheus metrics bind address host:port
dns_listen_addr, dns_allowed_clientsBuilt-in DNS server bind address and allowed clients

Run cenvero-str-ctl config set -h for the full whitelist and per-key help.

What you cannot set locally. Identity, credential and certificate fields — node_id, license_server, api_token, gateway_shared_key, and the TLS paths — and the cluster_* keys are not settable with config set by design. They are panel-assigned or provisioned; a cluster is formed with cluster create and joined with cluster join --code, and a member's identity is issued by its cluster. Reaching for one returns a clear message telling you where it is managed instead (e.g. TLS material is handled by cenvero-str-ctl tls …, and the API token by cenvero-str-ctl api-token …).
>
Every config set change applies on the next restart. It is saved immediately, but the running agent only picks it up when it restarts.

Running alongside Docker (or another host firewall)

Docker switches on a kernel setting that sends traffic crossing a Linux bridge through the host's firewall, and it sets that firewall to drop forwarded traffic by default. On a node that also runs Docker, that would drop traffic between your virtual machines on the workload bridge — even two of the same tenant — and traffic the node routes between the workload bridge and the internet.

The agent handles this by itself. Whenever the host passes bridged traffic through its firewall, it keeps rules that let exactly this traffic through:

  • traffic between the workloads on the workload bridge;
  • traffic the node routes between the workload bridge and its uplink — the interface named in gateway_wan_interface and the interfaces of the node's default routes — in both directions.

The rules sit at the top of the DOCKER-USER chain (the chain Docker keeps for your own rules), or at the top of FORWARD where there is no DOCKER-USER. The agent does this for IPv4, and for IPv6 where that is passed to the firewall too. Every 30 seconds it checks that the rules are still there and still match the uplink, and it takes them out again if the setting is switched off.

Nothing else is opened. Traffic between the workload bridge and any other interface is left to the host's firewall, as you and Docker set it up: Docker's containers stay behind Docker's own rules, and another hypervisor's bridge or a network on another card of the server stays closed to your virtual machines unless you allow it yourself — in DOCKER-USER, for instance. The same applies if the node routes workload traffic out through an interface other than its uplink (a second uplink or an overlay): let it through yourself. Stratum's own protections — tenant separation and address checks at every workload's port, and your Stratum firewall rules for the traffic the node routes — still apply to what the rules let through.

The rules carry the comment cenvero-stratum-workload-bridge; no other rule in the host's firewall is touched. To see them:

sudo iptables -S DOCKER-USER | grep cenvero-stratum
sudo iptables -S FORWARD | grep cenvero-stratum

They stay in place while the agent is stopped, so running virtual machines keep their traffic. If you remove Stratum from a server, stop the agent first and then take the rules out:

sudo systemctl stop cenvero-stratum
sudo /usr/local/bin/cenvero-stratum -remove-host-firewall-rules

Updates

Updates are pull-based: the agent polls a signed manifest from the panel and, when a newer fully-verified release for its channel is available, downloads and verifies it. A node on its own installs it, and it runs from the agent's next restart; a member of a cluster only keeps it ready for the cluster's rolling update (cenvero-str-ctl cluster update). You can check or trigger this manually:

# What runs, what is installed, staged and offered (contacts nothing)
cenvero-str-ctl update status

# Ask the update server what it offers (no download)
cenvero-str-ctl update check

# Install the version on offer and restart into it, following it through
sudo cenvero-str-ctl update apply --wait

update check reports the current version, the latest on offer for your license channel, and whether an update is available — without downloading anything. If the panel is unreachable it says so plainly rather than failing hard. update apply downloads, checks the checksum and the publisher's signature, installs the version in one step and restarts the agent into it. If a new version fails to start twice in a row, the previous one is restored automatically; update rollback goes back by hand. See Upgrades for the full model (channels, downgrade protection, rollback and the rolling update).

The node never refuses an update because its license is frozen, but the panel offers no updates to a node whose license has expired until it is renewed.

Health checks

The agent continuously self-checks core subsystems (disk space, memory, its own state store, the bridges, and packet processing) and attempts an automatic repair when one fails. Inspect or force a run:

# Latest result for every health check
cenvero-str-ctl heal status

# Force an immediate run now
sudo cenvero-str-ctl heal check

Troubleshooting

The agent won't start

1. Read the journal — it almost always names the cause:

journalctl -u cenvero-stratum -n 100 --no-pager
systemctl status cenvero-stratum

2. Confirm you are running as root and the host network came up:

systemctl status cenvero-str-network

The network unit must succeed first — it creates the bridges the agent attaches to. If it failed, fix the host networking and sudo systemctl restart cenvero-str-network.

3. Inspect the config the agent is loading:

sudo cenvero-str-ctl config show

A config that fails to decode falls back to defaults with a note in the output. A signed config that fails verification is rejected — re-fetch a fresh panel-signed config rather than hand-editing the file.

4. Check disk space and the data directory (/var/lib/cenvero-str/). A full disk stops the agent from writing its state; the health check reclaims rotated logs but a genuinely full volume needs operator action.

cenvero-str-ctl says the agent is unreachable (exit code 3)

The CLI talks to the agent over its local socket. If commands report the agent is unreachable, the agent process is not running — start it and re-check:

sudo systemctl start cenvero-stratum
cenvero-str-ctl status

License is not active

Check the installed license and this machine's hardware identity:

sudo cenvero-str-ctl license status
  • Expired / in grace / frozen — renew it. Existing traffic keeps running even when frozen; only new or changing operations are blocked. Renew with sudo cenvero-str-ctl license renew (or fetch a fresh one after renewing in your account). See Licensing for the warn → grace → freeze model.
  • Not activated — activate this machine with sudo cenvero-str-ctl license activate CNVR-XXXX-XXXX-XXXX-XXXX, then confirm it in your account.
  • Hardware-ID unavailable — license status shows the machine's hardware ID. The binding requires firmware/hardware identifiers that are not exposed inside generic virtual machines, so a node must be bare metal. If the ID shows as unavailable, you are likely running in an unsupported virtualized environment.
  • Wrong release channel — a stable license runs only stable builds and a pre-release license runs only beta/RC builds; a mismatch fails closed. Make sure the build you installed matches your license's channel.

TLS certificate is pending

After install the agent generates its own certificate; if a customer-approval or domain step is outstanding the certificate can show as pending. Confirm what the agent is using and re-issue if needed:

sudo cenvero-str-ctl config show          # shows the TLS cert/key paths in use
sudo cenvero-str-ctl tls -h               # certificate management commands

TLS material is managed by the certificate manager (cenvero-str-ctl tls …), not config set. Adding or removing a domain name with tls domain requests a new certificate, which waits for your approval in your account — see TLS/SSL & License Operations.

Node is not registered

Registration binds the node to its license and mints its node token. If a node never registered, re-check connectivity to the panel and the license, then let it re-register:

cenvero-str-ctl status                    # shows license + registration state
sudo cenvero-str-ctl license activate CNVR-XXXX-XXXX-XXXX-XXXX

Registration needs outbound HTTPS to your management/license server and a valid license. If the panel is unreachable, the agent keeps retrying — fix connectivity and it completes on its own.

"REST API disabled" — set an API token

If a client gets a REST API disabled (or 401/unauthorized) response, two settings govern access:

1. Is the service enabled? Confirm with cenvero-str-ctl service status. If rest is off, turn it back on and restart:

sudo cenvero-str-ctl service rest on
sudo systemctl restart cenvero-stratum

2. Is an API token set? The REST/gRPC/WebSocket APIs require the bearer token to be configured. cenvero-str-ctl api-token status reports whether one is set (never its value). The token is a credential, so it is not set with config set. If none is set, create one on the node and restart:

sudo cenvero-str-ctl api-token generate      # prints the token once
# or supply your own, read from stdin:
printf '%s' "$MY_TOKEN" | sudo cenvero-str-ctl api-token set
sudo systemctl restart cenvero-stratum

No reinstall is needed, and the token survives a later panel sync.

Where to look first

SymptomFirst check
Agent down / crash-loopingjournalctl -u cenvero-stratum -n 100
CLI "agent unreachable"systemctl status cenvero-stratum
License problemscenvero-str-ctl license status
A service not answeringcenvero-str-ctl service status
Settings not taking effectDid you systemctl restart cenvero-stratum?
Overall healthcenvero-str-ctl heal status

Next steps

↓ This page as JSON ↓ All documentation as JSON