Docs / Operations & Troubleshooting

CENVERO/DOCS

// Operations & Troubleshooting

Operations & Troubleshooting

Day-to-day operation of a Stratum node: managing its systemd units, finding the logs, turning individual network services on and off, changing settings, applying updates, and a runbook for the issues you are most likely to hit.

Everything here runs on the node itself as root. The operator tool, cenvero-str-ctl, talks to the running agent over a local socket — see the CLI Reference for the full command surface.

Systemd units

A node runs three systemd units, installed and enabled by the installer:

Unit	What it does
`cenvero-stratum`	The agent itself — control plane, APIs, and the in-kernel data plane.
`cenvero-str-watchdog`	A small separate watchdog that monitors the agent and restarts it if it stops responding. It is bound to the agent (`BindsTo`), so it follows the agent's lifecycle.
`cenvero-str-network`	A one-shot boot unit that brings up the host uplinks and the two bridges (`cnv-mgmt-br0`, `cnv-user-br0`) before the agent starts, so the interfaces are present when the agent attaches to them.

Manage them with the usual systemctl verbs:

# Status and recent log lines for the agent
systemctl status cenvero-stratum

# Start / stop / restart the agent (the watchdog follows it)
sudo systemctl restart cenvero-stratum
sudo systemctl stop cenvero-stratum
sudo systemctl start cenvero-stratum

# The watchdog and the boot-time network unit
systemctl status cenvero-str-watchdog
systemctl status cenvero-str-network

Restarting the agent is safe: because the data plane runs in the kernel and the interfaces stay held across the swap, the brief restart does not tear down existing traffic. Stopping the agent leaves the kernel data plane and the bridges in place.

Do not disable cenvero-str-network while the node is in service — it owns the host networking the agent attaches to.

Logs

The agent logs structured JSON to standard out, which under systemd is captured by the journal. That is the first place to look:

# Follow the agent's live log
journalctl -u cenvero-stratum -f

# Last 200 lines
journalctl -u cenvero-stratum -n 200

# Since a time, or only errors and worse
journalctl -u cenvero-stratum --since "1 hour ago"
journalctl -u cenvero-stratum -p err

# The watchdog has its own unit log
journalctl -u cenvero-str-watchdog -n 100

On-disk files live under /var/log/cenvero-str/ — most notably the installer log written during install. The agent's self-healing service prunes old rotated log files (older than 7 days) from this directory when disk space runs low, so it is also where any rotated logs accumulate.

Turn up the detail when you need it (see Changing settings below):

sudo cenvero-str-ctl config set log_level debug
sudo systemctl restart cenvero-stratum

Valid levels are debug, info, warn, and error. Set it back to info when you are done — debug is noisy.

Network services: on / off

The agent exposes several network services you can independently enable or disable. View them and their live state with service status, and flip each one with service <name> on|off:

# See every service: its switch, its address:port, and whether it is listening
cenvero-str-ctl service status

# Turn services off / on
sudo cenvero-str-ctl service rest off
sudo cenvero-str-ctl service metrics on

The toggleable services are:

Service	Name	What it is
REST API	`rest`	The HTTPS management & operator/billing API.
gRPC API	`grpc`	The gRPC management API.
WebSocket API	`websocket`	The live-events / streaming API.
Metrics	`metrics`	The Prometheus `/metrics` endpoint.
DNS	`dns`	The built-in DNS server.
DHCP	`dhcp`	The built-in DHCP server.

The verb-first form works too — service off rest, service on dns — and -h on the group shows the help.

A few notes:

A change takes effect on the next agent restart. After toggling a service,

apply it with sudo systemctl restart cenvero-stratum, then re-check with service status.

The local control socket can never be disabled — it is how cenvero-str-ctl

reaches the agent (including the command you just ran). Trying to turn it off is refused with an explanation.

Disabling the REST API cuts off remote callers. The REST API also serves the

operator/billing API (tenant suspend/resume/limit). Turning it off is allowed — it is your call — but the command warns you first; the local socket and the panel are unaffected.

Under the hood, service <name> on|off writes the matching off-switch into the same operator local-overrides file that config set uses (see below), so a panel re-sync never clobbers your choice.

Changing settings

config set adjusts a node's operational settings on a running node. It records your change in an operator local-override file that takes precedence over the panel-delivered config and survives a re-sync — it never edits the signed node config itself.

# Inspect the active configuration (paths, ports, API settings, services)
cenvero-str-ctl config show

# Change a setting, then restart to apply it
sudo cenvero-str-ctl config set api_rate_limit 2000
sudo systemctl restart cenvero-stratum

The key names match what you see in config show, so the same name round-trips between the two. Commonly-set keys include:

Key	Meaning
`log_level`	Agent log level: `debug`, `info`, `warn`, `error`
`api_bind_address`	IP the REST/gRPC/WebSocket APIs listen on (e.g. `0.0.0.0` or a management IP)
`api_rate_limit` / `api_rate_burst`	API requests per minute per key, and the burst allowance (`0` rate disables the limiter)
`api_allowed_ips`	Comma-separated IPs/CIDRs allowed to reach the API (empty = allow all)
`port_rest`, `port_grpc`, `port_websocket`	API listen ports
`metrics_bind_addr`	Prometheus metrics bind address `host:port`
`dns_listen_addr`, `dns_allowed_clients`	Built-in DNS server bind address and allowed clients

Run cenvero-str-ctl config set -h for the full whitelist and per-key help.

What you cannot set locally. Identity, signing, and cluster-secret fields — node_id, license_server, api_token, gateway_shared_key, and TLS/cluster material — are not settable with config set by design. They are panel-assigned or provisioned. Reaching for one returns a clear message telling you where it is managed instead (e.g. TLS material is handled by cenvero-str-ctl tls …, and the API token is rotated via the installer/panel).

Every config set change applies on the next restart. It is saved immediately, but the running agent only picks it up when it restarts.

Updates

Updates are pull-based: the agent polls a signed manifest from the panel and, when a newer fully-verified release for its channel is available, downloads, verifies, and applies it on its own. You can check or trigger this manually:

# What's installed now
cenvero-str-ctl status

# Ask the agent to check the manifest for a newer release (no download)
cenvero-str-ctl update check

# Apply an available update now (otherwise it applies on the next poll)
sudo cenvero-str-ctl update apply

update check reports the current version, the latest on offer for your license channel, and whether an update is available — without downloading anything. If the panel is unreachable it says so plainly rather than failing hard. update apply performs the pull, checksum + publisher-signature verification, an atomic in-place swap, and a watchdog-supervised restart, with self-rollback if the post-restart health check fails. See Upgrades for the full model (channels, downgrade protection, and rollback).

A node in the Frozen license state keeps running its current version but cannot pull updates until the license is renewed.

Health checks

The agent continuously self-checks core subsystems (disk space, memory, the database, the bridges, and the in-kernel data plane) and attempts an automatic repair when one fails. Inspect or force a run:

# Latest result for every health check
cenvero-str-ctl heal status

# Force an immediate run now
sudo cenvero-str-ctl heal check

Troubleshooting

The agent won't start

Read the journal — it almost always names the cause:

bash
   journalctl -u cenvero-stratum -n 100 --no-pager
   systemctl status cenvero-stratum

Confirm you are running as root and the host network came up:

bash
   systemctl status cenvero-str-network


   The network unit must succeed first — it creates the bridges the agent attaches
   to. If it failed, fix the host networking and

sudo systemctl restart cenvero-str-network

.
Inspect the config the agent is loading:

bash
   sudo cenvero-str-ctl config show


   A config that fails to decode falls back to defaults with a note in the output.
   A signed config that fails verification is rejected — re-fetch a fresh
   panel-signed config rather than hand-editing the file.

Check disk space and the data directory (/var/lib/cenvero-str/). A full disk


   stops the agent from writing its state; the health check reclaims rotated logs
   but a genuinely full volume needs operator action.

cenvero-str-ctl `says the agent is unreachable (exit code 3)`


The CLI talks to the agent over its local socket. If commands report the agent is
unreachable, the agent process is not running — start it and re-check:
sudo systemctl start cenvero-stratum
cenvero-str-ctl status
License is not active
Check the installed license and this machine's hardware identity:
sudo cenvero-str-ctl license status
Expired / in grace / frozen — renew it. Existing traffic keeps running even
  when frozen; only new or changing operations are blocked. Renew with

sudo cenvero-str-ctl license renew

 (or fetch a fresh one after renewing in
  your account). See Licensing for the warn → grace → freeze
  model.
Not yet activated — activate this machine, then confirm it in your account:

bash
  sudo cenvero-str-ctl license activate CNVR-XXXX-XXXX-XXXX-XXXX

Hardware-ID unavailable — license status shows the machine's hardware ID.


  The binding requires firmware/hardware identifiers that are not exposed inside
  generic virtual machines, so a node must be bare metal. If the ID shows as
  unavailable, you are likely running in an unsupported virtualized environment.
Wrong release channel — a stable license runs only stable builds and a
  pre-release license runs only beta/RC builds; a mismatch fails closed. Make sure
  the build you installed matches your license's channel.
TLS certificate is pending
After install the agent generates its own certificate; if a customer-approval or
domain step is outstanding the certificate can show as pending. Confirm what the
agent is using and re-issue if needed:
sudo cenvero-str-ctl config show          # shows the TLS cert/key paths in use
sudo cenvero-str-ctl tls -h               # certificate management commands

TLS material is managed by the certificate manager (cenvero-str-ctl tls …), notconfig set. Adding or removing a SAN domain re-generates the certificate.


Node is not registered
Registration binds the node to its license and mints its node token. If a node
never registered, re-check connectivity to the panel and the license, then let it
re-register:
cenvero-str-ctl status                    # shows license + registration state
sudo cenvero-str-ctl license activate CNVR-XXXX-XXXX-XXXX-XXXX
Registration needs outbound HTTPS to your management/license server and a valid
license. If the panel is unreachable, the agent keeps retrying — fix connectivity
and it completes on its own.
"REST API disabled" — set an API token
If a client gets a REST API disabled (or 401/unauthorized) response, two
settings govern access:

Is the service enabled? Confirm with cenvero-str-ctl service status. If

rest

 is off, turn it back on and restart:

bash
   sudo cenvero-str-ctl service rest on
   sudo systemctl restart cenvero-stratum


Is an API token set? The REST/gRPC/WebSocket APIs require the bearer token
   to be configured.

config show

 reports Token set: yes/no. The token is a
   credential, so it is not set with

config set

 — it is provisioned by the
   installer (which generates one and prints it) or rotated via the panel. If no
   token is set, re-run the installer / fetch a panel-signed config to provision
   one, then restart the agent.
Where to look first
Symptom First check
Agent down / crash-looping journalctl -u cenvero-stratum -n 100
CLI "agent unreachable" systemctl status cenvero-stratum
License problems cenvero-str-ctl license status
A service not answering cenvero-str-ctl service status
Settings not taking effect Did you systemctl restart cenvero-stratum?
Overall health cenvero-str-ctl heal status`
Next steps
CLI Reference — the full command surface.
Configuration — the node config and what it controls.
Licensing — the enforcement state machine.
Upgrades — how updates are pulled and verified.

Operations & Troubleshooting

Systemd units

Logs

Network services: on / off

Changing settings

Updates

Health checks

Troubleshooting

The agent won't start

cenvero-str-ctl says the agent is unreachable (exit code 3)

License is not active

TLS certificate is pending

Node is not registered

"REST API disabled" — set an API token

Where to look first

Next steps

cenvero-str-ctl `says the agent is unreachable (exit code 3)`