Exclusive Access · Invitation Only

Snapshots, Backups and Schedules

A machine's boot disk can be snapshotted, backed up and restored, like any data volume. This page covers the boot disk, snapshots, backups, restores, the schedules that take backups for you, and off-site copies. Each action is shown in the console, on the command line and over the API.

Part of Compute, in early access. Snapshots, backups and schedules come with a Compute licence (Licensing). Changing them needs a licence that is not frozen; reading them always works.

The boot disk

Every machine has a boot disk: the disk it starts from. The node keeps it as a volume with the role boot. It is listed with the machine's other disks, and volume list shows it with the machine's id. On the command line, vm disks <id> lists all of a machine's disks, its boot disk among them:

cenvero-str-ctl vm disks vm-3f9a1c2e
  • It stays with its machine. A boot disk cannot be detached from its machine, and it cannot be deleted while the machine holds it. The node refuses both with "a machine's boot disk is kept with the machine; back it up instead". To remove the disk, delete the machine.
  • It is resized from its machine. Grow it with vm resize while the machine is stopped (Lifecycle).
  • Machines made before this release get their boot disk recorded the first time the node starts after the upgrade. Only the node's own records change: no file moves, and no machine is restarted.

What a delete keeps

Deleting a machine removes its disk's current data and its snapshots. Its **backups stay**, as restore points you can restore into a new volume. The boot disk is then listed with no machine. A released boot disk takes no new snapshot, no new backup, no in-place restore and no schedule. The node answers with "this boot disk's machine was deleted; its backups can be restored into a new volume".

A released boot disk is removed when its last backup is deleted. Until then, deleting it is refused with "this boot disk still has backups; delete them first". A machine that had no backups leaves no boot disk behind.

Snapshots

A snapshot keeps a disk's state at one moment, on the node. Going back to one undoes everything written since. A snapshot is not a backup: it lives on the same disk as the volume. Take a backup to keep a copy apart from it on the node.

Taking a snapshot

  • In the console, open the machine's Snapshots tab and choose Take a snapshot. Give it a name, or leave the name empty for one with the date and time. Quiet the file systems first (recommended) is ticked by default.
  • Over the API, POST /api/v1/vms/{id}/snapshots takes {"name", "quiesce"}, and both may be left out. quiesce defaults to true.
  • On the command line, vm snapshot create <id> takes a snapshot of a machine's boot disk, and snapshot create <volume> takes one of any volume. Find a volume's id with volume list, where the boot disk's role is boot:
cenvero-str-ctl vm snapshot create vm-3f9a1c2e --name before-upgrade
cenvero-str-ctl volume list
cenvero-str-ctl snapshot create vol-7d4e2f10 --name before-upgrade

How the snapshot is taken depends on the machine:

The machineIts guest agent answersHow the snapshot is takenconsistency
RunningYesThe guest's file systems are frozen for the moment it is takenquiesced
RunningNoAs the disk is, while the machine runscrash-consistent
StoppedNot applicableAs the disk isoffline

Pass --no-quiesce (or "quiesce": false) to take a snapshot without freezing the file systems, even when the guest agent answers. For a moment after the node starts following an upgrade, a machine whose boot disk is not yet recorded answers "this machine's boot disk is being recorded; try again in a moment".

Going back to a snapshot, and deleting one

Going back to a snapshot discards everything written to the disk since it was taken. Nothing may use the disk while it happens. A disk attached to a machine needs that machine stopped; a disk that is detached can be reverted as it is. Deleting a snapshot merges it back into the disk, and no data is lost.

  • In the console, both are on the disk's own page, opened from the Disks card on the machine's Snapshots tab.
  • On the command line: snapshot revert <id> --yes, which also takes --discard-newer to delete the snapshots taken after it, and snapshot delete <id>.
  • Over the API: POST /api/v1/snapshots/{id}/revert answers 200 with the volume, and DELETE /api/v1/snapshots/{id} deletes one.

Backups

A backup is a copy of a disk, kept on the node. A full backup copies the whole disk. An incremental backup copies only what changed since the earlier backup it builds on. A backup is listed as complete only after the node has checked the finished image against what it should be: the disk's size and, for an incremental backup, the backup it builds on. An unfinished copy never looks finished.

A backup of a running machine is taken while the machine runs. A stopped machine is backed up in full (an incremental backup of it is refused), and it cannot start until that copy has finished.

Taking a backup

  • In the console, open the machine's Backups tab and choose Back up now. The kind is Automatic (an incremental backup when one can be taken, and a full one when not), Full, or Incremental.
  • Over the API, POST /api/v1/vms/{id}/backups takes {"kind", "keep"}. kind is full, incremental, or left empty for automatic. An incremental backup needs the disk attached to a running machine: asked for on any other disk, it is refused, with a suggestion to take a full backup. An automatic backup of a stopped machine is a full one. keep sets how many backups taken by hand the disk keeps, and 0 keeps every one. A schedule's own keep is not changed by it. The answer is 202 with the backup and its task.
  • On the command line, vm backup create <id> backs up a machine's boot disk, and volume-backup create <volume> backs up any volume by its id:
cenvero-str-ctl vm backup create vm-3f9a1c2e --kind full --keep 7
cenvero-str-ctl vm backup list vm-3f9a1c2e
cenvero-str-ctl volume-backup create vol-7d4e2f10 --kind full --keep 7 --wait
cenvero-str-ctl volume-backup list --volume vol-7d4e2f10

The backups list shows a Made column: By hand, or On a schedule for a backup a schedule took.

Restoring

A restore takes a complete backup back into a volume. Two ways are offered.

Into a new volume is the default. The backup becomes a new volume, named by you or, when you leave the name empty, by the backup. The disk the backup came from is not changed, and no stop is needed. A machine can take the new volume as a data disk: volume attach <volume> --vm <machine>.

  • In the console, Restore on a complete backup opens the restore wizard. Its Restore to choice is A new volume (the default) or In place: the volume's contents are replaced.
  • Over the API: POST /api/v1/volume-backups/{id}/restore takes {"name"}, which may be left out, and answers 202 with the volume and its task.
  • On the command line: volume-backup restore <backup> [--name N] [--wait].

In place replaces the disk's contents with the backup. It is allowed only when:

  • the machine is stopped, or the volume is detached from its machine (otherwise the node asks you to stop the machine first);
  • the disk has no snapshots, or the restore may delete them. A disk with snapshots refuses an in-place restore until it is told to delete them (discard_snapshots), because the snapshots are of the data being replaced;
  • the backup is no bigger than the disk.
  • In the console, the wizard offers In place only when those hold. Otherwise it leaves that choice off and says why.
  • Over the API: POST /api/v1/volume-backups/{id}/restore takes {"in_place": true, "discard_snapshots"}, and discard_snapshots is true to delete the snapshots as part of the restore.
  • On the command line: volume-backup restore <backup> --in-place --yes, with --discard-snapshots when the disk has snapshots. --yes confirms that the restore replaces the disk's current data.

A released boot disk can only be restored into a new volume.

Schedules

A backup schedule takes backups of one volume by itself. A disk has at most one schedule, and a boot disk can have one like any volume.

Setting one

  • In the console, a schedule is set on the disk. For a machine's boot disk, the editor is on the machine's Backups tab. For a volume, it is on the volume's Backups tab, above its list. The fields are When, Kind, Restore points to keep and Enabled, with Save schedule, Run now and Remove schedule. Storage's Backups view lists the schedules of the volumes you can see, read only.
  • Over the API, PUT /api/v1/volumes/{id}/backup-schedule takes {"expression", "kind", "keep", "enabled"}. enabled defaults to true. The answer is 200 with the schedule.
  • On the command line, volume schedule set takes the three values, which are all required, and --enabled=false keeps the schedule without running it:
cenvero-str-ctl volume schedule set vol-7d4e2f10 --expression daily@02:30 --kind incremental --keep 7
cenvero-str-ctl volume schedule show vol-7d4e2f10
cenvero-str-ctl volume schedule run vol-7d4e2f10
cenvero-str-ctl volume schedule remove vol-7d4e2f10 --yes
cenvero-str-ctl schedule list

When it runs

The expression says when. Every time in it is UTC, and the node shows each next run in UTC.

ExpressionRunsExample
daily@HH:MMEvery day at that timedaily@02:30
weekly@DAY-HH:MMOnce a week, on that day, at that timeweekly@sun-03:00
every-NhEvery N hours: the first run N hours after the schedule is saved, then N hours after each run startsevery-6h

DAY is sun, mon, tue, wed, thu, fri or sat, or the whole day name, in any case. The shortest interval is one hour: every-1h is the most often a schedule runs. every-0h is refused with "a schedule runs at most once an hour". An expression the node does not read is refused with "the schedule's time is not recognized: use daily at HH:MM, weekly on a day at HH:MM, or every N hours".

Kind is full, or incremental. An incremental backup builds on the last backup it can. When it cannot, because nothing earlier can be built on, or the machine is stopped, the backup is a full one.

Restore points to keep is a number from 1 to 1000. Out of that range the node answers "keep must be between 1 and 1000 restore points". Retention is described below.

A node holds up to 256 schedules. Past that, adding one is refused with "this node already has 256 volume backup schedules; remove one before adding another". Removing a schedule stops its future backups. The backups it made stay, and they are kept until you delete them.

A suspended tenant's schedules keep running, because they protect the tenant's data. A schedule on a volume being deleted cannot be changed: "this volume is being deleted, so its backup schedule cannot be changed".

Moving a machine

A machine's schedules pause while it moves to another member, so no scheduled backup changes a disk the copy reads. They leave with the machine: the member it leaves removes them, and the new member has them paused, without their backup target. The schedule's editor on the new member shows the sentence "Its backups are paused, because its machine was moved to this member. Save the schedule to run them here again." Saving the schedule on the new member runs it again, with the backup target you choose there, or none. The backups the schedules made stay on the member that made them. A boot disk's schedule stops a move until you remove it (Moving machines between nodes).

What a run does

  • A run happens at its time. The node checks the schedules every minute. A run that was due while the node was down happens once, at the first check after it starts, however many runs were missed.
  • Only one run of a schedule goes at a time. A run that comes due while the previous backup is still going is skipped. Run now while a backup is going waits for it, and the schedule shows "the schedule is already running; its next run waits for it".
  • A disk that is busy with another operation is waited for, up to thirty minutes. After that the run ends busy: "the volume stayed busy for thirty minutes; this run did not start".
  • Before each run, the node checks that its storage pool has room for the data the backup reads, plus 1 GiB. When it has not, the run does not start and fails with "the storage pool is full: free some space, or add a pool".
  • A run never starts on a disk that is being deleted.
  • Each run is a task of type backup.scheduled, owned by the system, and its log names the schedule. task list --type backup.scheduled lists them. A run that fails records "the backup did not finish: the task has the reason" as its last error, and the reason is in the task's log.

The schedule records the last run's result: completed, failed, **did not start: the disk was busy, or skipped** when the schedule or its disk went away. Over the API these are ok, failed, busy and skipped, with last_run and last_error.

Retention

Restore points are counted in two groups: the ones a schedule made, and the ones taken by hand. Each group has its own number. A schedule keeps the newest Restore points to keep of its own restore points. The backups taken by hand keep the newest keep set by a backup taken by hand, or by volume-backup retention. When a backup completes, the node removes the oldest restore points of its own group beyond its number: a scheduled run removes only the restore points its schedule made, and a backup taken by hand removes only the ones taken by hand.

A schedule never removes a backup taken by hand. When more than the number of them exist, the oldest go after the next backup taken by hand completes. A keep taken by hand changes only the backups taken by hand, so choose it with them in mind.

Older restore points go a whole set at a time. An incremental backup is kept with the backups it builds on, so no restore point is left unable to restore.

volume-backup retention <volume> --keep N sets the number of backups taken by hand that the volume keeps, and 0 keeps every backup. It does not change a schedule.

Failures and notifications

A run that fails, or ends busy, makes the disk's backup_failed reading 1. It returns to 0 when a later run completes, or when the schedule is disabled. The reading is checked every 15 seconds, so a condition on it fires soon after a failure.

To be told, create an alert condition on backup_failed, then attach an email or a webhook action to it. A condition with no target watches every disk. With "target" set to a disk's volume id, it watches that disk.

cenvero-str-ctl alert condition add '{"metric_type":"backup_failed","operator":"gt","threshold":0}'
cenvero-str-ctl alert action add <condition_id> email [email protected]
cenvero-str-ctl alert action add <condition_id> webhook https://hooks.example.net/stratum

The first command prints the condition with its id. Use that id in the second. Alerts describes the actions, the cooldown and how an alert resolves. Each disk is its own alert, and its cooldown (cooldown_secs, 300 by default) runs separately from the others'.

The notice names the disk whose backup failed: "the scheduled backup of volume vol-7d4e2f10 failed". The task for the run has the reason.

Off-site copies

An off-site copy is a second copy of a backup, kept in a bucket on an S3-compatible storage service that you choose. The backup stays on the node as well. The node makes the copy only after it has checked the backup, so a backup that did not check is never sent.

Changing a target needs a licence that includes Storage and is not frozen. Reading always works.

Adding a target

A backup target is one bucket on a service that speaks the S3 protocol. Add one on the command line or over the API:

cenvero-str-ctl storage target create --name offsite --endpoint https://storage.example.net --bucket stratum-backups --access-key-id KEY_ID --secret-file /root/offsite.secret
cenvero-str-ctl storage target test <target>

Each target has these settings:

  • Name: 1 to 64 characters, so you can tell the targets apart.
  • Endpoint: the service's address. It starts with https:// and may name a port. Plain http:// is refused for a service that is not on the same machine. An address with a path, a query or a user name in it is refused too.
  • Bucket: the bucket's name, as the service has it.
  • Region: optional. It defaults to us-east-1.
  • Prefix: optional. The node keeps its files under this prefix, which defaults to cenvero-stratum/.
  • Path style: off by default. Turn it on (--path-style) for a service that takes the bucket in the address's path rather than in its host name.
  • Credentials: an access key id and a secret access key. See the next part.

Adding a target does not contact the service. Test it to check the address and the credentials. An address the node cannot use is refused with "the backup target's address, bucket, region or prefix is not valid: check each one and try again".

Credentials are write-only

The access key id and the secret access key are sent when you add a target, and again whenever you change them. Nothing reads them back. A target shows only two things about its credentials:

  • key_hint: the last four characters of the key id, written as …abcd;
  • secret: set or not set.

To change either credential, give both together: storage target set <target> --access-key-id KEY_ID --secret-file FILE. Giving one without the other is refused with "give the access key id and the secret access key together, or leave both out".

The secret is never a command-line argument. Give it with --secret-file, which reads the contents of a file that holds it, with any trailing line break removed, or with --secret-stdin, which reads one line from standard input. Either way the secret does not appear in the command line or in the process list. The only value that can be read out of the node is a target's key, through the export described below. A target added or changed with --node sends its credentials to that server over the cluster's encrypted channel, which only the cluster's own servers can use.

Testing a target

The test action, storage target test <target> on the command line or POST /api/v1/storage/targets/{id}/test over the API, writes a small test file under the target's prefix, reads it back and removes it. Its result is kept as the target's last test, with the time it ran. A test that fails says why:

  • "the backup target refused the credentials"
  • "the backup target could not be reached; check its address and credentials"

Naming a target

A copy is sent only to a target that a backup or a schedule names. On the command line the target is named with --target, and over the API with "target" in the request.

  • A backup taken by hand of a machine's boot disk names its target with "target" in POST /api/v1/vms/{id}/backups, or with --target on vm backup create.
  • A backup taken by hand of any volume, a data volume included, names its target with "target" in POST /api/v1/volumes/{id}/backups, or with --target on volume-backup create.
  • A schedule names its target with "target" in PUT /api/v1/volumes/{id}/backup-schedule, or with --target on volume schedule set. Every backup the schedule takes is copied to that target. This works for any volume, a data volume included. Setting a schedule replaces it as a whole, so a schedule set without a target has none, and its backups are not copied.

A target the node does not have is refused with "no backup target has that id".

What a copy is

Once a backup has been checked on the node, the node makes its copy:

  1. The backup is encrypted before it leaves the node. Each backup has a key of its own, and that key is protected by the target's key. The target's key is protected by the node key, which is described in Losing the node key.
  2. The encrypted copy is sent to the bucket as one file, named by the backup's id, in a folder for the node that made it, under the target's prefix. Nodes that share a bucket and a prefix keep their copies apart.
  3. The node checks the copy on the target. It sends the copy in pieces. Afterwards the target must hold every piece that was sent, and the stored file must have the length that was sent.
  4. A small record of the copy is written beside the file. The copy is complete only when the record is on the target too. A copy that does not pass every check is removed from the target, and it is not kept.

The bucket holds encrypted data only. Without the target's key, nothing in it can be read. The file names show backup ids, never the names of volumes or machines. The test action's file holds no backup data, and it is removed at once.

A backup's copy has one of four states, shown as target_state in the API:

  • uploading: the copy is being sent. The backup cannot be deleted until the copy has finished.
  • complete: the copy is on the target and has been checked.
  • failed: the copy was not kept. See the next part.
  • removing: the backup was deleted, and its copy is being removed from the target.

A backup's task ends when its copy has been checked, so a task that is still running is still sending the copy.

When a copy fails

A copy fails when the target refuses the credentials, cannot be reached, or does not keep what was sent. It also fails when this node cannot read the target's key, or when the storage pool has no room for the encrypted copy. When a copy fails:

  • The backup stays on the node, as it was. It is marked failed, with the reason in target_error. A target does not keep a copy that failed its check: "the backup could not be checked after upload; it is not kept on the target".
  • A scheduled backup ends its run as failed. The schedule's last error reads "the backup did not finish: the task has the reason", and the task has the reason. The disk's backup_failed reading goes to 1, so an alert condition on it fires. See Failures and notifications.
  • A backup taken by hand shows the failure on its task and its copy. It does not raise the alert.
  • It is sent again. Once a day the node sends again each copy that failed, oldest first. If the target cannot be reached or refuses the credentials, the node stops there and tries again the next day.
  • Retention keeps it. A backup whose copy failed is not removed by retention. It stays on the node until its copy is sent or you delete it.
  • The next backup does not build on it. A backup with a target builds on an earlier backup only when the target holds that backup and all it builds on. Otherwise an automatic or scheduled backup is taken in full, and an incremental backup asked for by name is refused: "the backup target does not hold the backup this one would build on, so an incremental backup cannot be kept there: take a full backup".

A backup that fails before its copy starts holds nothing on the target, and can be deleted as any failed backup can.

An interruption is not a failure. If the agent stops or the node restarts during a copy, the copy starts again from the beginning at the next start, after the uploads it left open on the target are cancelled. Once a day the node also cancels its own uploads that have been open for more than a day.

Retention and deletion

  • Retention works as it does for local backups. A schedule keeps its newest restore points, and the copies of the restore points it removes are removed from the target too. Whole sets go together: a full backup and the incremental backups built on it.
  • Only what the target holds counts. Retention runs when a backup's copy has been checked or has failed, not before. A restore point counts towards the number to keep when the target holds it and all it builds on, and its last check did not fail. So a target that stops taking new copies keeps the copies it has, and the newest ones the number allows stay on it.
  • A local copy is pruned only after its copy is checked on the target. A backup whose copy is still being sent, or has failed, keeps its local copy. During a long outage these add up in the storage pool until the target returns or you delete them.
  • Deleting a backup deletes its copy. If the target will not remove the copy, the node keeps the backup's record until the copy is gone, and tries again at the next start and once a day.
  • A copy made before copies were kept in a folder for each node, which has no record, is never removed by retention or when its backup is deleted, so an operator can remove it by hand from the bucket once it is no longer needed.
  • A backup whose copy is still being sent cannot be deleted: "this backup is being copied to its backup target; try again when the copy has finished".

Restoring from a target

A restore reads the backup's local copy when the node has it, and checks it, as always. When the node no longer has the local copy, the restore reads the copy back from the target that holds it. You do not choose the target. The node finds it from the backup's record. A local copy that fails its check is replaced by the copy from the target, and stays on the node after the restore. A local copy that could not be read at that moment is not replaced: "this node could not read its copy of the backup just now, so nothing was restored: try again".

  • The node checks the copy as it reads it. Before the copy takes its place on the node, the node checks the copy's image: a complete backup must not name any other file behind it, and an incremental backup is joined to the copy it builds on, on this node. The node then checks the whole backup against the checksum recorded when the copy was sent.
  • Nothing is written to the new volume until those checks pass. A copy that fails them is refused: "the backup on its backup target could not be verified, so nothing was restored".
  • A restore needs the target on the node, with its credentials. If the copy is on neither the node nor its target, the restore is refused: "the backup's copy is not on this node and is not on its backup target".

To restore a copy on a node that did not make it, see Restoring on another node.

Copies on a target

storage target copies <target> on the command line, or GET /api/v1/storage/targets/{id}/copies over the API, lists the copies a target holds. The list is read from the copies on the target. It changes nothing on the target or on the node, and reading always works, even under a frozen licence. The node needs the target's credentials and key to read the list, so a target whose key is not on the node lists nothing, and says why.

Each copy shows its state:

  • restorable: it can be restored from this node.
  • incomplete: its data is missing or changed on the target, or a copy it builds on is not on the target, or cannot be restored from it.
  • unreadable: its record cannot be read with this target's key. The usual cause is the wrong key, and then the node says "this copy was made with another key: import the key it was made with, then try again". A record changed on the target is unreadable too.
  • refused: its record names something outside its own copy, so the node refuses it.

A copy that cannot be restored gives its reason in the list, in the same words the node gives when a restore of that copy is refused.

Checking copies

Each copy on a target is checked once a week. The checks of different copies are spread across the week, so the node never checks every copy at once. A check reads the copy back as far as is cheap to do: it confirms the length of the copy and the checksum the target reports for it, it reads the copy's record, and it reads back one part of the copy's data, chosen at random, which it checks with the target's key. A check changes nothing on the target, and it deletes nothing.

A copy whose check fails is marked check failed in its backup, with the reason in words. In the API these are target_check_state and target_check_error. The backup stays on the node, and its copy keeps its state. An alert can be raised for the failure: create an alert condition on copy_check_failed, as you would for a scheduled backup that fails, and attach an email or a webhook to it. A later check that passes clears the mark and resolves the alert.

To check one copy now, run cenvero-str-ctl storage target copy-check <target> <copy>. The answer says whether the copy passed, and a copy that failed says why. A target that cannot be reached, or that refuses the credentials, cannot judge the copy: the answer says so, and nothing is recorded.

Restoring on another node

A copy can be restored on a node that did not make it, for example when the node that made the copies is lost. The new node needs the target first:

  1. Add the target, with the same endpoint, bucket, prefix and credentials as the old node's target, using storage target create.
  2. Import the target's key from the export the old node took, with storage target key-import (see Exporting and importing a target key).
  3. List the copies with storage target copies, and restore one with storage target restore.
sudo cenvero-str-ctl storage target create --name offsite --endpoint https://storage.example.net --bucket stratum-backups --prefix cenvero-stratum/ --access-key-id KEY_ID --secret-file /root/offsite.secret
sudo cenvero-str-ctl storage target key-import <target> --key-file /root/offsite.key --yes
sudo cenvero-str-ctl storage target copies <target>
sudo cenvero-str-ctl storage target restore <target> bk-3b8e0a71 --name restored-data

The restore writes a new volume on the new node, as a task. The volume belongs to the operator, since a copy names no tenant. Its name is --name; when that is left out, it is built from the copy's volume name and the copy's id. It is kept in the default pool, or in the pool that --pool names. A restore is a change, so it needs a licence that includes Storage and is not frozen.

The copies the restore builds on are fetched from the target as well, and each one is checked before the new volume is written. While they are fetched, the node holds the room they take in the pool each copy is written to. Another create, copy, restore or backup that does not fit beside that room is refused with "the storage pool is full: free some space, or add a pool" until the fetch has ended. The room is given back when the fetch ends, whether it succeeded or failed. The node keeps no record of a copy it restored from another node, and it removes the fetched copies once the restore has finished.

A copy restores into the pool you name, or into the default pool when you name none. A copy that does not check is refused before anything reaches the new volume, whichever pool it is restored into.

Deleting a target

storage target delete <target> --yes removes a target's record. It is refused while a schedule names the target, or while a backup has a copy on it, even a failed one: "this backup target is still used by volume backup schedules or backups: remove them first". Removing a schedule does not delete its backups. To delete a target, remove its schedules, then delete the backups that have copies on it.

Exporting and importing a target key

Each target has a key of its own, and the node uses it to protect the target's copies. The key can be exported, and an exported key can be imported on a node that needs it:

  • Over the API, POST /api/v1/storage/targets/{id}/key-export returns the key in its answer, base64-encoded. It needs your password confirmed in the last five minutes in the web console, or the node's own API token. Any other token is refused with 403.
  • On the command line, run sudo cenvero-str-ctl storage target key-export <target> --yes on the node that holds the target. It cannot be sent to another node with --node.
  • Over the API, POST /api/v1/storage/targets/{id}/key-import takes the key the export printed, and sets the target's key from it. It needs the same confirmation as the export.
  • On the command line, run sudo cenvero-str-ctl storage target key-import <target> --key-file FILE --yes, or give the key on standard input with --key-stdin. It cannot be sent to another node with --node.

Every export is audited. Every import is audited too. The key is written to no log, task or audit record, and an import does not show the key again.

An import sets the target's key, and changes nothing when the key is the one the target has already. When the node can read the target's current key, a different key is refused while this node holds copies on the target: "this target already has its own key, and copies made with it are kept here: the key cannot be replaced while they are". A replaced key cannot read those copies. A key the node can no longer read, after it lost its node key, is replaced at once. A key that is not the right length is refused with "that is not a backup target's key: paste the whole key that the key export printed".

Take the export when the target is set up, because an export is only possible while the node still has its node key. Keep it where only the operator can read it, and not only on the node: a key kept only on the node is lost with it. Anyone who has the key and access to the bucket can read the copies. Without an export, no node can read the copies once the node key is lost (see Losing the node key).

Losing the node key

The node key is a secret the node keeps in its data directory. It protects the target credentials, the targets' keys, and the machines' user-data and SSH keys. These are kept sealed under it, never in plain text. The node makes the key the first time it needs one.

The node key is lost when its file is removed or cannot be read, for example when a node is rebuilt without its data directory. Losing it has these effects:

  • Target credentials and keys cannot be read. A target's test, its copies and its restores are refused with "this node cannot read the credentials of this backup target: enter them again before it is used". Enter the credentials again with storage target set <target> --access-key-id KEY_ID --secret-file FILE. This makes the node key again, when the node has none. The target's key is sealed under the node key too, and entering the credentials does not restore it: import it from an export, as described in Exporting and importing a target key.
  • Copies are not sent until the target's credentials and key are both on the node. Each copy fails with the same sentence, and its backup stays on the node.
  • First-boot settings cannot be read. A machine whose user-data or SSH keys cannot be read does not start, and cannot be cloned, until they are set again: "the first-boot settings of this machine cannot be read on this node: set them again before it starts". Set them again with vm cloud-init set <machine> --ssh-key-file FILE --user-data-file FILE, giving the SSH keys and the user-data file the machine should have. Once they are saved, the machine can start again. A running machine keeps running, and the refusal applies only when it next starts. The hostname and the DNS servers are not sealed, so they are not affected. See First boot.
  • Copies already in the bucket stay there. They can be read again on this node once the target's key is imported from an export taken before the loss. Without that export, no node can read them.
  • Volumes and local backups are not affected. They are not sealed under the node key.

Recovery. The node makes a new node key the next time it must seal a value, and anything saved from then on is sealed under it. Values sealed under the lost key stay unreadable, so enter them again as described above. A key file that is still there but cannot be read is never replaced: the node refuses to save secrets until it can read the file again.

Each target needs both steps: its credentials entered again, and its key imported from an export taken earlier. Without an export, its copies cannot be read on any node, and the target cannot send new copies. Create a new target to keep making copies.

Backups on another node. A backup that was never sent to a target can be restored only on the node that holds it. A backup sent to a target can be restored on another node, as described in Restoring on another node. A restore makes a volume, not a machine. A backup holds a disk, not the machine's first-boot settings. So none come back with a restore, and a machine's settings are set on the node that runs it.

Storage pools

A pool is a place on the node where volumes and backups are kept. Every node has a default pool, which holds the machines' disks and every volume and backup made before pools existed. You can add directory pools, and block pools on storage the host already has. Storage pools describes both kinds, what a block pool needs on the host, and what works on each.

  • A volume, a machine or a restore takes its pool when it is made: volume create --pool, vm create --pool, and volume-backup restore <id> --pool. Leave --pool out for the default pool. A machine's boot disk stays on the pool the machine was made on, and moving one is not possible yet.
  • A pool that still holds volumes cannot be deleted, and the default pool cannot be deleted at all.
  • A backup of a volume on a block pool is a full backup, and it is written to the default pool. That pool needs room for the whole volume.
  • Before a create, a restore, a copy or a scheduled backup starts, the node checks that the pool has the room the work needs. A full pool refuses the work with "the storage pool is full: free some space, or add a pool". A pool whose free space the node cannot read refuses it too, with "the storage pool's free space cannot be read right now, so nothing was copied: try again later". Nothing is written in either case.

Over the API

MethodPathAnswer
GET/api/v1/vms/{id}/disks200 with the machine's disks, the boot disk among them
GET/api/v1/vms/{id}/snapshots200 with the boot disk's snapshots
POST/api/v1/vms/{id}/snapshots202 with the snapshot and its task
GET/api/v1/vms/{id}/backups200 with the boot disk's backups
POST/api/v1/vms/{id}/backups202 with the backup and its task
POST/api/v1/volume-backups/{id}/restore202 with the new or restored volume and its task
GET/api/v1/volumes/{id}/backup-schedule200 with the schedule, or 404 when there is none
PUT/api/v1/volumes/{id}/backup-schedule200 with the schedule
DELETE/api/v1/volumes/{id}/backup-schedule200
POST/api/v1/volumes/{id}/backup-schedule/run202 with the schedule and its task
GET/api/v1/backup-schedules200 with the schedules of the volumes you can see

Every route here is operator-only: a tenant-scoped key cannot reach them. Reading needs VM.Audit for a machine's routes and Storage.Audit for a volume's schedule. Taking snapshots and backups needs VM.Snapshot and VM.Backup. Restoring and setting or running a schedule need Storage.Backup.

See also

↓ This page as JSON ↓ All documentation as JSON