{
    "product": "Cenvero Stratum",
    "generated_at": "2026-11-16T13:19:22+00:00",
    "format": "cenvero-docs-v1",
    "document_count": 1,
    "documents": [
        {
            "slug": "compute/backups",
            "title": "Snapshots, Backups and Schedules",
            "category": "Compute",
            "url": "https://stratum.cenvero.com/docs/compute/backups",
            "headings": [
                {
                    "level": 1,
                    "text": "Snapshots, Backups and Schedules"
                },
                {
                    "level": 2,
                    "text": "The boot disk"
                },
                {
                    "level": 3,
                    "text": "What a delete keeps"
                },
                {
                    "level": 2,
                    "text": "Snapshots"
                },
                {
                    "level": 3,
                    "text": "Taking a snapshot"
                },
                {
                    "level": 3,
                    "text": "Going back to a snapshot, and deleting one"
                },
                {
                    "level": 2,
                    "text": "Backups"
                },
                {
                    "level": 3,
                    "text": "Taking a backup"
                },
                {
                    "level": 3,
                    "text": "Restoring"
                },
                {
                    "level": 2,
                    "text": "Schedules"
                },
                {
                    "level": 3,
                    "text": "Setting one"
                },
                {
                    "level": 3,
                    "text": "When it runs"
                },
                {
                    "level": 3,
                    "text": "Moving a machine"
                },
                {
                    "level": 3,
                    "text": "What a run does"
                },
                {
                    "level": 3,
                    "text": "Retention"
                },
                {
                    "level": 3,
                    "text": "Failures and notifications"
                },
                {
                    "level": 2,
                    "text": "Off-site copies"
                },
                {
                    "level": 3,
                    "text": "Adding a target"
                },
                {
                    "level": 3,
                    "text": "Credentials are write-only"
                },
                {
                    "level": 3,
                    "text": "Testing a target"
                },
                {
                    "level": 3,
                    "text": "Naming a target"
                },
                {
                    "level": 3,
                    "text": "What a copy is"
                },
                {
                    "level": 3,
                    "text": "When a copy fails"
                },
                {
                    "level": 3,
                    "text": "Retention and deletion"
                },
                {
                    "level": 3,
                    "text": "Restoring from a target"
                },
                {
                    "level": 3,
                    "text": "Copies on a target"
                },
                {
                    "level": 3,
                    "text": "Checking copies"
                },
                {
                    "level": 3,
                    "text": "Restoring on another node"
                },
                {
                    "level": 3,
                    "text": "Deleting a target"
                },
                {
                    "level": 3,
                    "text": "Exporting and importing a target key"
                },
                {
                    "level": 2,
                    "text": "Losing the node key"
                },
                {
                    "level": 2,
                    "text": "Storage pools"
                },
                {
                    "level": 2,
                    "text": "Over the API"
                },
                {
                    "level": 2,
                    "text": "See also"
                }
            ],
            "word_count": 7059,
            "markdown": "# Snapshots, Backups and Schedules\n\nA machine's boot disk can be snapshotted, backed up and restored, like any data\nvolume. This page covers the boot disk, snapshots, backups, restores, the schedules\nthat take backups for you, and off-site copies. Each action is shown in the console,\non the command line and over the API.\n\n> Part of [Compute](/docs/compute/overview), in early access. Snapshots, backups\n> and schedules come with a Compute licence ([Licensing](/docs/licensing)).\n> Changing them needs a licence that is not frozen; reading them always works.\n\n## The boot disk\n\nEvery machine has a **boot disk**: the disk it starts from. The node keeps it as a\nvolume with the role `boot`. It is listed with the machine's other disks, and\n`volume list` shows it with the machine's id. On the command line, `vm disks <id>`\nlists all of a machine's disks, its boot disk among them:\n\n```bash\ncenvero-str-ctl vm disks vm-3f9a1c2e\n```\n\n- **It stays with its machine.** A boot disk cannot be detached from its machine,\n  and it cannot be deleted while the machine holds it. The node refuses both with\n  \"a machine's boot disk is kept with the machine; back it up instead\". To remove\n  the disk, delete the machine.\n- **It is resized from its machine.** Grow it with `vm resize` while the machine is\n  stopped ([Lifecycle](/docs/compute/lifecycle)).\n- **Machines made before this release** get their boot disk recorded the first\n  time the node starts after the upgrade. Only the node's own records change:\n  no file moves, and no machine is restarted.\n\n### What a delete keeps\n\nDeleting a machine removes its disk's current data and its snapshots. Its **backups\nstay**, as restore points you can restore into a new volume. The boot disk is then listed\nwith no machine. A released boot disk takes no new snapshot, no new backup, no\nin-place restore and no schedule. The node answers with \"this boot disk's machine\nwas deleted; its backups can be restored into a new volume\".\n\nA released boot disk is removed when its last backup is deleted. Until then,\ndeleting it is refused with \"this boot disk still has backups; delete them first\".\nA machine that had no backups leaves no boot disk behind.\n\n## Snapshots\n\nA **snapshot** keeps a disk's state at one moment, on the node. Going back to one\nundoes everything written since. A snapshot is not a backup: it lives on the same\ndisk as the volume. Take a backup to keep a copy apart from it on the node.\n\n### Taking a snapshot\n\n- **In the console**, open the machine's **Snapshots** tab and choose **Take a\n  snapshot**. Give it a name, or leave the name empty for one with the date and\n  time. **Quiet the file systems first (recommended)** is ticked by default.\n- **Over the API**, `POST /api/v1/vms/{id}/snapshots` takes `{\"name\", \"quiesce\"}`,\n  and both may be left out. `quiesce` defaults to `true`.\n- **On the command line**, `vm snapshot create <id>` takes a snapshot of a machine's\n  boot disk, and `snapshot create <volume>` takes one of any volume. Find a volume's\n  id with `volume list`, where the boot disk's role is `boot`:\n\n```bash\ncenvero-str-ctl vm snapshot create vm-3f9a1c2e --name before-upgrade\ncenvero-str-ctl volume list\ncenvero-str-ctl snapshot create vol-7d4e2f10 --name before-upgrade\n```\n\nHow the snapshot is taken depends on the machine:\n\n| The machine | Its guest agent answers | How the snapshot is taken | `consistency` |\n|---|---|---|---|\n| Running | Yes | The guest's file systems are frozen for the moment it is taken | `quiesced` |\n| Running | No | As the disk is, while the machine runs | `crash-consistent` |\n| Stopped | Not applicable | As the disk is | `offline` |\n\nPass `--no-quiesce` (or `\"quiesce\": false`) to take a snapshot without freezing\nthe file systems, even when the guest agent answers. For a moment after the node\nstarts following an upgrade, a machine whose boot disk is not yet recorded answers\n\"this machine's boot disk is being recorded; try again in a moment\".\n\n### Going back to a snapshot, and deleting one\n\nGoing back to a snapshot discards everything written to the disk since it was\ntaken. Nothing may use the disk while it happens. A disk attached to a machine needs\nthat machine stopped; a disk that is detached can be reverted as it is. Deleting a\nsnapshot merges it back into the disk, and no data is lost.\n\n- **In the console**, both are on the disk's own page, opened from the **Disks**\n  card on the machine's Snapshots tab.\n- **On the command line:** `snapshot revert <id> --yes`, which also takes\n  `--discard-newer` to delete the snapshots taken after it, and `snapshot delete <id>`.\n- **Over the API:** `POST /api/v1/snapshots/{id}/revert` answers **200** with the\n  volume, and `DELETE /api/v1/snapshots/{id}` deletes one.\n\n## Backups\n\nA **backup** is a copy of a disk, kept on the node. A **full** backup copies the\nwhole disk. An **incremental** backup copies only what changed since the earlier\nbackup it builds on. A backup is listed as complete only after the node has checked\nthe finished image against what it should be: the disk's size and, for an incremental\nbackup, the backup it builds on. An unfinished copy never looks finished.\n\nA backup of a running machine is taken while the machine runs. A stopped machine is\nbacked up in full (an incremental backup of it is refused), and it cannot start until\nthat copy has finished.\n\n### Taking a backup\n\n- **In the console**, open the machine's **Backups** tab and choose **Back up now**.\n  The kind is **Automatic** (an incremental backup when one can be taken, and a\n  full one when not), **Full**, or **Incremental**.\n- **Over the API**, `POST /api/v1/vms/{id}/backups` takes `{\"kind\", \"keep\"}`. `kind`\n  is `full`, `incremental`, or left empty for automatic. An incremental backup needs\n  the disk attached to a running machine: asked for on any other disk, it is refused,\n  with a suggestion to take a full backup. An automatic backup of a stopped machine is\n  a full one. `keep` sets how many backups taken by hand the disk keeps, and `0` keeps\n  every one. A schedule's\n  own keep is not changed by it. The answer is **202** with the backup and its task.\n- **On the command line**, `vm backup create <id>` backs up a machine's boot disk,\n  and `volume-backup create <volume>` backs up any volume by its id:\n\n```bash\ncenvero-str-ctl vm backup create vm-3f9a1c2e --kind full --keep 7\ncenvero-str-ctl vm backup list vm-3f9a1c2e\ncenvero-str-ctl volume-backup create vol-7d4e2f10 --kind full --keep 7 --wait\ncenvero-str-ctl volume-backup list --volume vol-7d4e2f10\n```\n\nThe backups list shows a **Made** column: **By hand**, or **On a schedule** for a\nbackup a schedule took.\n\n### Restoring\n\nA restore takes a complete backup back into a volume. Two ways are offered.\n\n**Into a new volume** is the default. The backup becomes a new volume, named by\nyou or, when you leave the name empty, by the backup. The disk the backup came from\nis not changed, and no stop is needed. A machine can take the new volume as a data\ndisk: `volume attach <volume> --vm <machine>`.\n\n- In the console, **Restore** on a complete backup opens the restore wizard. Its\n  **Restore to** choice is **A new volume (the default)** or **In place: the volume's\n  contents are replaced**.\n- Over the API: `POST /api/v1/volume-backups/{id}/restore` takes `{\"name\"}`, which\n  may be left out, and answers **202** with the volume and its task.\n- On the command line: `volume-backup restore <backup> [--name N] [--wait]`.\n\n**In place** replaces the disk's contents with the backup. It is allowed only when:\n\n- the machine is stopped, or the volume is detached from its machine (otherwise the\n  node asks you to stop the machine first);\n- the disk has no snapshots, or the restore may delete them. A disk with snapshots\n  refuses an in-place restore until it is told to delete them (`discard_snapshots`),\n  because the snapshots are of the data being replaced;\n- the backup is no bigger than the disk.\n\n- In the console, the wizard offers **In place** only when those hold. Otherwise it\n  leaves that choice off and says why.\n- Over the API: `POST /api/v1/volume-backups/{id}/restore` takes\n  `{\"in_place\": true, \"discard_snapshots\"}`, and `discard_snapshots` is `true` to\n  delete the snapshots as part of the restore.\n- On the command line: `volume-backup restore <backup> --in-place --yes`, with\n  `--discard-snapshots` when the disk has snapshots. `--yes` confirms that the\n  restore replaces the disk's current data.\n\nA released boot disk can only be restored into a new volume.\n\n## Schedules\n\nA **backup schedule** takes backups of one volume by itself. A disk has at most one\nschedule, and a boot disk can have one like any volume.\n\n### Setting one\n\n- **In the console**, a schedule is set on the disk. For a machine's boot disk, the\n  editor is on the machine's **Backups** tab. For a volume, it is on the volume's\n  **Backups** tab, above its list. The fields are **When**, **Kind**, **Restore\n  points to keep** and **Enabled**, with **Save schedule**, **Run now** and\n  **Remove schedule**. Storage's **Backups** view lists the schedules of the volumes\n  you can see, read only.\n- **Over the API**, `PUT /api/v1/volumes/{id}/backup-schedule` takes\n  `{\"expression\", \"kind\", \"keep\", \"enabled\"}`. `enabled` defaults to `true`. The\n  answer is **200** with the schedule.\n- **On the command line**, `volume schedule set` takes the three values, which are\n  all required, and `--enabled=false` keeps the schedule without running it:\n\n```bash\ncenvero-str-ctl volume schedule set vol-7d4e2f10 --expression daily@02:30 --kind incremental --keep 7\ncenvero-str-ctl volume schedule show vol-7d4e2f10\ncenvero-str-ctl volume schedule run vol-7d4e2f10\ncenvero-str-ctl volume schedule remove vol-7d4e2f10 --yes\ncenvero-str-ctl schedule list\n```\n\n### When it runs\n\nThe **expression** says when. Every time in it is UTC, and the node shows each\nnext run in UTC.\n\n| Expression | Runs | Example |\n|---|---|---|\n| `daily@HH:MM` | Every day at that time | `daily@02:30` |\n| `weekly@DAY-HH:MM` | Once a week, on that day, at that time | `weekly@sun-03:00` |\n| `every-Nh` | Every N hours: the first run N hours after the schedule is saved, then N hours after each run starts | `every-6h` |\n\n`DAY` is `sun`, `mon`, `tue`, `wed`, `thu`, `fri` or `sat`, or the whole day name,\nin any case. The shortest interval is one hour: `every-1h` is the most often a\nschedule runs. `every-0h` is refused with \"a schedule runs at most once an hour\".\nAn expression the node does not read is refused with \"the schedule's time is not\nrecognized: use daily at HH:MM, weekly on a day at HH:MM, or every N hours\".\n\n**Kind** is `full`, or `incremental`. An incremental backup builds on the last\nbackup it can. When it cannot, because nothing earlier can be built on, or the\nmachine is stopped, the backup is a full one.\n\n**Restore points to keep** is a number from 1 to 1000. Out of that range the node\nanswers \"keep must be between 1 and 1000 restore points\". Retention is described\nbelow.\n\nA node holds up to **256 schedules**. Past that, adding one is refused with \"this\nnode already has 256 volume backup schedules; remove one before adding another\".\nRemoving a schedule stops its future backups. The backups it made stay, and they are\nkept until you delete them.\n\nA suspended tenant's schedules keep running, because they protect the tenant's\ndata. A schedule on a volume being deleted cannot be changed: \"this volume is being\ndeleted, so its backup schedule cannot be changed\".\n\n### Moving a machine\n\nA machine's schedules pause while it moves to another member, so no scheduled backup changes a disk the copy reads. They leave with the machine: the member it leaves removes them, and the new member has them paused, without their backup target. The schedule's editor on the new member shows the sentence **\"Its backups are paused, because its machine was moved to this member. Save the schedule to run them here again.\"** Saving the schedule on the new member runs it again, with the backup target you choose there, or none. The backups the schedules made stay on the member that made them. A boot disk's schedule stops a move until you remove it ([Moving machines between nodes](/docs/clustering/live-migration)).\n\n### What a run does\n\n- A run happens at its time. The node checks the schedules every minute. A run that\n  was due while the node was down happens once, at the first check after it starts,\n  however many runs were missed.\n- Only one run of a schedule goes at a time. A run that comes due while the previous\n  backup is still going is skipped. **Run now** while a backup is going waits for it,\n  and the schedule shows \"the schedule is already running; its next run waits for it\".\n- A disk that is busy with another operation is waited for, up to thirty minutes.\n  After that the run ends busy: \"the volume stayed busy for thirty minutes; this run\n  did not start\".\n- Before each run, the node checks that its storage pool has room for the data the\n  backup reads, plus 1 GiB. When it has not, the run does not start and fails with \"the\n  storage pool is full: free some space, or add a pool\".\n- A run never starts on a disk that is being deleted.\n- Each run is a task of type `backup.scheduled`, owned by the system, and its log\n  names the schedule. `task list --type backup.scheduled` lists them. A run that\n  fails records \"the backup did not finish: the task has the reason\" as its last\n  error, and the reason is in the task's log.\n\nThe schedule records the last run's result: **completed**, **failed**, **did not start:\nthe disk was busy**, or **skipped** when the schedule or its disk went away. Over the\nAPI these are `ok`, `failed`, `busy` and `skipped`, with `last_run` and `last_error`.\n\n### Retention\n\nRestore points are counted in two groups: the ones a schedule made, and the ones\ntaken by hand. Each group has its own number. A schedule keeps the newest\n**Restore points to keep** of its own restore points. The backups taken by hand keep\nthe newest `keep` set by a backup taken by hand, or by `volume-backup retention`.\nWhen a backup completes, the node removes the oldest restore points of its own group\nbeyond its number: a scheduled run removes only the restore points its schedule made,\nand a backup taken by hand removes only the ones taken by hand.\n\n**A schedule never removes a backup taken by hand.** When more than the number of\nthem exist, the oldest go after the next backup taken by hand completes. A `keep`\ntaken by hand changes only the backups taken by hand, so choose it with them in mind.\n\nOlder restore points go a whole set at a time. An incremental backup is kept with the\nbackups it builds on, so no restore point is left unable to restore.\n\n`volume-backup retention <volume> --keep N` sets the number of backups taken by hand\nthat the volume keeps, and `0` keeps every backup. It does not change a schedule.\n\n### Failures and notifications\n\nA run that fails, or ends busy, makes the disk's `backup_failed` reading 1. It\nreturns to 0 when a later run completes, or when the schedule is disabled. The\nreading is checked every 15 seconds, so a condition on it fires soon after a failure.\n\nTo be told, create an alert condition on `backup_failed`, then attach an email or a\nwebhook action to it. A condition with no target watches every disk. With\n`\"target\"` set to a disk's volume id, it watches that disk.\n\n```bash\ncenvero-str-ctl alert condition add '{\"metric_type\":\"backup_failed\",\"operator\":\"gt\",\"threshold\":0}'\ncenvero-str-ctl alert action add <condition_id> email ops@example.net\ncenvero-str-ctl alert action add <condition_id> webhook https://hooks.example.net/stratum\n```\n\nThe first command prints the condition with its `id`. Use that id in the second.\n[Alerts](/docs/monitoring#alerts-tell-me-when-something-happens) describes the actions,\nthe cooldown and how an alert resolves. Each disk is its own alert, and its cooldown\n(`cooldown_secs`, 300 by default) runs separately from the others'.\n\nThe notice names the disk whose backup failed: \"the scheduled backup of volume\nvol-7d4e2f10 failed\". The task for the run has the reason.\n\n## Off-site copies\n\nAn **off-site copy** is a second copy of a backup, kept in a bucket on an\nS3-compatible storage service that you choose. The backup stays on the node as well.\nThe node makes the copy only after it has checked the backup, so a backup that did not\ncheck is never sent.\n\nChanging a target needs a licence that includes Storage and is not frozen. Reading\nalways works.\n\n### Adding a target\n\nA **backup target** is one bucket on a service that speaks the S3 protocol. Add one on\nthe command line or over the API:\n\n```bash\ncenvero-str-ctl storage target create --name offsite --endpoint https://storage.example.net --bucket stratum-backups --access-key-id KEY_ID --secret-file /root/offsite.secret\ncenvero-str-ctl storage target test <target>\n```\n\nEach target has these settings:\n\n- **Name**: 1 to 64 characters, so you can tell the targets apart.\n- **Endpoint**: the service's address. It starts with `https://` and may name a port.\n  Plain `http://` is refused for a service that is not on the same machine. An address\n  with a path, a query or a user name in it is refused too.\n- **Bucket**: the bucket's name, as the service has it.\n- **Region**: optional. It defaults to `us-east-1`.\n- **Prefix**: optional. The node keeps its files under this prefix, which defaults to\n  `cenvero-stratum/`.\n- **Path style**: off by default. Turn it on (`--path-style`) for a service that takes\n  the bucket in the address's path rather than in its host name.\n- **Credentials**: an access key id and a secret access key. See the next part.\n\nAdding a target does not contact the service. Test it to check the address and the\ncredentials. An address the node cannot use is refused with \"the backup target's\naddress, bucket, region or prefix is not valid: check each one and try again\".\n\n### Credentials are write-only\n\nThe access key id and the secret access key are sent when you add a target, and again\nwhenever you change them. Nothing reads them back. A target shows only two things about\nits credentials:\n\n- `key_hint`: the last four characters of the key id, written as `…abcd`;\n- `secret`: `set` or `not set`.\n\nTo change either credential, give both together: `storage target set <target>\n--access-key-id KEY_ID --secret-file FILE`. Giving one without the other is refused with\n\"give the access key id and the secret access key together, or leave both out\".\n\nThe secret is never a command-line argument. Give it with `--secret-file`, which reads the\ncontents of a file that holds it, with any trailing line break removed, or with\n`--secret-stdin`, which reads one line from standard input. Either way the secret does not\nappear in the command line or in the process list. The only value that can be read out of\nthe node is a target's key, through the export described below. A target added or changed\nwith `--node` sends its credentials to that server over the cluster's encrypted channel,\nwhich only the cluster's own servers can use.\n\n### Testing a target\n\nThe test action, `storage target test <target>` on the command line or `POST\n/api/v1/storage/targets/{id}/test` over the API, writes a small test file under the\ntarget's prefix, reads it back and removes it. Its result is kept as the target's last\ntest, with the time it ran. A test that fails says why:\n\n- \"the backup target refused the credentials\"\n- \"the backup target could not be reached; check its address and credentials\"\n\n### Naming a target\n\nA copy is sent only to a target that a backup or a schedule names. On the command line\nthe target is named with `--target`, and over the API with `\"target\"` in the request.\n\n- **A backup taken by hand** of a machine's boot disk names its target with `\"target\"` in\n  `POST /api/v1/vms/{id}/backups`, or with `--target` on `vm backup create`.\n- **A backup taken by hand** of any volume, a data volume included, names its target with\n  `\"target\"` in `POST /api/v1/volumes/{id}/backups`, or with `--target` on\n  `volume-backup create`.\n- **A schedule** names its target with `\"target\"` in\n  `PUT /api/v1/volumes/{id}/backup-schedule`, or with `--target` on `volume schedule set`.\n  Every backup the schedule takes is copied to that target. This works for any volume, a\n  data volume included. Setting a schedule replaces it as a whole, so a schedule set\n  without a target has none, and its backups are not copied.\n\nA target the node does not have is refused with \"no backup target has that id\".\n\n### What a copy is\n\nOnce a backup has been checked on the node, the node makes its copy:\n\n1. The backup is encrypted before it leaves the node. Each backup has a key of its own,\n   and that key is protected by the target's key. The target's key is protected by the\n   node key, which is described in [Losing the node key](#losing-the-node-key).\n2. The encrypted copy is sent to the bucket as one file, named by the backup's id, in a\n   folder for the node that made it, under the target's prefix. Nodes that share a bucket\n   and a prefix keep their copies apart.\n3. The node checks the copy on the target. It sends the copy in pieces. Afterwards the\n   target must hold every piece that was sent, and the stored file must have the length\n   that was sent.\n4. A small record of the copy is written beside the file. The copy is complete only when\n   the record is on the target too. A copy that does not pass every check is removed from\n   the target, and it is not kept.\n\nThe bucket holds encrypted data only. Without the target's key, nothing in it can be\nread. The file names show backup ids, never the names of volumes or machines. The test\naction's file holds no backup data, and it is removed at once.\n\nA backup's copy has one of four states, shown as `target_state` in the API:\n\n- **uploading**: the copy is being sent. The backup cannot be deleted until the copy has\n  finished.\n- **complete**: the copy is on the target and has been checked.\n- **failed**: the copy was not kept. See the next part.\n- **removing**: the backup was deleted, and its copy is being removed from the target.\n\nA backup's task ends when its copy has been checked, so a task that is still running is\nstill sending the copy.\n\n### When a copy fails\n\nA copy fails when the target refuses the credentials, cannot be reached, or does not keep\nwhat was sent. It also fails when this node cannot read the target's key, or when the\nstorage pool has no room for the encrypted copy. When a copy fails:\n\n- **The backup stays on the node**, as it was. It is marked **failed**, with the reason\n  in `target_error`. A target does not keep a copy that failed its check: \"the backup\n  could not be checked after upload; it is not kept on the target\".\n- **A scheduled backup** ends its run as failed. The schedule's last error reads \"the\n  backup did not finish: the task has the reason\", and the task has the reason. The\n  disk's `backup_failed` reading goes to 1, so an alert condition on it fires. See\n  [Failures and notifications](#failures-and-notifications).\n- **A backup taken by hand** shows the failure on its task and its copy. It does not raise\n  the alert.\n- **It is sent again.** Once a day the node sends again each copy that failed, oldest first.\n  If the target cannot be reached or refuses the credentials, the node stops there and tries\n  again the next day.\n- **Retention keeps it.** A backup whose copy failed is not removed by retention. It stays\n  on the node until its copy is sent or you delete it.\n- **The next backup does not build on it.** A backup with a target builds on an earlier\n  backup only when the target holds that backup and all it builds on. Otherwise an automatic\n  or scheduled backup is taken in full, and an incremental backup asked for by name is\n  refused: \"the backup target does not hold the backup this one would build on, so an\n  incremental backup cannot be kept there: take a full backup\".\n\nA backup that fails before its copy starts holds nothing on the target, and can be deleted as\nany failed backup can.\n\nAn interruption is not a failure. If the agent stops or the node restarts during a copy,\nthe copy starts again from the beginning at the next start, after the uploads it left open\non the target are cancelled. Once a day the node also cancels its own uploads that have been\nopen for more than a day.\n\n### Retention and deletion\n\n- **Retention** works as it does for local backups. A schedule keeps its newest restore\n  points, and the copies of the restore points it removes are removed from the target too.\n  Whole sets go together: a full backup and the incremental backups built on it.\n- **Only what the target holds counts.** Retention runs when a backup's copy has been checked\n  or has failed, not before. A restore point counts towards the number to keep when the target\n  holds it and all it builds on, and its last check did not fail. So a target that stops taking\n  new copies keeps the copies it has, and the newest ones the number allows stay on it.\n- **A local copy is pruned only after its copy is checked on the target.** A backup whose\n  copy is still being sent, or has failed, keeps its local copy. During a long outage these\n  add up in the storage pool until the target returns or you delete them.\n- **Deleting a backup deletes its copy.** If the target will not remove the copy, the node\n  keeps the backup's record until the copy is gone, and tries again at the next start and\n  once a day.\n- A copy made before copies were kept in a folder for each node, which has no record, is\n  never removed by retention or when its backup is deleted, so an operator can remove it by\n  hand from the bucket once it is no longer needed.\n- A backup whose copy is still being sent cannot be deleted: \"this backup is being copied\n  to its backup target; try again when the copy has finished\".\n\n### Restoring from a target\n\nA restore reads the backup's local copy when the node has it, and checks it, as always.\nWhen the node no longer has the local copy, the restore reads the copy back from the\ntarget that holds it. You do not choose the target. The node finds it from the backup's\nrecord. A local copy that fails its check is replaced by the copy from the target, and stays\non the node after the restore. A local copy that could not be read at that moment is not\nreplaced: \"this node could not read its copy of the backup just now, so nothing was restored:\ntry again\".\n\n- The node checks the copy as it reads it. Before the copy takes its place on the node, the\n  node checks the copy's image: a complete backup must not name any other file behind it,\n  and an incremental backup is joined to the copy it builds on, on this node. The node then\n  checks the whole backup against the checksum recorded when the copy was sent.\n- Nothing is written to the new volume until those checks pass. A copy that fails them is\n  refused: \"the backup on its backup target could not be verified, so nothing was restored\".\n- A restore needs the target on the node, with its credentials. If the copy is on neither\n  the node nor its target, the restore is refused: \"the backup's copy is not on this node\n  and is not on its backup target\".\n\nTo restore a copy on a node that did not make it, see\n[Restoring on another node](#restoring-on-another-node).\n\n### Copies on a target\n\n`storage target copies <target>` on the command line, or `GET /api/v1/storage/targets/{id}/copies`\nover the API, lists the copies a target holds. The list is read from the copies on the target. It\nchanges nothing on the target or on the node, and reading always works, even under a frozen\nlicence. The node needs the target's credentials and key to read the list, so a target whose key\nis not on the node lists nothing, and says why.\n\nEach copy shows its state:\n\n- **restorable**: it can be restored from this node.\n- **incomplete**: its data is missing or changed on the target, or a copy it builds on is not on\n  the target, or cannot be restored from it.\n- **unreadable**: its record cannot be read with this target's key. The usual cause is the wrong\n  key, and then the node says \"this copy was made with another key: import the key it was made\n  with, then try again\". A record changed on the target is unreadable too.\n- **refused**: its record names something outside its own copy, so the node refuses it.\n\nA copy that cannot be restored gives its reason in the list, in the same words the node gives\nwhen a restore of that copy is refused.\n\n### Checking copies\n\nEach copy on a target is checked once a week. The checks of different copies are spread across\nthe week, so the node never checks every copy at once. A check reads the copy back as far as is\ncheap to do: it confirms the length of the copy and the checksum the target reports for it, it\nreads the copy's record, and it reads back one part of the copy's data, chosen at random, which\nit checks with the target's key. A check changes nothing on the target, and it deletes nothing.\n\nA copy whose check fails is marked **check failed** in its backup, with the reason in words. In\nthe API these are `target_check_state` and `target_check_error`. The backup stays on the node, and\nits copy keeps its state. An alert can be raised for the failure: create an alert condition on\n`copy_check_failed`, as you would for a scheduled backup that fails, and attach an email or a\nwebhook to it. A later check that passes clears the mark and resolves the alert.\n\nTo check one copy now, run `cenvero-str-ctl storage target copy-check <target> <copy>`. The answer\nsays whether the copy passed, and a copy that failed says why. A target that cannot be reached, or\nthat refuses the credentials, cannot judge the copy: the answer says so, and nothing is recorded.\n\n### Restoring on another node\n\nA copy can be restored on a node that did not make it, for example when the node that made\nthe copies is lost. The new node needs the target first:\n\n1. Add the target, with the same endpoint, bucket, prefix and credentials as the old node's\n   target, using `storage target create`.\n2. Import the target's key from the export the old node took, with `storage target key-import`\n   (see [Exporting and importing a target key](#exporting-and-importing-a-target-key)).\n3. List the copies with `storage target copies`, and restore one with `storage target restore`.\n\n```bash\nsudo cenvero-str-ctl storage target create --name offsite --endpoint https://storage.example.net --bucket stratum-backups --prefix cenvero-stratum/ --access-key-id KEY_ID --secret-file /root/offsite.secret\nsudo cenvero-str-ctl storage target key-import <target> --key-file /root/offsite.key --yes\nsudo cenvero-str-ctl storage target copies <target>\nsudo cenvero-str-ctl storage target restore <target> bk-3b8e0a71 --name restored-data\n```\n\nThe restore writes a **new volume** on the new node, as a task. The volume belongs to the\noperator, since a copy names no tenant. Its name is `--name`; when that is left out, it is\nbuilt from the copy's volume name and the copy's id. It is kept in the default pool, or in\nthe pool that `--pool` names. A restore is a change, so it needs a licence that includes Storage\nand is not frozen.\n\nThe copies the restore builds on are fetched from the target as well, and each one is checked\nbefore the new volume is written. While they are fetched, the node holds the room they take in\nthe pool each copy is written to. Another create, copy, restore or backup that does not fit\nbeside that room is refused with \"the storage pool is full: free some space, or add a pool\"\nuntil the fetch has ended. The room is given back when the fetch ends, whether it succeeded or failed. The node\nkeeps no record of a copy it restored from another node, and it removes the fetched copies once\nthe restore has finished.\n\nA copy restores into the pool you name, or into the default pool when you name none. A copy\nthat does not check is refused before anything reaches the new volume, whichever pool it is\nrestored into.\n\n### Deleting a target\n\n`storage target delete <target> --yes` removes a target's record. It is refused while a\nschedule names the target, or while a backup has a copy on it, even a failed one: \"this\nbackup target is still used by volume backup schedules or backups: remove them first\".\nRemoving a schedule does not delete its backups. To delete a target, remove its schedules,\nthen delete the backups that have copies on it.\n\n### Exporting and importing a target key\n\nEach target has a key of its own, and the node uses it to protect the target's copies. The\nkey can be exported, and an exported key can be imported on a node that needs it:\n\n- **Over the API**, `POST /api/v1/storage/targets/{id}/key-export` returns the key in its\n  answer, base64-encoded. It needs your password confirmed in the last five minutes in the\n  web console, or the node's own API token. Any other token is refused with **403**.\n- **On the command line**, run `sudo cenvero-str-ctl storage target key-export <target>\n  --yes` on the node that holds the target. It cannot be sent to another node with `--node`.\n- **Over the API**, `POST /api/v1/storage/targets/{id}/key-import` takes the key the export\n  printed, and sets the target's key from it. It needs the same confirmation as the export.\n- **On the command line**, run `sudo cenvero-str-ctl storage target key-import <target>\n  --key-file FILE --yes`, or give the key on standard input with `--key-stdin`. It cannot be\n  sent to another node with `--node`.\n\nEvery export is audited. Every import is audited too. The key is written to no log, task or\naudit record, and an import does not show the key again.\n\nAn import sets the target's key, and changes nothing when the key is the one the target has\nalready. When the node can read the target's current key, a different key is refused while\nthis node holds copies on the target: \"this target already has its own key, and copies made\nwith it are kept here: the key cannot be replaced while they are\". A replaced key cannot read\nthose copies. A key the node can no longer read, after it lost its node key, is replaced at\nonce. A key that is not the right length is refused with \"that is not a backup target's key:\npaste the whole key that the key export printed\".\n\nTake the export when the target is set up, because an export is only possible while the node\nstill has its node key. Keep it where only the operator can read it, and not only on the node:\na key kept only on the node is lost with it. Anyone who has the key and access to the bucket\ncan read the copies. Without an export, no node can read the copies once the node key is lost\n(see [Losing the node key](#losing-the-node-key)).\n\n## Losing the node key\n\nThe **node key** is a secret the node keeps in its data directory. It protects the target\ncredentials, the targets' keys, and the machines' user-data and SSH keys. These are kept\nsealed under it, never in plain text. The node makes the key the first time it needs one.\n\nThe node key is lost when its file is removed or cannot be read, for example when a node\nis rebuilt without its data directory. Losing it has these effects:\n\n- **Target credentials and keys cannot be read.** A target's test, its copies and its\n  restores are refused with \"this node cannot read the credentials of this backup target:\n  enter them again before it is used\". Enter the credentials again with `storage target set\n  <target> --access-key-id KEY_ID --secret-file FILE`. This makes the node key again, when the\n  node has none. The target's key is sealed under the node key too, and entering the\n  credentials does not restore it: import it from an export, as described in\n  [Exporting and importing a target key](#exporting-and-importing-a-target-key).\n- **Copies are not sent** until the target's credentials and key are both on the node. Each\n  copy fails with the same sentence, and its backup stays on the node.\n- **First-boot settings cannot be read.** A machine whose user-data or SSH keys cannot be\n  read does not start, and cannot be cloned, until they are set again: \"the first-boot\n  settings of this machine cannot be read on this node: set them again before it starts\".\n  Set them again with `vm cloud-init set <machine> --ssh-key-file FILE --user-data-file\n  FILE`, giving the SSH keys and the user-data file the machine should have. Once they are\n  saved, the machine can start again. A running machine keeps running, and the refusal\n  applies only when it next starts. The hostname and the DNS servers are not sealed, so they are not\n  affected. See [First boot](/docs/compute/virtual-machines#first-boot).\n- **Copies already in the bucket stay there.** They can be read again on this node once the\n  target's key is imported from an export taken before the loss. Without that export, no node\n  can read them.\n- **Volumes and local backups are not affected.** They are not sealed under the node key.\n\n**Recovery.** The node makes a new node key the next time it must seal a value, and\nanything saved from then on is sealed under it. Values sealed under the lost key stay\nunreadable, so enter them again as described above. A key file that is still there but\ncannot be read is never replaced: the node refuses to save secrets until it can read the\nfile again.\n\nEach target needs both steps: its credentials entered again, and its key imported from an\nexport taken earlier. Without an export, its copies cannot be read on any node, and the\ntarget cannot send new copies. Create a new target to keep making copies.\n\n**Backups on another node.** A backup that was never sent to a target can be restored only\non the node that holds it. A backup sent to a target can be restored on another node, as\ndescribed in [Restoring on another node](#restoring-on-another-node). A restore makes a\nvolume, not a machine. A backup holds a disk, not the machine's first-boot settings. So\nnone come back with a restore, and a machine's settings are set on the node that runs it.\n\n## Storage pools\n\nA pool is a place on the node where volumes and backups are kept. Every node has a\n**default pool**, which holds the machines' disks and every volume and backup made\nbefore pools existed. You can add **directory pools**, and **block pools** on storage the\nhost already has. [Storage pools](/docs/compute/storage-pools) describes both kinds, what a\nblock pool needs on the host, and what works on each.\n\n- A volume, a machine or a restore takes its pool when it is made: `volume create --pool`,\n  `vm create --pool`, and `volume-backup restore <id> --pool`. Leave `--pool` out for the\n  default pool. A machine's boot disk stays on the pool the machine was made on, and moving\n  one is not possible yet.\n- A pool that still holds volumes cannot be deleted, and the default pool cannot be\n  deleted at all.\n- A backup of a volume on a block pool is a full backup, and it is written to the default\n  pool. That pool needs room for the whole volume.\n- Before a create, a restore, a copy or a scheduled backup starts, the node checks that\n  the pool has the room the work needs. A full pool refuses the work with \"the storage\n  pool is full: free some space, or add a pool\". A pool whose free space the node cannot\n  read refuses it too, with \"the storage pool's free space cannot be read right now, so\n  nothing was copied: try again later\". Nothing is written in either case.\n\n## Over the API\n\n| Method | Path | Answer |\n|---|---|---|\n| `GET` | `/api/v1/vms/{id}/disks` | **200** with the machine's disks, the boot disk among them |\n| `GET` | `/api/v1/vms/{id}/snapshots` | **200** with the boot disk's snapshots |\n| `POST` | `/api/v1/vms/{id}/snapshots` | **202** with the snapshot and its task |\n| `GET` | `/api/v1/vms/{id}/backups` | **200** with the boot disk's backups |\n| `POST` | `/api/v1/vms/{id}/backups` | **202** with the backup and its task |\n| `POST` | `/api/v1/volume-backups/{id}/restore` | **202** with the new or restored volume and its task |\n| `GET` | `/api/v1/volumes/{id}/backup-schedule` | **200** with the schedule, or **404** when there is none |\n| `PUT` | `/api/v1/volumes/{id}/backup-schedule` | **200** with the schedule |\n| `DELETE` | `/api/v1/volumes/{id}/backup-schedule` | **200** |\n| `POST` | `/api/v1/volumes/{id}/backup-schedule/run` | **202** with the schedule and its task |\n| `GET` | `/api/v1/backup-schedules` | **200** with the schedules of the volumes you can see |\n\nEvery route here is operator-only: a tenant-scoped key cannot reach them. Reading\nneeds `VM.Audit` for a machine's routes and `Storage.Audit` for a volume's schedule.\nTaking snapshots and backups needs `VM.Snapshot` and `VM.Backup`. Restoring and\nsetting or running a schedule need `Storage.Backup`.\n\n## See also\n\n- [Templates and clones](/docs/compute/templates), for a machine made from a copy of its boot disk\n- [Images](/docs/compute/images)\n- [Lifecycle and restarts](/docs/compute/lifecycle)\n- [Current limits](/docs/compute/limits)\n- [Monitoring](/docs/monitoring), for alert conditions and their actions\n"
        }
    ]
}