Maintenance and Evacuation
Part of Clustering and Compute, in early access. Maintenance needs every member of the cluster to run a version that supports it. Until then the answer is "Every member must run a version that supports this."
Maintenance takes a member out of placement while you service it. While a member is in maintenance:
- nothing new is placed on it. The cluster places a new machine on another member. A create that names the member is refused with 409, and its answer gives the member's reason, "This member is in maintenance, so nothing is placed on it and nothing moves onto it."
- nothing moves onto it. A move to the member is refused the same way.
- its machines keep running. Maintenance does not stop them.
Maintenance does not end by itself. You end it when the work is done.
Evacuation moves a member's machines off it before you service it. You read the plan first, and then a run moves the planned machines, one at a time by default. Each machine's move is an ordinary move, as Moving machines between nodes describes.
Turning maintenance on and off
Maintenance is set per member. Changing it needs Cluster.Modify on the member, which only the Administrator role holds (Access control). Reading it needs Cluster.Audit.
- In the console, open Node › Maintenance on the member. Put into maintenance opens a dialog that shows the plan. Its Move its machines off first box starts an evacuation. End maintenance ends maintenance.
- On the command line,
sudo cenvero-str-ctl node maintenance onputs this member into maintenance, andoffends it.--evacuatealso moves the machines off, and--modechooses how they move:auto, the default, oroffline. For another member, add--nodewith its ID or name.
sudo cenvero-str-ctl node maintenance on
sudo cenvero-str-ctl node maintenance on --evacuate
sudo cenvero-str-ctl --node edge-2 node maintenance on
sudo cenvero-str-ctl node maintenance off
sudo cenvero-str-ctl node maintenance show
- Over the API,
PUT /api/v1/nodes/{node}/maintenancewith{"enabled": true}puts a member into maintenance, and{"enabled": false}ends it. Both answer 200 with the member's state. Add"evacuate": true, with an optional"mode"ofautooroffline, to start the evacuation: the answer is 202 with the run's task. A"mode"ofliveis refused with 409 and "Moving a running machine is not available yet. Choose an offline move.", and the member's maintenance stays as it was.GET /api/v1/nodes/{node}/maintenancereads it.
Starting maintenance, with or without an evacuation, needs the cluster feature in the licence, and a frozen licence refuses it. Ending maintenance is never refused by the freeze.
Read the plan first
The plan lists a member's machines, largest first (by memory, then by machine ID). It names only the machines you may see; its totals count every machine. For each machine it gives the member the machine would move to, or the reasons it stays. The plan changes nothing. Each machine is judged with the same check a move makes, so a machine that its move would refuse is listed as staying, with the move's own sentence. The member that holds the machines makes this check, so on its own plan a machine planned to move is one its move can start. A plan read through another member leaves this check out: it lists only the plan's own reasons to stay, and each move is checked again when it begins.
- In the console, the plan is the table in the Put into maintenance dialog, and it is on the Node › Maintenance page.
- On the command line,
sudo cenvero-str-ctl node evacuation-plan. For another member, add--node. - Over the API,
GET /api/v1/nodes/{node}/evacuation-plan.
The answer has one line for each machine:
{
"node": "3f2a0c1e-9b7d-4c1a-8e2f-0123456789ab",
"concurrency": 1,
"planned_count": 1,
"skipped_count": 1,
"machines": [
{"order": 1, "id": "vm-3f9a1c2e", "name": "db-1", "tenant_id": "t-acme", "vcpus": 4, "memory_mib": 8192,
"state": "planned", "planned": "7c9e6679-7425-40de-944b-e07fc1f90ae7", "skipped": null, "reasons": []},
{"order": 2, "id": "vm-7d4e2f10", "name": "cache-1", "tenant_id": "t-acme", "vcpus": 2, "memory_mib": 2048,
"state": "skipped", "planned": null,
"skipped": "This machine has snapshots or backup checkpoints, so it cannot move yet. Remove them first.",
"reasons": ["This machine has snapshots or backup checkpoints, so it cannot move yet. Remove them first."]}
]
}
A machine stays on its member, and is skipped in the plan, when:
- "This machine's member has not yet reported it to the cluster." The cluster does not have the machine in its records yet.
- "This machine is already on another member, so it is not moved again." It has moved already.
- "This machine has a public address that the provider routes to its member, so it cannot move."
- "This machine has snapshots or backup checkpoints, so it cannot move yet. Remove them first."
- "No member can take this machine. The reasons are listed for each member." The other members' reasons follow, as they do for a placement.
- Any other sentence of What stops a move, such as a disk on a block pool, or a boot disk whose backup schedule must be removed first.
Running an evacuation
An evacuation is one task on the member, with one move for each planned machine. You start it from the Put into maintenance dialog, with Move its machines off first, on the command line with --evacuate, or over the API with "evacuate": true. The member goes into maintenance first, so nothing moves onto it while the run goes.
- Each planned machine moves to the member the cluster chooses when its move starts, by the rule a create uses. That member can differ from the one the plan showed.
- Machines move one at a time by default. The cluster's
migration_concurrencysetting, 1 or 2, lets two moves run at once. See Cluster compute settings. - A failed move does not stop the others. Its line on the task says "This machine's move did not finish. The run carries on with the other machines." and gives the machine's own sentence.
- Each move needs
VM.Migrateon its machine, as a single move does, whichever member you asked. A machine you may not move fails with "You do not have permission to move this machine.", and the run carries on with the others. - The task ends as succeeded, with how many machines moved and how many were left out. It ends as failed when a machine did not move, and it names those machines. It ends as cancelled when you stop it before every machine has been started. As it ends, the run looks at the member's machines again: a machine it did not plan that is still on the member, such as one that arrived while it ran, makes the task end as failed and is named, so run the evacuation again to move it.
- Cancelling the evacuation's task stops it after the moves already under way: no further machine starts. The cancel is never refused by the licence freeze.
- Ending maintenance stops the evacuation in the same way, whichever member runs it: no further machine starts, and the moves already under way finish.
- A second evacuation of the same member is refused with "An evacuation of this member is already running. Wait for it to finish, or cancel it."
- Evacuation does not end maintenance. The member stays in maintenance until you end it with
node maintenance off. - A run started again skips the machines already moved, which the plan lists as already on another member.
- Use
offlineorauto. Moving a running machine is not available yet for an evacuation or a bulk move, which stop each machine and start it on its new member. A move while running is for one machine, from its own Migrate action or withvm migrate --mode live.
Moving several machines at once
A bulk move takes a list of machines and moves them to one member, or to the best member for each machine. It is one task. It takes the machines that the member you ask owns. A machine another member holds is refused with "This machine is on another member. Run the command on that member, with --node set to it." The console sends each owner its own machines, so that sentence reaches you on the command line or over the API.
- In the console, Datacenter › Migrations ticks the machines to move together. You can tick only the machines you may move. Choose the member and the mode, then Move the selected machines. The console sends one request to each member that owns some of the ticked machines, and it reports each member's answer.
- On the command line,
sudo cenvero-str-ctl vm bulk-migrate vm-3f9a1c2e vm-7d4e2f10 --to auto, run on the member that holds the machines, or with--nodefor that member. - Over the API,
POST /api/v1/cluster/migrationstakes{"vms": ["vm-3f9a1c2e", "vm-7d4e2f10"], "node": "<member id>" or "auto", "mode": "offline"}, and answers 202 with the run's task.
Every machine is judged before any move starts. Each member judges its own machines. If one of them cannot move, nothing moves. The answer is 409 with "No machine was moved, because some of the machines cannot move. The reasons are listed for each machine.", and a list naming each machine that cannot move with its reason. A bulk move names from 1 to 100 machines, each once. Once every machine has been judged, the moves run as an evacuation does: one at a time by default, and the concurrency setting applies.
Creating machines while a member is in maintenance
The cluster places a new machine on a member that is not in maintenance. A create that names a member in maintenance is refused with 409, and nothing is recorded. The answer gives the sentence above as that member's reason.
Rolling updates
A rolling update of the cluster (Upgrades) uses maintenance on each member it restarts. Once every member runs a version that supports moves, the member goes into maintenance before the update waits for its moves, and it comes out of maintenance once it runs the new version. A member that was already in maintenance stays in it. The update waits at most 30 minutes for a member's moves to end. While the update holds a member in maintenance, no machine moves off it either: such a move is refused with "This machine's member is being updated, so no machine moves off it until the update has passed it. Try again after the update."
What maintenance does not do
- It does not stop, change or delete the member's machines, its tenants or its networks.
- Evacuation moves machines only. A cluster network's host gateway stays on its member. If one runs on the member, move it first with
cluster network host-gateway <id> --member <member id>; the cluster does not move it for you (Cluster networks). - It does not move a gateway pair's address. See Gateway High Availability.
Who may do what
| Action | Privilege | Licence |
|---|---|---|
| Put a member into maintenance | Cluster.Modify on the member (Administrator only) | cluster |
| End a member's maintenance | Cluster.Modify on the member (Administrator only) | none: ending is never refused |
| Read a member's maintenance and its plan | Cluster.Audit | none (reads are never refused) |
| Start an evacuation | Cluster.Modify on the member, as part of the maintenance change | cluster; each move also needs compute |
| Move several machines at once | VM.Migrate on each machine | compute |
See also
- Moving machines between nodes — how one machine moves, and what stops it.
- Placing machines — the rules a member must meet to take a machine.
- Clustering Overview — forming a cluster and managing every member from any of them.
- API Reference — the maintenance, plan and move routes.
- CLI Reference — the
nodeandvmcommands.