Exclusive Access · Invitation Only

Moving Machines Between Nodes

Part of Clustering and Compute, in early access. A move needs every member of the cluster to run a version that supports it.

A machine can move from one member of a cluster to another, in one of two ways:

  • Stop, move and start (the offline mode, and auto, which does the same). The machine is stopped, its disks are copied to the new member and checked there, and the machine starts on the new member.
  • Move while running (the live mode). The machine keeps running. Its memory is carried to the new member while it runs, it pauses at the switch until the new member has recorded the move (usually a short wait, with no fixed upper limit), and it runs on the new member. Its disks stay where they are, on storage that both members share, so nothing is copied.

Both keep the machine's name, its ID, its addresses and its settings. In this page, the old member is the one the machine leaves, and the new member is the one it moves to.

A machine runs on one member at a time. In a stop, move and start move, the old member keeps its copy until the new member has started the machine. In a move while running, the old member gives up its copy once the memory has arrived on the new member, and the machine runs only on the new member once the move commits.

What a move does

A stop, move and start move does this:

StepWhat happens
prepareThe new member takes the machine's settings and makes its network ready: the same addresses and hardware addresses, and the same separation from other tenants' machines. The machine keeps running, and the new member does not list it or let it start until the commit.
copyThe machine is stopped. If the new member lacks a linked clone's base image, the base is copied first. Then the machine's disks, its data volumes, and its firmware settings and virtual TPM state are copied over the connection the members keep between them. Each file is checked against the list the old member signed before it is kept.
commitThe new member records that it owns the machine. From here the move cannot be cancelled.
startingThe new member records that it is about to start the machine.
startThe machine starts on the new member.
cleanupThe old member removes its copy of the machine. The move is finished.

The machine is stopped for the copy, so its downtime lasts about as long as the copy does, plus the base image's copy when the new member needs one. How long the copy takes depends on the size of the disks and on the connection between the members. The copy keeps to the cluster's migration bandwidth, which is 256 MiB/s unless an operator changes it in the API.

A copy is not ended for taking a long time. The new member reports its progress at least every two minutes. A copy that reports nothing for 15 minutes is ended, and the machine returns to its old member; see When a move ends without finishing.

What a live move does

A move while running keeps the machine running until its switch. The phases below are the steps of the move. The API and the command line show them by these names, and the mode is live.

PhaseWhat happens
prepareThe same preparation as a stop, move and start move: the new member takes the machine's settings and makes its network ready, and does not list or start the machine until the commit. The old member also writes a check into each shared storage pool the machine uses. The machine keeps running.
switchingThe old member records the switch, and then the machine's memory is carried to the new member over the connection between the members, at the cluster's migration bandwidth. The machine keeps running. The new member first reads the check: if it cannot see the shared storage as the old member does, no memory is carried.
releasedThe transfer is confirmed, and the old member's copy of the machine is gone. The machine waits paused on the new member until the new member has recorded itself as the owner.
commit, starting, startThe new member records itself as the owner, claims the machine, resumes it and records that it runs there. A commit the cluster refuses for now is tried again while the machine waits paused. If the paused machine will not resume on the new member, the new member starts it from its shared disks instead, and if that fails too, the move ends (When a move ends without finishing).
cleanup, then doneThe old member removes its definition of the machine, its records and its network state. It never removes the disks: the new member runs the machine from them. The move is finished.

The pause that begins at the switch is the only time the machine does not run. It lasts until the new member has recorded itself as the owner, which is usually short. It has no fixed upper limit: the new member keeps trying to record itself as the owner while the machine waits paused. Its disks and its data volumes are not copied, because a move while running needs them on shared storage (What a live move needs). A base image the new member does not hold is copied before the prepare, as a stop, move and start move copies it.

The steps keep their names in the API and on the command line: prepare, copy, commit, starting, start and cleanup. During a move while running, the copy step is the switch and the transfer, and the commit step is the wait for the cluster to record the new member. In the console those two steps read Moving the memory and Waiting for the cluster to record the new member.

What a live move needs

A move while running is refused, before anything is recorded or stopped, unless each of these holds. The sentence for each refusal, and what to do about it, is in What stops a live move.

  • The machine runs. A stopped machine moves stopped, in a stop, move and start move.
  • Shared storage. Every disk and data volume of the machine is on a storage pool that the old member declares shared, and the new member declares the same pool shared, at the same location. Only a directory pool on a filesystem that can be shared qualifies, as Shared storage pools describes. A disk stored outside the shared storage of its pool, for example through a link into local storage, does not count. A linked clone's base image is not on that storage: the new member keeps its own copy of it, and the copy must sit at the place the disk records. A move that finds it elsewhere is refused, with the sentence in What stops a live move.
  • The processor. The new member can give the machine its processor model, under the cluster's processor policy (Placing machines).
  • The same processor, for a machine on its server's own processor. A machine that names no processor model moves while it runs only to a member that reports the same processor model as its current server, and every processor feature that server has. A member that does not is refused with "The new member cannot give this machine's processor model." (What stops a live move).
  • No backup running. A backup of the machine that is in progress must finish first.
  • No saved state. The machine has no saved state.
  • No UEFI firmware and no virtual TPM. A move while running does not carry them. A stop, move and start move does.
  • The new member supports it. It runs a version that moves a running machine. A move to a member that does not is refused with the sentence in What stops a live move.
  • The same security setup on both members. Both members run the same security setup for virtual machines. A member whose setup differs does not refuse the move before it starts: the transfer fails, and the move ends with "The machine could not be carried to the new member while it runs, so it kept running on its previous member." The move does not say which setting differs, so check that both members match (When a move ends without finishing). A member that runs its machines with no security confinement at all cannot receive a move while running, even from a member set up the same way.
  • A virtualization version this release knows. A member accepts from the old member only what it knows a move while running needs, and judges the machine's definition against what it holds. After a host's virtualization packages are upgraded, they may ask for something new. The transfer then fails with the same sentence, the machine keeps running on its previous member, and the new member's log names what it refused. A stop, move and start move still works, and a Stratum update that recognises the new version restores moves while running.

The checks of a stop, move and start move also hold for a move while running: a machine with snapshots or backup checkpoints, a disk on a block pool, a public address that the provider routes to its member, a suspended tenant, and a network that the new member does not have are refused in either mode.

Starting a move

In the console, open the machine and press Migrate in its header. Choose the member: The best member for it, or one of the members that can take the machine (the others are listed with their reasons). Under How it moves, choose one of three modes:

  • Stop, move and start is the offline move.
  • Move while running is the live move. It appears only on a member that lists the live stage in its features, and it is off, with the reason, for a machine that is not running. The member you choose as the new member must list the live stage too, or the move is refused (What stops a live move). Once you choose it, the dialog says that the move needs its pool shared between both members, and gives the two sentences of What stops a live move.
  • The cluster chooses: stop, move and start is auto. In this release it stops, moves and starts the machine, as Stop, move and start does.

Move the machine starts the move. The machine's Migrations tab lists its moves, with the mode and the phase of each, and a move's page shows its steps and the bytes copied, or moved during a move while running. The page offers Cancel the move only while the move can be cancelled. Both the Migrate action and the Migrations tab appear on a member of a cluster.

On the command line, name the new member by its ID, as cenvero-str-ctl cluster status lists it, or ask for auto:

# Move the machine while it runs to a named member, and wait until the move has finished
sudo cenvero-str-ctl vm migrate vm-3f9a1c2e --to 7c9e6679-7425-40de-944b-e07fc1f90ae7 --mode live --wait

# Stop, move and start the machine, with the cluster choosing the member
sudo cenvero-str-ctl vm migrate vm-3f9a1c2e --to auto

# The machine's moves, and the steps of one of them
sudo cenvero-str-ctl vm migrations vm-3f9a1c2e
sudo cenvero-str-ctl migration show mig-3f9a1c2e5b7d4a10

--mode takes auto (the default), offline, or live. auto and offline both stop, move and start the machine. live moves it while it runs. With --wait, the command follows the move's task and exits with an error when the move did not succeed.

Over the API, POST /api/v1/vms/{id}/migrate takes {"node": "<member id>"} or {"node": "auto"}, and an optional "mode", which is auto (the default), offline or live. The answer is 202 with the move and its task. Add ?check=1 to judge the move without starting it: the answer is 200 with {"check": "ok"} when the checks the member makes pass, or 409 with the sentence that refuses it. For a live move, the new member judges a named processor itself when the move starts, so a dry run can answer ok and the move still end with the processor sentence (see below the checks). You can call any member: when another member holds the machine, it passes the request on.

A move that cannot start is refused, and nothing is recorded or stopped. The API answers 409. A named member that cannot take the machine gives "No member can take this machine. The reasons are listed for each member.", with that member's reasons in candidates; for auto the answer lists every member's reasons. The tables below give the sentences.

Choosing the new member

A move asks the question a create asks: which members can take this machine? The rules are the ones on Placing machines. The new member must have room for the machine's disks, enough free processors and memory, the hardware acceleration the machine needs, the processor model it needs, and every network it uses. The tenant must not be suspended, and the new member must not already run another machine of the machine's anti-affinity group. A move while running also needs the shared storage that What a live move needs describes.

A move is the one operation that may fetch an image. A linked clone's base image must be on the new member, or the image library must hold a verified copy of it on some member. In the second case the move copies the base to the new member before its disks. The base copy is a task of its own, linked to the move, and the machine's stopped time includes it. If the library holds no copy either, the member is not offered, and its reason is "The image is not on this member. Copy it to this member first." A create never fetches an image. A move while running copies a missing base the same way, before it prepares the new member, and the machine keeps running meanwhile.

A network matters most. A cluster network counts as present on every member that can carry the cluster's networks, so a machine on a cluster network can move between them. A network you create on one member is not copied to another, and it has a different ID there, so a machine on such a network cannot move to another member. The answer says "This member does not have a network the machine needs." To move that machine, give it a cluster network instead.

With auto, the cluster picks the best eligible member: the one that already holds the image, then the one with the most memory left, then the one with the fewest machines, then the lower member ID.

What moves with the machine

  • Its ID, name and host name.
  • Its addresses and hardware addresses, so the machine keeps its place on its networks.
  • Its disks and its data volumes, with their contents. A stop, move and start move copies them. A move while running leaves them on the shared storage where they are.
  • Its UEFI firmware settings, and for a machine with a virtual TPM, the TPM's state. A stop, move and start move carries them. A move while running does not (What a live move needs).
  • Its boot order, its protection and its start-at-boot setting.
  • Its first-boot settings: its SSH keys, its user data and its cloud-init record.
  • Its tenant, its anti-affinity group and its processor model.
  • Its name servers. A machine with name servers of its own keeps them. A machine without its own takes its member's, so a move to a member with different name servers is refused until you set the machine's own (see below).

The backups a machine has made stay on the member that made them. A move does not copy them. A move also pauses the machine's backup schedules while it runs, so no scheduled backup changes a disk the copy reads. Its schedules arrive on the new member paused and without their backup target. They run again once you save them there; see Snapshots, backups and schedules.

What stops a move

A move is refused before it starts when the machine or the request cannot take it. The answer is 409, and nothing is recorded or stopped.

The sentenceWhat to do
"This machine has a public address that the provider routes to its member, so it cannot move." The provider routes the address to one member, and moving it needs an action the platform does not take.Keep the machine on its member.
"This machine has snapshots or backup checkpoints, so it cannot move yet. Remove them first." A backup taken while the machine runs can leave a checkpoint.Delete the machine's snapshots, as Snapshots, backups and schedules describes, and move it once no checkpoint is left.
"This machine has a disk on a block pool, so it cannot move yet."Keep the machine on its member.
"This machine has volumes, firmware settings or a virtual TPM that a move cannot carry yet." This is the answer for a boot disk that has a backup schedule. The rest of the machine moves, but the schedule does not.Remove the boot disk's schedule with sudo cenvero-str-ctl volume schedule remove <volume id> --yes. The boot disk's volume ID is listed by vm disks <id>. Its backups stay on the member that made them.
"A volume of this machine is being changed, so the machine cannot move until that finishes."Wait for the change to finish, then move.
"This tenant has used all of its volumes across the cluster." A move's volumes need slots under the tenant's cluster limit.Free a volume, or raise the tenant's volume limit.
"Moving a running machine is not available yet. Choose an offline move." A bulk move or an evacuation asks for the live mode.Choose auto or offline, which stop, move and start the machine (Maintenance and evacuation).
"This tenant is suspended, so its machines cannot move or be placed."Resume the tenant.
"This machine is already being moved. Wait for that move to end, then try again." A machine has one move at a time.Wait for the move to end (GET /api/v1/vms/{id}/migrations), then move it again.
"You do not have permission to move this machine." Moving a machine needs VM.Migrate on it, in an evacuation or a bulk move as well.Ask an administrator for VM.Migrate on the machine.
"This member is in maintenance, so nothing is placed on it and nothing moves onto it."Choose another member, or end the member's maintenance (Maintenance and evacuation).

Most of the other reasons are a member's reasons. They are the same sentences as on Placing machines, and the answer gives each member's reasons in candidates. The console lists them for each member in the move's wizard. The ones a move meets most often are:

The sentenceWhat to do
"This member does not have the room the machine's disk needs."Free room on the member, or choose another.
"This member does not have enough free processors or memory for the machine."Choose another member.
"This member does not have a network the machine needs."Give the machine a cluster network, as above.
"This member cannot give the machine the processor model it asks for."Choose another member.
"This member cannot give the machine hardware acceleration."Choose a member that has hardware acceleration.
"This member's name servers differ from the machine's, and the machine has none of its own. Set the machine's name servers first."Set the machine's own name servers with vm cloud-init set <id> --dns <list>. They apply at the machine's next start.
"The image is not on this member. Copy it to this member first."Copy the image to a member, so that the library holds a copy (The cluster image library).
"Another machine of this group is on that member."Choose another member.
"This member's licence does not allow this change."Check the member's licence.

The move's own checks run on the new member once the move has started. A move that one of them refuses ends as failed with the same sentence. The machine stays on its member, or goes back to it and starts again there if it was running.

A member that has just started needs a few seconds before it can take a move. A move to it waits, for up to three minutes, and nothing is stopped or copied while it waits. If the member is still not ready after that, the move ends as failed with "The move stopped responding and was ended. The machine was returned to its previous member."

The sentenceWhat to do
"The new member could not protect this machine's network port, so the machine did not move."Choose another member, and ask the cluster why this one could not take it.
"The new member does not have room for this machine's disks."Free room on the new member, or choose another member.
"The new member's storage already holds this machine's disk, so the machine cannot move there." A stop, move and start move copies a disk onto the new member, so it cannot land on storage that the two members share, where the disk already is.Choose a member whose storage does not hold the disk. To move the machine between members that share its storage, move it while it runs.
"This machine uses its member's DNS servers, and the new member's are different. Set the machine's own DNS servers first."Set the machine's own name servers, as above.
"This machine has a disc that the new member does not hold. Eject the disc first, or copy its image to the new member."Eject the disc, or copy its image to the new member. A move never ejects a disc.
"The new member has no storage pool with the name of this machine's volume's pool. Create one there first."Create a pool with that name on the new member.
"The new member already has a volume with this volume's name for this tenant."Rename the volume, or choose another member.
"The new member cannot give this machine's processor model."Choose another member.
"This machine needs hardware virtualisation, which the new member does not have."Choose a member that has hardware acceleration.

A machine can have one move at a time. A second move of it is refused until the first one has ended.

What stops a live move

A move while running is refused in the same way as a stop, move and start move, for every refusal in the tables above. It is also refused for the conditions of What a live move needs. Each of the conditions below is a refusal on its own, and a move is refused with the first one that does not hold. These checks record nothing and stop nothing. A dry run (?check=1) gives the same answer, except for a named processor, which the new member judges when the move starts (see below the checks).

The sentenceWhat to do
"This machine is not running, so it cannot move while it runs. Choose an offline move."Start the machine, then move it while it runs, or choose a stop, move and start move.
"A backup of this machine is running. Wait for it to finish, then move the machine while it runs."Wait for the backup to finish, then try the move again.
"This machine has a saved state, so it cannot move while it runs. Remove the saved state first." Stratum does not make a saved state.Remove the saved state, or choose a stop, move and start move.
"This machine has firmware settings or a virtual TPM, which a live move cannot carry. Stop it and move it instead."Choose a stop, move and start move, which carries the firmware settings and the virtual TPM state.
"The new member does not support moving a running machine yet. Update it, or choose an offline move."Update the new member, or choose a stop, move and start move.
"The new member cannot give this machine's processor model."Choose another member that can give it (Placing machines).
"A base image of this machine is not at the same place on the new member, so the machine cannot move while it runs. Choose an offline move."Keep the image library at the same location on both members, or choose a stop, move and start move.
"Every disk of this machine must be on a storage pool that is shared between the members. Choose an offline move, or move the disks to a shared pool first."Declare the pool shared on this member, where its filesystem can be shared (Shared storage pools), or choose a stop, move and start move.
"The new member does not share this machine's storage pool. Share the pool with that member first."Declare the same pool shared on the new member, at the same location.
"A disk of this machine is stored outside the shared storage of its pool. Move the disk into the shared pool, or choose an offline move."Move the disk into the shared pool, or choose a stop, move and start move.

With auto, the cluster first chooses the member, and these conditions are then judged for that member.

A pool that a machine uses while it runs cannot be taken back as shared until that move ends. Taking the declaration back is refused with "A machine is moving while it runs on this pool, so the pool must stay shared until that move ends.", and with "Whether a machine is moving on this pool could not be checked right now. Try again when the node answers." when the node cannot tell whether one is. Stop sharing the pool once the move has ended (Shared storage pools).

A move that passes these checks and then fails before its switch, or at it, ends as described in When a move ends without finishing.

A named processor is the one check that the new member answers for itself, when the move starts. It can refuse a processor that its own processor cannot run, before the machine moves, so the move ends with "The new member cannot give this machine's processor model." and the machine keeps running where it was. A dry run cannot give that answer for the new member, so choose another member that can give the processor.

What you see during a live move

The API and the command line show each phase by its name, as the first column below gives it. The console shows the phase in words, as the second column gives them. In the console the mode reads Move while running, and the move's page leads with "This moves the machine while it runs. The machine's new owner is recorded before it resumes on the new member."

PhaseWhat the console showsWhat it means
preparePreparing the new memberThe new member is getting ready, and the machine runs. A move can be cancelled here.
switchingMoving the machine while it runs, with the copy step reading Moving the memoryThe memory is being carried, and the machine runs. The progress is shown as moved. A move cannot be cancelled from here.
releasedWaiting for the cluster to record the new memberThe old member's copy is gone. The machine is paused on the new member until that member is recorded as the owner. A move cannot be cancelled from here.
commitRecording the new ownerThe new member records itself as the owner.
startingClaiming the machine on the new memberThe new member claims the machine.
startStarting on the new memberThe new member resumes the machine and records that it runs there.
doneFinishedThe move is finished.
failedFailed, with the reason on the move's pageThe move ended. The machine is on its old member, or, after a decision, where the decision put it (Deciding a member lost during a live move). If the new member could not start the machine, the move ends, and the machine stays there, stopped, to start or delete as any other machine.
cancelledCancelledThe move was cancelled before its switch, and the machine stays on its old member.

With --wait, a move while running ends with the same outcome: the command succeeds when the machine runs on the new member, and it exits with the move's sentence when the move failed.

What a move does not do yet

  • Several machines at once while they run. An evacuation and a bulk move move machines stopped. A move while running is for one machine.
  • UEFI firmware and virtual TPM state while running. A move while running refuses them. A stop, move and start move carries them.
  • Machines whose storage is not shared. A move while running needs shared storage for every disk. A machine on other storage moves stopped.
  • Snapshots, checkpoints and backups. A machine with snapshots or backup checkpoints cannot move in either mode, and a backup never moves.
  • Public addresses and block pools. A machine with a public address that the provider routes to its member, or a disk on a block pool, cannot move in either mode.
  • A boot disk's backup schedule. It stops a move in either mode until it is removed.

Cancelling a move

An offline move can be cancelled until its commit, while its prepare and copy steps run. A move while running can be cancelled only in its prepare step. From its switch on, a cancel is refused with "This move cannot be cancelled from this member right now. Try again shortly."

  • In the console, press Cancel the move on the move's page. The button is shown only while the move can be cancelled.
  • On the command line, run sudo cenvero-str-ctl migration cancel <move id> on the member the machine is moving from. From another member, add --node with that member.
  • Over the API, POST /api/v1/migrations/{id}/cancel answers 200 with the move. A request sent to another member is passed on to the member the machine moves from.

The machine goes back to its member, and it starts again there if it was running. The new member removes what it made for the move, and it never removes a disk that the move did not place. The backup schedules the move paused run again on the old member. A cancel is never refused by the licence freeze.

After the commit a cancel is refused with "The move has passed its commit and cannot be cancelled. It will finish, or the owner will be set back while the source still has its files."

When a move ends without finishing

  • A stop, move and start move that fails before the new member starts the machine ends as failed, with the reason on its page. The machine stays on its member, or goes back to it, and starts again there if it was running.
  • A stop, move and start move that stops making progress is ended from its record, and the machine goes back to its member. A copy that reports nothing for 15 minutes is ended this way, and so is an offline move's commit that the new member has not claimed within 10 minutes. The sentence is "The move stopped responding and was ended. The machine was returned to its previous member."
  • A move while running that fails before its switch ends as failed, with the reason on its page. The machine keeps running on its old member, and nothing is stopped or copied.
  • A move while running whose check the old member cannot write ends as failed, with "The new member cannot see this machine's shared storage as the source does. Check that both members mount the same storage." The machine keeps running on its old member, and the check is removed.
  • A move while running that fails at its switch. If the memory cannot be carried to the new member, for example because the new member cannot read the check, or because the connection between the members moves no bytes for five minutes, the move ends as failed with "The machine could not be carried to the new member while it runs, so it kept running on its previous member." The machine keeps running on its old member. The new member discards its paused copy, and the shared disks are not touched.
  • A move while running is not ended by time at its switch or its release. The cluster never expires a move there. It ends when its transfer ends, or when the operator decides it because a member was lost (Deciding a member lost during a live move).
  • The old member never starts the machine by itself once the move is at its switch. While the machine is still on the old member, a start is refused with "This machine is part of a move in progress, so it cannot start until the move ends." Once the machine has left the old member, the refusal is "This machine was moved to another member, so it cannot start on this one. If that member was lost during the move, this member starts the machine again once the operator decides the move." That is the refusal both before the move's release is recorded and after it.
  • A member that restarts in the middle of a move while running settles it from its record. The old member decides from its own copy: a copy that is gone means the transfer finished, and a copy that runs means it failed. The new member resumes its paused machine only once the release and the commit are recorded.
  • A move that the new member has claimed is never set back by the cluster on its own, because the machine may already run there. The new member records the start once the machine runs. If the new member restarts and finds the machine not running, it makes sure the machine is off and sets the move back.
  • A new member that cannot resume the machine starts it from its shared disks. The new member holds the machine paused until it has recorded itself as the owner. If the paused machine will not resume there, the new member keeps trying for about two minutes, then discards the paused copy and starts the machine from its shared disks. Nothing is copied, and the disks are not changed. The move then goes on to its end. A move that the operator decided because its old member was lost is not started this way: its paused copy keeps trying to resume, and nothing is discarded. A new member that cannot yet read or discard its paused copy, or cannot yet record the machine's address for itself in the cluster, waits and tries again; only a refusal to resume counts toward the two minutes.
  • A new member that cannot start the machine either ends the move with "The machine could not start on the new member from its shared storage, so the move has ended. It is stopped on the new member: start it there, or delete it, as any other machine." The machine stays on the new member, stopped, and it is that member's machine from then on. The old member removes its definition and records of the machine, and keeps the shared disks.
  • A machine a move holds cannot be changed, copied or deleted. While a move is active, changing the machine (a restart, a resize, a network or settings change included), copying it or deleting it is refused on the old member, with "This machine is part of a move in progress, so it cannot be changed, copied or deleted until the move ends. If a member of the move was lost, the move ends when the operator decides it." The new member refuses them with the same sentence once it lists the machine, which happens after the move's commit. Until then it does not list the machine, so these answer as they do for a machine it does not hold. If the cluster cannot yet say whether a move holds the machine, the change is refused with "The cluster cannot say yet whether a move holds this machine, so this request was not carried out. Try again in a moment." and it can be tried again. When a move was decided because the member it moved to was lost, the machine on its old member is refused the same things with "This machine was moved to another member, and its disks are shared with that member, so it cannot be changed, copied or deleted here. If that member was lost during the move, this member starts the machine again from those disks, so wait for it to run here, then try again." Its disks stay in place until then.
  • Once the new member has started the machine, the move finishes, and the old member removes its copy.
  • A member that restarts in the middle of a stop, move and start move settles the move from its record when it starts again. A move that had not committed ends, and the machine stays on, or returns to, its old member. A move that had committed, and whose new member had not yet claimed the machine, is set back the same way, while the old member still has its files.

A move that has ended stays in the machine's history for 30 days.

Deciding a member lost during a live move

A move while running that is at its switch or its release cannot finish on its own if a member is lost. Neither can a move whose new member has started the machine, when its old member is lost before it removes its copy. The machine may still run on the lost member, and two running copies on the same shared disks would corrupt it. So nothing starts or moves the machine until the operator decides.

The decision is made with the command line or the API. It needs Cluster.Modify on the cluster, which only the Administrator role holds by default. A decision needs Compute in the licence, and a frozen licence never refuses it. The console does not offer it. A decision made over the API is written to the audit log with your identity, and a lost source's attestation is in that entry.

The new member is lost. The decision names the old member as the owner, and the move must be released, so the old member's copy is confirmed gone. The old member starts the machine again from its shared storage, and the move ends as failed with "The new member was lost during the move. The machine is started on the member the operator chose, from its shared storage." Run it on the old member, or add --node with it:

sudo cenvero-str-ctl migration decide mig-3f9a1c2e5b7d4a10 --lost <new member id> --owner <old member id>

The old member is lost. The decision names the new member as the owner, and it needs --source-down: your confirmation that the old member is down. Use it only when you know the old member is down. --source-down tells Stratum that you have confirmed the old member's server is switched off. Stratum cannot check that for itself, and two running copies of a machine on the same shared disks can corrupt them. If the old member comes back after the decision, it stops its copy of the machine when it next checks the move, which happens about once a minute, and it never starts the machine again; stopping that copy does not touch the disks. The new member takes the machine from its paused copy, and the move goes on to its commit with "The source was lost during the move, and the operator confirmed that it is down. The machine runs on the new member." A move the new member has already started is decided the same way, once the old member is down: the machine keeps running on the new member, and the move ends as done. The old member removes its leftover copy of the machine when it next checks the move, which happens about once a minute, or when it starts again. Run it on the new member, or add --node with it:

sudo cenvero-str-ctl migration decide mig-3f9a1c2e5b7d4a10 --lost <old member id> --owner <new member id> --source-down

Over the API, POST /api/v1/migrations/{id}/decide takes {"lost": "<member id>", "owner": "<member id>"}, with "source_down": true for a lost old member. The answer is 200 with the move. A decision that reaches the other member is passed on to the member that takes the machine. The command line refuses a decision for the other member, and asks for --node.

A decision can be refused:

The sentenceWhat to do
The decision must name the member that was lost ("lost") and the member that takes the machine ("owner"), and they must differ.Send both members, and two different ones.
"The machine's copy on its member is not yet confirmed gone, so the decision must wait until it is."Wait until the move is released, then decide.
"This move is not waiting for a decision. Only a move whose member was lost during its switch, its release or its start can be decided, and only once."Check the move's phase with migration show.
"To take the machine from a source that is down, confirm the source is down with source_down set to true."Add --source-down, once you know the old member is down.
"The new member does not hold the machine's paused copy, so it cannot take the machine."Decide only when the new member holds the machine paused.
"The machine still has a copy on its member, so there is nothing to start."Check the machine on its old member.
"The decision is made by the member that takes the machine. Run it there, with --node set to that member."Run the command with --node set to the member that takes the machine.
"The decision must name the move's two members: one as the member that was lost, and the other as the member that takes the machine."Name the two members of the move.
"The decision could not be recorded. Try again shortly."Try again shortly.
"This member cannot read the machine's state yet. Try again shortly."Try again shortly.
"There is no such move in this cluster."Check the move's ID.

Who may move a machine

Moving a machine needs the VM.Migrate privilege on that machine, in either mode. The Administrator, Operator and VMAdmin roles hold it (Access control). A tenant's key cannot move a machine, and its answer is 403.

Deciding a member lost during a live move needs Cluster.Modify on the cluster. Only the Administrator role holds it.

A move needs Compute in the licence, and a frozen licence refuses a new move. Cancelling a move, and deciding a lost member, are never refused by the freeze.

Moving a workload you run yourself

If you run your own hypervisor or containers and move a workload between hosts yourself, Stratum keeps the network side consistent: you move the endpoint from the source node to the destination node. Being in a cluster does not move an endpoint for you. Each endpoint still belongs to the node you attach it on (see Clustering Overview).

Stratum provides a network model that lets such a workload keep its IP and MAC when it moves between nodes, if you have prepared both nodes for it.

What has to be in place first

Neither of these happens automatically:

  • The network exists on the destination node too. A network you create on a node belongs to that node. Create one with the same subnet and gateway on the destination; it gets its own network id there, and you use that id when you attach. A cluster network is one network stretched over the members that carry it, so it has the same id on each of them and needs no copy.
  • The two nodes are joined by an overlay carrying that subnet, so workloads left on the source node can still reach the one that moved, and the other way round. See Networking Overview → Stretching a network across nodes.

Why the network identity survives a move

An endpoint's hardware address is derived from its IP unless you supplied one yourself. Claiming the same IP on the destination therefore reproduces the same IP and MAC pair, so ARP and NDP caches on other workloads stay valid and active TCP connections are not reset by the move itself. Validate cross-host forwarding against your own topology before production rollout.

Moving an endpoint

Moving a workload's network identity from one node to another is a detach on the source followed by an attach on the destination, claiming the same IP:

# On the source node — free the endpoint
sudo cenvero-str-ctl network detach <endpoint-id>

# On the destination node — re-claim the same IP on its copy of the network
sudo cenvero-str-ctl network attach <destination-network-id> --ip 10.20.0.50

The destination node registers the IP↔MAC pairing in its packet path, so the workload's traffic is accepted there. If you registered overlay peers with workload hardware addresses, update those entries on the other nodes so they point at the destination node; a peer registered without a hardware address needs no change, because the address is learned from traffic. Sequence the detach and attach close together, and cut over the workload itself at the same time, to minimize the window where the endpoint is unbound.

If the workload has its own hardware address, name it again

This is the detail that catches people out.

When you let Stratum choose the hardware address, it is derived from the IP. Re-claiming the same address on another node therefore reproduces the identical hardware address on its own, which is exactly why the identity survives the move with nothing extra from you.

But if you originally attached the endpoint with a hardware address of your own, because the workload already had one, then claiming just the IP on the destination gives it the derived address instead, not the one it had. The workload's identity changes mid-move: peer ARP caches point at an address that no longer answers, and the anti-spoof binding now expects a different address than the workload actually sends, so its traffic is dropped as spoofed.

Pass the same hardware address again and the move is clean:

# The workload has its own hardware address — supply it on the destination too
sudo cenvero-str-ctl network attach <network-id> --ip 10.20.0.50 --mac 52:54:00:ab:01:02

If you are unsure which case you are in, read the endpoint back before detaching it and use whatever address it reports.

Pre-move checklist

  • The destination node has a network with the same subnet: cenvero-str-ctl network list
  • The overlay between the two nodes is up in both directions: cenvero-str-ctl vxlan peers <vni> on each
  • The network between the nodes has sufficient bandwidth
  • Firewall rules are per node: recreate on the destination any rule that references the endpoint's address — the IP is preserved, so the rule itself does not change

See also

↓ This page as JSON ↓ All documentation as JSON