Exclusive Access · Invitation Only

Placing Machines in a Cluster

When a cluster has several members, a new machine can be placed on the member that suits it best, instead of on the member you happened to ask. One rule decides that choice for every create. It looks at each member's free processors, memory and disk, the networks and the image the machine needs, its tenant's limits, its licence, its processor, and the machines of its anti-affinity group. Every member that cannot take the machine is listed with the reason.

Part of Clustering and Compute, in early access. Placement needs every member to run a version that supports it.

Until every member does, a placement request is answered with **"Every member must run a version that supports this.", and on a node in no cluster with "This needs a cluster of servers: create one or join one first."**

Placement only chooses. It never creates a machine by itself, and the member it chooses judges the create by its own licence, its freeze, its tenant limit and its room, as it would for any create it receives.

Choosing the member for a new machine

Let the cluster choose:

sudo cenvero-str-ctl vm create --name web-1 --tenant t-acme --image img-3f9a1c2e \
  --vcpus 2 --memory 4096 --disk 20 --network tier-web --placement auto

Or name the member that should create it. The root --node flag runs the command on that member, and that member creates the machine:

sudo cenvero-str-ctl --node edge-2 vm create --name web-1 --image img-3f9a1c2e --vcpus 2

--placement takes only auto, and it cannot be combined with --node: a placement chooses the member itself. Without either, the node you type on creates the machine.

Over the API, a create takes "placement": "auto" or "node": "<member id>", and not both ("Give placement or node, not both."). A placed create answers as a create does on a single node, 201 with the machine and its task. When no member can take the machine, the answer is 409, with the sentence **"No member can take this machine. The reasons are listed for each member."** and the reasons of every member. A named member that cannot take it is answered the same way, with that member's reasons only.

Seeing every member's answer

Ask the cluster where a machine of a given size would go, without creating anything:

cenvero-str-ctl cluster placement --vcpus 2 --memory-mib 4096 --disk-gib 20 \
  --image img-3f9a1c2e --network tier-web --tenant t-acme

Over the API: GET /api/v1/cluster/placement, with the query parameters vcpus, memory_mib, disk_gib, image, tenant, group, routed, cpu_model, and networks (repeated for each network). It answers 200 with the request as it was judged, in request, and one entry for each member, in candidates:

{
  "request": { "vcpus": 2, "memory_mib": 4096, "disk_gib": 20, "tenant": "t-acme" },
  "candidates": [
    { "node": "edge-1", "eligible": true, "rank": 1, "reasons": [] },
    { "node": "edge-2", "eligible": false, "rank": null,
      "reasons": ["This member does not have the room the machine's disk needs."] }
  ]
}

Each entry says whether the member is eligible, gives its rank (null when it is not eligible), and lists its reasons. The eligible members come first, in rank order. The others follow by member id, each with its reasons. The answer shows no tenant's names and no address of another tenant's.

Which members are eligible

A member is excluded, with the reason in its answer, when:

  • It cannot be reached from the node that asks, or it is stale: "This member cannot be reached from this node right now, so nothing is placed on it."
  • It does not run a version that supports placement. The answer is "Every member must run a version that supports this."
  • The tenant is suspended. "This tenant is suspended, so its machines cannot move or be placed."
  • The tenant has used its machines across the cluster. "This tenant has used all of its machines across the cluster." (see the tenant quota)
  • Its licence does not allow the change. "This member's licence does not allow this change."
  • It is in maintenance. "This member is in maintenance, so nothing is placed on it and nothing moves onto it." (see Maintenance and evacuation)
  • Machines cannot run on it. "Machines cannot run on this member right now."
  • It does not hold the image the machine starts from. "The image is not on this member. Copy it to this member first." (see the image library). A move is the exception: a member that lacks the image can take the machine when the library holds a copy, and the move copies it there (Moving machines between nodes).
  • It already runs a machine of the same anti-affinity group. "Another machine of this group is on that member."
  • It does not have the room. The machine's disk, plus 1 GiB kept free, must fit in the member's disk pool. "This member does not have the room the machine's disk needs."
  • It does not have enough free processors or memory. The machine's vCPUs, added to the vCPUs already committed there, may not exceed the member's processor count times the cluster's overcommit ratio (four by default). Its memory, added to the memory already committed, may not exceed the member's memory; memory is not overcommitted. "This member does not have enough free processors or memory for the machine."
  • It lacks a network the machine needs. A cluster network counts as present on every member that can carry the cluster's networks. "This member does not have a network the machine needs."
  • It cannot carry the cluster's networks, and the machine names one. A member carries them when its traffic checks are loaded and cover its overlay address. "This member cannot carry the cluster's networks."
  • It does not hold the public address the machine asks for. "This member does not hold the public address the machine asks for."
  • It cannot give the processor model the machine needs, or, for a machine that gets a named model, it is a hardware member whose processor is not reported. "This member cannot give the machine the processor model it asks for."
  • It cannot give the firmware the machine needs (UEFI, Secure Boot or a TPM). Firmware a member has not confirmed is not offered. "This member cannot start a machine that needs UEFI, Secure Boot or a TPM."
  • It cannot give hardware acceleration, which the machine needs. "This member cannot give the machine hardware acceleration."
  • A move would change its name servers. A machine with name servers of its own keeps them. A machine without its own takes its member's, so a member whose list differs cannot take it on a move: "This member's name servers differ from the machine's, and the machine has none of its own. Set the machine's name servers first." (see Moving machines between nodes)

Among the eligible members, the first is the one that already holds the image, then the member with the most memory left after the machine, then the one with the fewest machines, and then the one with the lower member id.

The machine's processor model

A machine's processor is either the processor of the member it runs on, or a named model that every member with hardware acceleration in the cluster can run. The policy is a cluster setting, cluster_cpu_policy, with two values:

  • baseline, the default. A new machine created on a member with hardware acceleration gets the cluster's common named model, once every member runs a version that supports it, two or more members have hardware acceleration and report their processors, and those processors share a named model.
  • host. A new machine keeps its member's own processor.

A machine created under emulation keeps the emulator's processor. A machine created before every member ran a version that supports it keeps the definition it was created with.

The model is recorded when the machine is created. Every later definition of the machine uses that record: a start, and a change of its size. The policy does not change a machine that already exists.

When a machine gets a named model, placement excludes a member with hardware acceleration whose processor is not reported, with **"This member cannot give the machine the processor model it asks for."** To find the members that can give a named model, ask with cenvero-str-ctl cluster placement --cpu-model <model>, or send cpu_model in the query. A member gives a named model when it reports that model, or that model with a version number (for example EPYC-Rome-v1 for EPYC-Rome).

Anti-affinity

A machine can carry an anti-affinity group. Machines of one tenant that carry the same group are kept on different members: placement does not choose a member that already runs one of them, and a member refuses a second machine of its group.

  • The name is 1 to 63 lowercase letters, digits and hyphens, and starts and ends with a letter or digit. Otherwise: "An anti-affinity group is 1 to 63 lowercase letters, digits and hyphens, starting and ending with a letter or digit."
  • A group belongs to one tenant. Two tenants can use the same group name, and their machines never keep each other apart.
  • A machine needs a tenant. A machine with no tenant cannot take a group: "Anti-affinity needs a machine that belongs to a tenant."
  • It needs every member to run a version that supports it. On a node in no cluster, or while a member does not: "Anti-affinity needs a cluster in which every member runs cluster level 5."
  • It is a hard rule. A create whose group no member can take is refused with the placement answer. A create that names a member already running the group is refused with "Another machine of this group is on that member."

Set the group when you create the machine, or change it afterwards:

sudo cenvero-str-ctl vm create --name db-1 --tenant t-acme --image img-3f9a1c2e --placement auto --anti-affinity db-pair
sudo cenvero-str-ctl vm options set db-2 --anti-affinity db-pair
sudo cenvero-str-ctl vm options set db-2 --anti-affinity ""      # clear it

Over the API, a create takes "anti_affinity": "<group>", and PUT /api/v1/vms/{id}/options takes {"anti_affinity": "<group>"}, with an empty value to clear it.

Three limits to know:

  • A clone takes no group. A clone starts without one, and you set its group yourself.
  • Two creates for one group that are placed at the same moment on different members can both succeed. Each member refuses a second machine of the group on itself, but a group that is being placed on two members at once is not yet seen by both.
  • A group keeps machines apart; it does not bring them together.

In the console, the create wizard offers the group once a tenant is chosen, and a machine's Options tab shows the field for a tenant's machine once every member runs a version that supports it.

Who can do what

ActionPrivilegeLicence
Read a placement answerVM.Auditnone (reads are never refused)
Create a machine, placed or namedVM.Allocatecompute
Set a machine's anti-affinity groupVM.Config.Optionscompute

A tenant's key cannot use the placement route; it answers 403.

See also

↓ This page as JSON ↓ All documentation as JSON