taiki.be

VMware → OpenShift Virtualization migration

The problem

Roughly 2,500 production and non-production VMs on vSphere across six sites, needing to be off VMware before the end of 2027. The target was OpenShift Virtualization on the bare-metal OpenShift footprint that already carried the containerised workloads — one platform, one lifecycle, one GitOps pipeline instead of two.

The estate is not greenfield. It is long-lived RHEL guests with static addressing, NICs pinned by name, and application owners who will not accept a long outage window per VM.

Constraints that shaped the design

  • Bare metal, not a hyperscaler. Compute is HPE Synergy; storage is Fibre Channel block from HPE Primera/Alletra arrays.
  • Copy bandwidth is the schedule. At this VM count, read-over-the-network and write-into-a-PVC is the whole timeline. The data had to move inside the array.
  • Guests must survive the transplant. vmxnet3 becomes virtio, interface names change, and anything hardcoding a NIC name breaks on first boot.
  • Everything reconciled by GitOps. Cluster state is FluxCD-managed, so migration tooling had to arrive as declarative config, not imperative one-offs.

Proof of concept

Validated the things that are cheap to assume and expensive to be wrong about, across compute, network and storage:

  • CPU hot-add works at socket level only, and needs the rollout strategy set to live-update for the change to land without a restart. Guests also needed the nested-virt KVM module blacklisted before hot-add behaved.
  • Memory hot-add confirmed working.
  • Live migration measured at 355–388 ms of guest pause. That number is what turned “we cannot take downtime” from an objection into a solved problem.

Networking

VM traffic lands on a bridged network attachment definition on a tagged VLAN, with the default pod interface explicitly disabled — VMs keep the same L2 they had on vSphere.

The trap: cloud-init has to be written against the interface name the VM will have after migration, not the one it has now. Getting that backwards costs a console session per VM, which at this scale is the difference between a pipeline and a project.

All service VIPs are held by keepalived on a dedicated pair rather than distributed across the cluster.

Guest preparation, automated

Rather than a checklist someone works through by hand, guest prep runs as a migration pre-hook: rebind the MAC for the NIC rename, write an explicit default gateway, set MTU, ensure the guest agent is present, enable a serial console getty, and strip the legacy gateway-device directive that survives from the vSphere era.

Seven distinct failure modes we had already hit manually, closed in one automated step. This is the single highest-leverage piece of the pipeline — the migration itself is reliable; the guests are what surprise you.

Data path

The largest engineering item was making the copy happen inside the storage array rather than over the network, which required source changes to the migration tooling’s storage backend. Written up separately in Teaching Forklift to offload VM disk copies to the array.

Platform work around it

  • Backup validated as part of the platform rather than bolted on afterwards, via the CSI VolumeSnapshot path.
  • An EgressIP return-path problem needed solving before VM traffic behaved: with OVN-Kubernetes in shared gateway mode and a host-owned VLAN subinterface, SYN-ACKs were dropped. The fix was a dedicated OVS bridge with a localnet bridge mapping.
  • Agent-based installs and machine-config rolling updates both needed runbooks of their own before the platform was repeatable rather than hand-nursed.

Where it stands

The pipeline works end to end. A canary VM migrated with array-side copy in roughly 55 seconds wall clock, about 11 seconds of which was the clone. Batch migration is documented as a runbook, and orchestration is moving off the ageing automation controller onto a lighter replacement.

I do not have a sustained-throughput figure across a full batch window yet. That comes from the first production waves, and I would rather leave the gap visible than estimate it.

What I would do differently

Validate the storage data path first, not last. Compute and network questions are answerable in an afternoon. The copy path determined whether the whole programme fit inside its window, and it was the one that needed patches to someone else’s source.