One of our customers needed more resources in one of his private OpenStack based clouds. Together with our customer we decided that it would be in our shared best interest to utilize the new AMD platform. The high-level idea was to completely replace the Intel hypervisors in one private cloud with AMD based hypervisors and integrate the Intel hypervisors in another private cloud from the same customer. This way we would have two homogeneous environments. One with Intel hypervisors and one with AMD based ones.
However the customer’s private cloud is used for his production and an effort to reduce impact was mandatory. At the same time it was also important to finish the migration quickly, as the customer needed the additional resources.
In order to help our customer have the resources as soon as possible, we formulated an action plan to migrate his staging VMs during the day and announce a maintenance window at night to allow for short downtimes of production VMs during restart.
Migrating the staging VMs first gave us experience, safety and confidence for the production VM migration.
The complete process for the migration looked as follows:
- Disable the source hypervisor in the scheduler
- Change the
nova.conf on the source hypervisor to use limited CPU flags
nova-compute on the source hypervisor
- Stop the VMs which need to be migrated
- Start the VMs which need to be migrated
- Change the
nova.conf on the source hypervisor back to use all CPU flags
- Restart nova-compute on the source hypervisor
- Start a live-migration to the destination hypervisor
- wait until this is finished
- repeat for next VM
- Stop the migrated VMs
- Start the migrated VMs again
We migrated 228 staging VMs during the day in around 6 hours and 60 production VMs in 90 minutes during the maintenance window. Reduced performance during the period with the
kvm64 feature set was not observed. However our customer does not rely on special feature sets nor does he really have compute heavy workloads. So your own experience may vary.