The Virtual Machine Fails to Start After Upgrading the Hosted Engine OVA Version From v4.2 to v4.3

When you upgrade the hosted engine OVA version from v4.2 (rhvm-appliance-4.2-20190416.1.el7.ova) to v4.3 (rhvm-appliance-4.3-20190722.0.el7.ova), the virtual machine that hosts the hosted engine fails to start.

Symptoms

The following error appears in the vdsm.log file located in the /var/log/vdsm directory:

libvirtError: unsupported configuration: maximum vCPU count must not be less than current vCPU count
2019-11-11 00:20:56,662-0500 INFO  (vm/86ca19fe) [virt.vm] (vmId='86ca19fe-a96a-4bf8-8d35-73441b55e0da') Changed state to Down: unsupported configuration: maximum vCPU count must not be less than current vCPU count (code=1) (vm:1690)

When you attempt to start the hosted engine manually, the following message appears:

# hosted-engine --vm-start
VM exists and is down, cleaning up and restarting
VM in WaitForLaunch

Resolution

Perform the following steps on one of the HyperScale appliance nodes:

1. Save a copy of the virtual machine configuration file vm.conf on the hosted engine in the tmp directory, and then rename it custom-vm.conf:

   # cp /var/run/ovirt-hosted-engine-ha/vm.conf /var/tmp/custom-vm.conf
   

2. To make the custom-vm.conf file a legacy configuration file, remove the Base64 encoded XML from the file:

   # sed -i '/^xmlBase64=/d' /var/tmp/custom-vm.conf
   

3. If the hosted engine virtual machine has more than 32 vCPUs provisioned, set the value for maxVCpus to 32 or 64:

   # sed -i 's/^maxVCpus=16/maxVCpus=32/' /var/tmp/custom-vm.conf
   

4. Verify that the attribute values are set correctly:

   # egrep 'smp|maxVCpus' /var/tmp/custom-vm.conf
   smp=32
   maxVCpus=32
   

5. Set the mode for the hosted engine to global maintenance:

   # hosted-engine --set-maintenance --mode=global
   

6. Start the hosted engine virtual machine by using the edited custom-vm.conf file:

   # hosted-engine --vm-start --vm-conf=/var/tmp/custom-vm.conf
   

7. Remove the global maintenance mode for the hosted engine:

   # hosted-engine --set-maintenance --mode=none
   

8. After an hour, the virtual machine updates the configuration file /var/run/ovirt-hosted-engine-ha/vm.conf with the new maxVCpus value. This update allows you to start the hosted engine virtual machine without using the custom configuration file. Run the following command to verify the update:

   # egrep 'smp|maxVCpus' /var/run/ovirt-hosted-engine-ha/vm.conf
   smp=32
   maxVCpus=32
   

   The value is 32.

To make the change reflect in other Hyperscale appliance nodes, perform the following steps in the RedHat Virtualization portal.

1. Edit the value for the Memory Size. You can increase or decrease the value.
2. Click OK.
3. After an hour, run the command to verify the updated value.

   # egrep 'smp|maxVCpus' /var/run/ovirt-hosted-engine-ha/vm.conf
   smp=32
   maxVCpus=32
   

4. Shut down, and then start the hosted engine virtual machine.

   # hosted-engine --vm-shutdown
   # hosted-engine --vm-start
   

   The hosted engine virtual machine starts without any issues. Note: For more information, see the RedHat ticket: Hosted Engine virtual machine not starting up.