Error Code [72:106] Failed virtual machines lists the VDDK version as too high

Issue

When backing up certain virtual machines you may see an error similar to the description below.

Failure Reason: ERROR CODE [91:129]: Failed to back up all the virtual machines. Please check event viewer for individual virtual machine failure message.
Source: proxy, Process: vsbkp

Files failed to back up:
[VirtualMachine1] VDDK version too high.
[VirtualMachine2] VDDK version too high.
[VirtualMachine3] VDDK version too high.

VDDK Support for the Virtual Server Agent with VMware

The Virtual Server Agent (VSA) for Commvault Service Pack 10 or later includes multiple VDDK versions, including the latest certified VDDK version and an earlier version that can be used for backups of ESX versions that are no longer supported by VMware.

• VDDK 6.7.2: Will be loaded for vCenters running vSphere 6.7 Update 2 or later.
• VDDK 6.7.1: Will be loaded for vCenters running vSphere 6.7 or 6.7 Update 1.
• VDDK 6.7.0 EP1: Will not be loaded but is available if needed.
• VDDK 6.5.2: Will be loaded for vCenters running vSphere 6.5.
• VDDK 6.5: Will not be loaded but is available if needed. VDDK 6.5 only supports ESXi 5.5, ESXi 6.0, and ESXi 6.5.
• VDDK 6.0.2: Will be loaded by default for vCenters that do not require VDDK 6.5. This VDDK can be used for ESXi versions prior to ESXi 5.5 that are supported by Commvault.

The VSA package includes VDDKs in the following folder structure:

\Base\Vmware\VDDK\bin
\Base\Vmware\VDDK\VDDK672\bin
\Base\Vmware\VDDK\VDDK671\bin
\Base\Vmware\VDDK\VDDK670\bin
\Base\Vmware\VDDK\VDDK652\bin
\Base\Vmware\VDDK\VDDK650\bin
\Base\Vmware\VDDK\VDDK602\bin

The VDDK is not loaded until the VDDK is needed for a backup or restore operation. Once a VSA proxy has loaded a more recent VDDK version, it cannot revert to using an older version.

After upgrading to VDDK 6.5.2, the first backup might use the wrong transport mode, but subsequent jobs will honor the configured transport mode. This can occur during a multi-proxy backup when the coordinator node detects the newer vSphere version, but other proxies that are involved in the backup are still using the older VDDK. As a result, the first backup might use NBD instead of SAN mode.

Resolution

Use separate VSA proxies to support different VDDK versions

A single physical or virtual machine can only support one active VDDK version, and different VSA proxies must be configured to support different VDDK versions. If VSA proxies are deployed as virtual machines, the VSA proxy VMs supporting different VDDK versions must be hosted on different ESX servers. In environments that include vSphere 6.5 or 6.7, as well as ESX hosts that are not supported by vSphere 6.5 or 6.7, configure separate VSA subclients and assign different VSA proxies to each of the subclients:

• Configure one subclient and VSA proxy to support older hosts with proxies using VDDK 6.0.2
• Configure a separate subclient and VSA proxy to support vSphere 6.5 with proxies using VDDK 6.5.
• Configure a separate subclient and VSA proxy to support vSphere 6.7 with proxies using VDDK 6.7.1 or 6.7.2.

There is an option to force the proxy to use a specific VDDK. This is not a way to work around the VMware limitation for the VDDK versions, but it can help with troubleshooting.

System Requirements for Virtual Server Agent with VMware Virtual Disk Development Kit 6.0.2 Release Notes Virtual Disk Development Kit Release Notes for vSphere 6.5