Increase in Backup Time after Fix for VMware CBT Error for expanded VMDKs (Fixed in Service Pack 9)

After a virtual machine disk (VMDK) is expanded beyond 128 GB or any doubling of 128 GB, VMware ESXi 4.x or ESXi 5.x can return incorrect information about virtual machine disk sectors, and Changed Block Tracking (CBT) can produce unrecoverable backups.

This is a flaw in VMware APIs used by backup software, and can lead to data loss if not addressed immediately.

Symptoms

SP9 provides a crucial fix to prevent data loss. As a result of applying this service pack, the time and disk space for backups is increased.

Any virtual machine is at risk if VMDKs have been expanded beyond 128 GB, 256 GB, 512 GB, 1024 GB, or any doubling of 128 GB. For example, the issue can occur after expanding a VMDK from 100 GB to 150 GB, or after expanding a VMDK from 150 GB to 300 GB.

Any backup performed since the disk expansion can contain invalid data, including full backups.

Causes

The VMware command QueryChangedDiskAreas("*"), which is used to return a list of allocated disk sectors, returns an incorrect list of allocated virtual machine disk sectors and does not return allocated portions of the VMDK file.

For more information, see QueryChangedDiskAreas command returns incorrect sectors after extending virtual machine vmdk file with Change Block Tracking (CBT) enabled (2090639).

Resolution

For Service Pack 9 and later, the Commvault software automatically detects expanded disks and resets CBT as described below.

For Version 10 Service Pack 8:

The following HotFixes resolve this issue for all virtual machines:

• SP8-HotFix 71 CBTAutoReset WinX64
• SP8-HotFix 71 CBTAutoReset Win32

To access one of these HotFixes, use the links above or go to Cloud Services, log in, access the Download Center, and select ServicePack/Hotfix > SP8.

Install the HotFix on Virtual Server Agent (VSA) proxies.

After applying the HotFix or Service Pack 9, the following actions are taken:

1. The first backup job resets CBT for all virtual machines with a disk size greater than 128 GB. The CBT Status for the affected virtual machines is displayed as Reset.
2. The software stores the size of each VMDK file in the .vmx file for each virtual machine.
3. For any backup jobs going forward, the backup job compares the current size of each VMDK file with the size recorded in the .vmx file.
4. If a disk has been expanded, the software resets CBT and backs up all blocks for the virtual machine's VMDKs.

For older service packs and Versions 9__or 8:

You can use the CheckChangeTracking tool to reset CBT on specific affected virtual machines. This tool resets CBT without requiring that the virtual machine be restarted.

1. Go to Cloud Services, log in, access the Download Center, and select Tools > Virtual Server.
2. DownloadCVToolCheckChangeTracking and extract the contents to a folder on any Virtual Server Agent proxy for VMware.
3. For each affected virtual machine, perform the following steps to reset CBT.
   1. Navigate to the folder where the tool was extracted.
   2. From a command prompt, run the following command to disable CBT for the virtual machine: CheckChangeTracking -hostesx_or_vcenter_host-userusername-passwordpassword-vmvirtual_machine_name-disable
   3. Create a user-defined snapshot for the virtual machine to get updated disk usage information.
   4. Delete the snapshot.
   5. Run the following command to enable CBT for the virtual machine: CheckChangeTracking -hostesx_or_vcenter_host-userusername-passwordpassword-vmvirtual_machine_name-enable

The tool automatically re-enables CBT for the next backup job, which will back up all blocks for VMDKs even if an incremental backup is requested. CBT is also enabled for future incremental backups.

More Information

As a result of resetting CBT (by using the current service pack, applying the HotFix, or using the CheckChangeTracking tool), the time required for backups increases because the first backup after resetting CBT backs up all blocks for any virtual machines on which CBT was reset. This change can also increase disk storage requirements for backups. The change does not affect scheduled backup cycles or data aging.