NetApp Backup fails when unable to remove snapshot

Ontap 9.0P2 allows filer to manually remove restartable backup jobs causing possible race condition when manual removal done at same time as removal by Commvault software

Symptoms

NAS.Log entries similar to:

Data get_env failed NDMP_UNDEFINED

NasBackupSession::getBackupEnv() - data_get_env() failed.

End of backup - not committing the entire index.

UPDATEJM: Stream Restart FileCountDelta:[2408] FileCount:[2408]

Causes

Known NetApp bug: 1048565.

"The restartable backup cleanup daemon deletes invalid and obsolete restartable backup contexts. It runs as per a pre-defined interval. If a manual deletion of a restartable context is attempted through command line interface (CLI) or Data Management Application (DMA), and the daemon also happens to be accessing the same restartable context at that time, the manual deletion operation might result in a system disruption due to a race condition between the two operations."

Resolution

Commvault manually disabled this feature on the filers so only our software will remove the snapshots.