Failed Client Authentication attempts slow down CommCell Performance

If a MediaAgent or Client cannot communicate with CommServe services, the MediaAgent or Client should immediately disable itself and remain disabled until re-enabled by administrator. This may not happen or may affect other CommCell components attempting to communicate with a deleted or deconfigured component.

Symptoms

One or multiple backup jobs start up but go straight to Pending State reporting any of the following errors:

Error Code19:1327 Attempt start error: [...]

Error Code 9:90 Authentication failed for host [...]. Network password does not match.

Error Code 40:52 Failed to contact CommServe Services from the MediaAgent[...] during the start of the Data Transfer operation. Please check the network connectivity from this MediaAgent to the CommServe and make sure services are running on the CommServe.

Error Code 22:30 Could not establish connection to [...] on attempt [...].

Error Code 19:1131 Waiting for the services on the client [...] to come online.

Error Code 19:1597 Failed to start phase [...] on [...] due to network error [...]. Will attempt to restart. Please check if this product'sservices are running on the remote host.

Error Code 7:77 Unable to run [...] on the client. [CVSession::authenticateClient]:Remote system [FQDNservername]. Failed authentication returned from server.

Error Code 7:102 Unable to run [...] on the client.

Error Code 7:111 Unable to run [...] on the client. [...]?

Error Code 9:36 [...]:Remote system [...]. Could not get the password from platform information or the database -authentication failed.

Error Code 9:91 CVSession Authentication failed for host [...]. Password is not available on the host.

Error Code 9:90 Authentication failed for host [...]. Network password does not match.

Error Code 9:89 Authentication failed for host [...]. Could not find the client entry in database.

Error Code 9:42 [...]:Remote system [...]. Password received from client did not match the database password for that client - authentication failed.

Error Code 9:41 [...]:Remote system [...]. Could not encrypt challenge string when connecting - authentication failed.

Error Code 9:40 [...]:Remote system [...]. Unexpected message type received [...=...] - authentication failed.

Error Code 9:39 [...]:Remote system [...]. Failed authentication returned from client.

Error Code 9:38 [...]:Remote system [...]. Failed authentication returned from server.

Error Code 9:37 [...]:Remote system [...]. Could not find the client password in the database - authentication failed.

Causes

MediaAgent or Client services are programmed to automatically disable themselves from further communication to CommServe when the service detects that it is no longer part of CommCell. If the software is not uninstalled, this automatic disablement of services may not always happen. In certain cases, disabling of client services could result in other machines not being able to communicates to these hosts causing the symptoms described.

Resolution

Method 1: Do a check readiness on the Clients or MediaAgents from CommCell console. This task will auto correct this issue.

Method 2. Check the registry keys' status on the CommCell Component(s) affected.

For Windows OS systems:

Under HKEY_LOCAL_MACHINE\SOFTWARE\CommVault Systems\Galaxy\Instance001\Session\

If the current nChatterFlag registry key data decimal value is (1), it must be set to (0) (Zero) default data decimal value.

Setting this to (0) enables communication to the CS

Under HKEY_LOCAL_MACHINE\SOFTWARE\CommVault Systems\Galaxy\Instance001\CommServe\

If the current sCSHOSTNAME registry key string value has _Disabled_ appended in front of the string value; remove the _Disable_ to allow communication to the CommServe.

If the Commvault Firewall Utility is deployed as a controlled connection between the MediaAgent or Client and the CS the following additional registry key will be modified.

Under HKEY_LOCAL_MACHINE\SOFTWARE\CommVault Systems\Galaxy\Instance00x\Firewall\

If the current nDISABLE_CVFWD registry key data decimal value is (1), it must be set to (0) (Zero) default data decimal value.

Setting this to (0) enables controlled connection to the CommServe.

For UNIX/Linux OS systems:

It is recommended to use Commvault stop for the Commvault software services on UNIX/Linux MA and CL systems prior to making the changes to the .properties files.

Under /etc/CommVaultRegistry/Galaxy/Instance001/Session/.properties

If the current nChatterFlag entry value is (1), it must be set to (0) (Zero) default value.

Setting this to (0) enables communication to the CS.

Under /etc/CommVaultRegistry/Galaxy/Instance001/Commserve/.properties

If the current sCSHOSTNAME entry value has _Disabled_ appended in front of the value; remove the _Disable_ to allow communication to the CommServe.

If the Commvault Firewall Utility is deployed as a controlled connection between the MediaAgent or Client and the CS the following additional registry key will be modified.

Under /etc/CommVaultRegistry/Galaxy/Instance001/Firewall/.properties

If the current nDISABLE_CVFWD entry value is (1), it must be set to (0) (Zero) default value.

Setting this to (0) enables controlled connection to the CommServe.

The Commvault software services on the MediaAgent(s) or Client(s) must be recycled. Use the Process Manager utility for Windows systems. On UNIX/Linux OS systems use Commvault start for the Commvault services.

Test Backups and Restores to confirm data protection has been restored.