Error Code 19:1131 Waiting for the services on the client [...] to come online.

Jobs sit in the Job controller waiting with Error Code 19:1131 Waiting for the services on the client [...] to come online.

Symptoms

This issue occurs when one or more jobs sit in the Job controller waiting with no progress.  These clients may have had their services cycled, or the Operating System may have rebooted, though that is not always the case.

Causes

The error message is generally caused when the Commserve has lost network connectivity to the client services and cannot regain them.

There are multiple causes for this error message:

1. The client Services are not online yet or in a stuck/bad state
2. Network connectivity is inconsistent
3. The required ports are not open (port 8400, 8401, 8403),
4. The CS hostname, Client name and Client hostname are incorrect in registry

Resolution

Cause 1:The client Services are not online yet or in a stuck/bad state Do a check readiness on the Clients or Media Agents from CommCell console.  This task will auto correct this issue.  If not, log onto the client having the issue and confirm the status of the services.  Cycle them if necessary.  If they do not start, then the issue is likely another cause.

Cause 2: Network connectivity is inconsistent This can happen if there are issues across the network and traffic is inconsistent.  As a workaround (to get the job to complete), create a One Way Firewall as per the instructions here: https://documentation.commvault.com/11.21/expert/7220_setting_up_connections_on_commserve_computer_for_one_way_direct_connection_commserve_to_client.html

Cause 3:The required ports are not open (port 8400, 8401, 8403) If the services are online, run port listening tests to see if the required ports are open.  If not, check with your network administrator.  Check all required ports here: https://documentation.commvault.com/commvault/v11/article?p=105347.htm

Cause 4: The CS hostname, Client name and Client hostname are incorrect in registry On the client in question, confirm the following registry keys are correct: 1. Under HKEY_LOCAL_MACHINE\SOFTWARE\CommVault Systems\Galaxy\Instance001\CommServe\sCSHOSTNAME, confirm the Commserve Host Name is accurate 2. Under KEY_LOCAL_MACHINE\SOFTWARE\CommVault Systems\Galaxy\Instance001\CommServe\sCSClientName, confirm Commserve Client name is accurate 3. Under HKEY_LOCAL_MACHINE\SOFTWARE\CommVault Systems\Galaxy\Instance001\Machines\<Client Name>\sHOSTNAME, confirm the Client Host Name is accurate