Error Code: [24:42] Description: Cannot start restore program on host [10.10.10.10*sourceclient_recovery*8111*8402] - a network error occurred or the product's services are not running. Source: MediaAgent, Process: fsIndexedRestore

During a File System restore or a 1-Touch restore, the MediaAgent's fsIndexedRestore process is unable to connect to the sourceclient_recovery.

Symptoms

The MediaAgent (MA) was unable to open a tunnel to the target client on the CVFWD port. For normal clients, the default port is 8403. For 1-Touch clients, CVFWD is bound on port 8114. The Job Details page displayed this error message.

• Error Code: [24:42] Description: Cannot start restore program on host [10.10.10.10*sourceclient_recovery*8111*8402] - a network error occurred or the product's services are not running. Source: MediaAgent, Process: fsIndexedRestore

Causes

This could occur because of different root causes.

• The MA is unable to resolve the target client Host Name.
• The MA is not using the proper network route to reach the target client.
• Port 8403 for File System restore, or 8114 for 1-Touch restore, is blocked by a network firewall.
• The MA is using a larger MTU than the target client, or a network device such as a router, between them. This device has a lower MTU than the MA and target client, which is fragmenting packets.

Resolution

The ideal option is to open the ports on the network, or address any routing, resolution or other network related issues.

To resolve the issue during a restore operation that must be completed quickly, perform the following work around:

1. For 1-Touch, open the target client's console, where the YaST2 GUI is running. For Linux clients, SSH into the target client.
2. For 1-Touch, press ALT+F2 or ALT+F3 to open a new TTY. On either 1-Touch or a UNIX client, ensure that the MediaAgent is resolvable, and that the following command works, replacing the 'MediaAgent' with the actual MA host name or a Fully Qualified Domain Name.
   • /opt/commvault/Base/cvping MediaAgent 8403
   • If applicable, change 8403 with your custom CVFWD port that the MA is configured to listen on.
3. For 1-Touch, return to the YaST2 GUI using ALT+F1 or using the command
   • chvt 1
4. In the CommCell Console or Command Center, create two client groups, '1touch_clients' and '1touch_infrastructure'.
5. Add the \_recovery client to the 1touch_clients group, and the MediaAgents from the source client's Storage Policy into the 1touch_infrastructure group. There is no need to add the source client itself to any of these groups.
6. Create a One Way Topology from the 1touch_clients group, to the 1touch_infrastructure group, making sure to enable the Encrypt traffic option.
7. Resume the job once the topology has been created.

Note:

1. If you notice the same error message coming from a different MA, add that MA to the 1touch_infrastructure group, and push the network topology.
2. If the message is coming from the same MA, go back to the target client and verify the cvfwd.log for connectivity failures/errors with the MA.