Article ID: 55003
Article Type: Troubleshooting
Last Modified:
Data aging jobs on Oracle clients complete with an RMAN crosscheck failure message.
A data aging job with a status of "Completed w/ one or more errors" shows the following error:
Error Code: [32:320] Failed to Cross-Check Oracle Clients Source: [source], Process: [process]
In addition, one or more of the following events may be associated with the job details:
RMAN Crosscheck failed for Oracle instance [instance] on client [client].
Establishing connection to Client services failed during Crosscheck. Make sure the Product's services are running.
Failed to retrieve Oracle Instance Properties during crosscheck operation.
The Oracle crosscheck operation that runs during data aging updates the Oracle control file or the RMAN catalog database with information about aged backup pieces. The Data Aging job is marked completed with errors when one of the Oracle instances fails when running a crosscheck operation.
The following are the common causes of crosscheck failures:
If backups are actively running for Oracle or Oracle RAC instances these kinds of failures usually don't occur since Commvault services are running and database connections are successful. When a client or Oracle instance is no longer running backup operations or the server is in maintenance mode the above errors are possible.
If the Oracle server is offline and you want to prune the data on the Oracle client, bring up the Oracle server if it is not running or bring up the Oracle instance if it is DOWN. The next data aging job should then be able to finish the crosscheck operation.
Alternatively, you can disable the RMAN crosscheck operation for the Oracle instance. For more information, see Disabling the RMAN Crosscheck. When the crosscheck is disabled, the control file or the RMAN catalog database is no longer updated with information about aged backup pieces. Therefore, the DBA must run periodic crosscheck backup commands to invalidate aged backup pieces.
The events associated with the job details may show the following events, with the Oracle instance and client details for the job that failed:
RMAN Crosscheck failed for Oracle instance [instance] on client [client].
Establishing connection to Client services failed during Crosscheck. Make sure the Product's services are running.
For the above events, look at the DataAging.log file for the job ID of the job that indicated it completed with errors. The log file usually indicates that the network connection failed and that the remote computer refused the network connection. Bring up the Oracle server if it is not running or bring up the Oracle instance if it is DOWN.
For Oracle RAC data aging jobs, the job details may show the following event:
Failed to retrieve Oracle Instance Properties during crosscheck operation.
The Oracle RAC instance and client details are not displayed in the event message, but you can determine them by looking at the DataAging.log file, which contains log messages like the following. The messages indicate that none of the Oracle RAC instances are reachable. Start up at least one Oracle RAC server if none are running, or bring up at least one Oracle RAC instance.
9660 35a4 04/26 10:25:00 121 OraCrossCheck::RecvOraParams() - Unable to determine the OracleRacInstance to continue crosscheck. Please make sure at least One Instance is in OPEN/MOUNTED mode and resume the job
9660 35a4 04/26 10:25:00 121 OraCrossCheck::GetInstanceInfo() - RecvOraParams() failed for instanceId=9.
9660 35a4 04/26 10:25:00 121 OraCrossCheck::CrossCheckForOraClientAndInstance() - GetInstanceInfo() failed.
9660 35a4 04/26 10:25:00 121 OraCrossCheck::CrossCheckForOraClientAndInstance() - End Oracle CrossCheck for jobId=121 CommCellId=2 clientId=4 clientName=unixdb-rac instanceId=9, instanceName=rac1nde jobControl=000000D3DF0EB010. CROSSCHECK_STATUS=FAILURE
9660 35a4 04/26 10:25:00 121 OraCrossCheck::CrossCheckForOraClientAndInstanceLists() - CrossCheckForOraClientAndInstance() failed for clientId=4 instanceId=9.
9660 35a4 04/26 10:25:00 121 OraCrossCheck::CrossCheckForOraClientAndInstanceLists() - Failed to retrieve Oracle instance property from CS DB, m_oraSID is NULL or clientName is empty