Error Code 19:1161 The number of running jobs has exceeded the limit on the maximum number [<Number>] allowed.

Job Pending Reason error code [19:1161]

Symptoms

With many active jobs running concurrently, some jobs go to pending state with error code 19:1161

Causes

This error message is delivered because the High Watermark Level for CommCell jobs was reached or exceeded. Alternate cause - Library, drives, or mountpaths are offline or inaccessible.

Resolution

Verify all library, drives, and mountpaths are online and accessible by configured MediaAgents.  Verify sufficient number of writers (equal to or more than aggregate streams from concurrently used storage policies) for disk library and mulitplexing factor (if enabled) for tape library.

To change the High Watermark setting if it may too low for the number of job streams concurrently running, use the following steps to update the high watermark setting from the CommCell Console:

1. Suspend all jobs and kill the jobs that cannot be suspended. When changing the high watermark setting, no jobs should be running in the Job Controller window or in the command line. We recommend to take screen captures of the jobs in the Job Controller to easily restart the ones that will be killed.
2. From the CommCell Console toolbar menu, click Control Panel.
3. Under the System section, click Job Management.
4. In the High Watermark Levelbox, add 25 to the current value and click OK. Monitor the CommCell and check if the issue subsided. If the issue persists, increase the High Watermark Level option again by 25. Do this until the issue subsides.
5. Restart all services. See Restarting a Service for more information.
6. Resume all suspended jobs and restart the jobs that you killed.
7. Monitor the CommCell and check if the issue subsided. If the issue persists, increase the High Watermark Level option again by 25. Do this until the issue subsides.