CommServe response time seems slow affecting jobs, reports, and CommCell Console performance

CommServe performance appears degraded or slow. Jobs are not progressing or there are a high number of jobs waiting in the queue. Report generation takes longer than normal/expected. CommCell Console updates, screens take longer to load and display

Symptoms

Invetigation shows one of more of the folling symptoms

Performance degrades afer OS upgrade or update - or over a period of time.
Slow SQL database I/O performance
SQL Lock timeouts and deadlocks in job manager logs
Running custom SQL queries on the CommServe database
Repeated resource allocation failures
Resource allocation loop takes additional time
CommCell Console updates/screen refresh take longer to load and display

Causes

Possible causes

System requirements for level of activity not met.
Available disk space for SQL Transaction logs is low. This can result from Full SQL Recovery mode being enabled and transaction logs not being backed up and pruned)
When a custom SQL query is run on the CommServe database this creates locks on SQL tables.
Incorrect data path configuration in a Storage Policy causing resource allocation to loop and take longer than expected.
Resource allocation for a job can fail repeatedly if there are lots of jobs running on a given Storage Policy
Temp space is low (Database space check warnings issued)
Lack of useful DB statistics. SQL Server 2008 R2 or SQL Server 2012 creates and destroys some memory structures about global statistics instead of caching them. (Microsoft Hotfix available)
Slow network communication.
High number of CommCell Console or WebConsole sessions open.
(HP Proliant) HP Power Profile not set to High Performance (default is set to Balanced Power and Performance)

Resolution

Close unnecessary CommCell Console/Web Console sessions. If persistent high number of Web Consoles are open consider moving WebConsole server and client to different host.
Verify sufficent disk space, I/O performance, and memory on CommServe host. Review System Requirements for all Commvault compnents installed on CommServe host.
Run DBMaintenance to refresh SQL indexes and statistics.
Monitor Disk I/O. Check for long Disk I/O queue/waits. If needed, move/separate high disk I/O applications on different disks. Performance may be improved by separating SQL databases, OS swap space (pagefile.sys), Commvault Index Cache, and Commvault logs on different disks.
Check network bandwidth/performance. Slow performance on the CommServe may be the result of network congestion or incorrect network performance settings.
Review the following Microsoft KB articles:
If host is HP Proliant, check BIOS Power Profile is set to High Performance.
Check the custom SQL queries to ensure they do not cause locks on SQL tabels. The SQL statements should be written with NO LOCKS specified where appropriate.
Review the data path configuration for correctness
Review the number of subclients being backed up concurrently to the same storage policy and redistribute them appropriately.
If performance degrade occurs after OS upgrade/update and none of the above actions improve performance, contact customer support. You may need to do a fresh install of the new OS and re-install Commvault software on the host. Make sure you have several DR Backup copies (Database dumps) and latest Commvault software and services packs before proceeding.