CommServe response time seems slow affecting jobs, reports, and CommCell Console performance
Article ID: CS0017 CommServe performance appears degraded or slow. Jobs are not progressing or there are a high number of jobs waiting in the queue. Report generation takes longer than normal/expected. CommCell Console updates, screens take longer to load and display
Invetigation shows one of more of the folling symptoms
- Performance degrades afer OS upgrade or update - or over a period of time.
- Slow SQL database I/O performance
- SQL Lock timeouts and deadlocks in job manager logs
- Running custom SQL queries on the CommServe database
- Repeated resource allocation failures
- Resource allocation loop takes additional time
- CommCell Console updates/screen refresh take longer to load and display
- System requirements for level of activity not met.
- Available disk space for SQL Transaction logs is low. This can result from Full SQL Recovery mode being enabled and transaction logs not being backed up and pruned)
- When a custom SQL query is run on the CommServe database this creates locks on SQL tables.
- Incorrect data path configuration in a Storage Policy causing resource allocation to loop and take longer than expected.
- Resource allocation for a job can fail repeatedly if there are lots of jobs running on a given Storage Policy
- Temp space is low (Database space check warnings issued)
- Lack of useful DB statistics. SQL Server 2008 R2 or SQL Server 2012 creates and destroys some memory structures about global statistics instead of caching them. (Microsoft Hotfix available)
- Slow network communication.
- High number of CommCell Console or WebConsole sessions open.
- (HP Proliant) HP Power Profile not set to High Performance (default is set to Balanced Power and Performance)
- Close unnecessary CommCell Console/Web Console sessions. If persistent high number of Web Consoles are open consider moving WebConsole server and client to different host.
- Verify sufficent disk space, I/O performance, and memory on CommServe host. Review System Requirements for all Commvault compnents installed on CommServe host.
- Run DBMaintenance to refresh SQL indexes and statistics.
- Monitor Disk I/O. Check for long Disk I/O queue/waits. If needed, move/separate high disk I/O applications on different disks. Performance may be improved by separating SQL databases, OS swap space (pagefile.sys), Commvault Index Cache, and Commvault logs on different disks.
- Check network bandwidth/performance. Slow performance on the CommServe may be the result of network congestion or incorrect network performance settings.
- Review the following Microsoft KB articles:
- If host is HP Proliant, check BIOS Power Profile is set to High Performance.
- Check the custom SQL queries to ensure they do not cause locks on SQL tabels. The SQL statements should be written with NO LOCKS specified where appropriate.
- Review the data path configuration for correctness
- Review the number of subclients being backed up concurrently to the same storage policy and redistribute them appropriately.
- If performance degrade occurs after OS upgrade/update and none of the above actions improve performance, contact customer support. You may need to do a fresh install of the new OS and re-install Commvault software on the host. Make sure you have several DR Backup copies (Database dumps) and latest Commvault software and services packs before proceeding.