CommServe response time seems slow affecting jobs, reports, and CommCell Console performance

Article ID: CS0017 CommServe performance appears degraded or slow. Jobs are not progressing or there are a high number of jobs waiting in the queue. Report generation takes longer than normal/expected. CommCell Console updates, screens take longer to load and display

Symptom

Invetigation shows one of more of the folling symptoms

  • Performance degrades afer OS upgrade or update - or over a period of time.  
  • Slow SQL database I/O performance
  • SQL Lock timeouts and deadlocks in job manager logs
  • Running custom SQL queries on the CommServe database
  • Repeated resource allocation failures
  • Resource allocation loop takes additional time
  • CommCell Console updates/screen refresh take longer to load and display

Cause

Possible causes

  • System requirements for level of activity not met.
  • Available disk space for SQL Transaction logs is low. This can result from Full SQL Recovery mode being enabled and transaction logs not being backed up and pruned)
  • When a custom SQL query is run on the CommServe database this creates locks on SQL tables.
  • Incorrect data path configuration in a Storage Policy causing resource allocation to loop and take longer than expected.
  • Resource allocation for a job can fail repeatedly if there are lots of jobs running on a given Storage Policy
  • Temp space is low  (Database space check warnings issued) 
  • Lack of useful DB statistics. SQL Server 2008 R2 or SQL Server 2012 creates and destroys some memory structures about global statistics instead of caching them.  (Microsoft Hotfix available)
  • Slow network communication.
  • High number of CommCell Console or WebConsole sessions open.
  • (HP Proliant) HP Power Profile not set to High Performance (default is set to Balanced Power and Performance)

 

Resolution

  • Close unnecessary CommCell Console/Web Console sessions. If persistent high number of Web Consoles are open consider moving WebConsole server and client to different host.
  • Verify sufficent disk space, I/O performance, and memory on CommServe host. Review System Requirements for all Commvault compnents installed on CommServe host. 
  • Run DBMaintenance to refresh SQL indexes and statistics.
  • Monitor Disk I/O. Check for long Disk I/O queue/waits. If needed, move/separate high disk I/O applications on different disks. Performance may be improved by separating SQL databases, OS swap space (pagefile.sys), Commvault Index Cache, and Commvault logs on different disks.
  • Check network bandwidth/performance.  Slow performance on the CommServe may be the result of network congestion or incorrect network performance settings.
  • Review the following Microsoft KB articles:
  • If host is HP Proliant, check BIOS Power Profile is set to High Performance.
  • Check the custom SQL queries to ensure they do not cause locks on SQL tabels.  The SQL statements should be written with NO LOCKS specified where appropriate.
  • Review the data path configuration for correctness
  • Review the number of subclients being backed up concurrently to the same storage policy and redistribute them appropriately. 
  • If performance degrade occurs after OS upgrade/update and none of the above actions improve performance, contact customer support. You may need to do a fresh install of the new OS and re-install Commvault software on the host. Make sure you have several DR Backup copies (Database dumps) and latest Commvault software and services packs before proceeding.