Katalyst: Performance problems after infrastructure changes

Description

• After changes to VMs, storage, or network (e.g., migration to new hypervisor, SAN, or load balancer), users report:
  • Slow loading of experiments or plates.
  • Timeouts or intermittent errors (HTTP 502, 504).
  • Delays in attaching or processing analytical data.
• Katalyst and Spectrus DB versions have not changed; only the infrastructure did.

Solution

• Collect baseline and current metrics:
  • CPU, memory, and disk I/O on Katalyst and Spectrus DB servers.
  • Network latency and bandwidth between app, DB, and file shares.
• Common causes and checks:
  • Storage performance: new SAN/NAS may have higher latency or limited IOPS.
  • Network routing: additional hops or firewalls can increase latency.
  • Resource sizing: CPU/memory reduced by new virtual machine templates.
• Actions:
  • Verify that the new infrastructure meets or exceeds the original resource specifications recommended for Katalyst.
  • Optimize:
    • Database configuration (e.g., memory, connection limits).
    • JVM and application server settings for Katalyst.
  • Review proxy/load balancer timeout settings in case slower backends require longer timeouts.
• After tuning:
  • Test typical workloads (open experiments, run reprocessing, export data).
  • Monitor for a period to ensure stability under real user load.