Katalyst: Handling 502 Proxy errors on RPC calls, general approach

Description

• Users may receive errors in Katalyst where the UI shows a generic server error and the stack trace includes:
  StatusCodeException: 502 Proxy Error with a message like:
  “The proxy server received an invalid response from an upstream server. Reason: Error reading from remote server.”
• Affected calls can include:
  • /Katalyst/dispatch/RemoteExperimentNames
  • Export‑related RPC actions, etc.
• The Katalyst application itself may still be running.

Solution

• Treat any 502 Proxy Error as primarily a network/infrastructure concern:
  1. Check upstream server health
     • Confirm the Katalyst app server is running and reachable directly (bypassing the proxy, if possible).
     • Look for application errors in Katalyst logs at the time of the 502.
  2. Review proxy/load balancer configuration
     • Increase or align:
       • Client‑side timeout (how long the proxy waits for browser requests).
       • Backend timeout (how long the proxy waits for Katalyst).
     • Confirm persistent connections/keep‑alive are configured appropriately.
  3. Consider usage patterns
     • Users often trigger 502s after:
       • Leaving sessions idle, then resuming work.
       • Running long operations (large exports, heavy calculations).
     • Adapt timeouts and session‑management policies to expected usage.
  4. Post‑fix validation
     • Reproduce the previously failing action (e.g., idle then resume, export large experiment).
     • Ensure no new 502 errors appear and the UI recovers gracefully.
• Keep a short runbook linking error text to proxy settings and responsible infrastructure contacts so that operations teams can react quickly.