Spotfire Server: Ignite is causing server restarts with "slowClientQueueLimit exceeded"

The Ignite component of Spotfire Server can cause multiple restarts of the server with the following error observed in the server.log

WARN 2024-03-30T14:17:43,942+0000 [] communication.tcp.TcpCommunicationSpi: Client node outbound message queue size exceeded slowClientQueueLimit, the client will be dropped (consider changing 'slowClientQueueLimit' configuration property) ...

The Ignite system maintains an outbound message queue for communication. The size limit of this queue is controlled by the parameter slowClientQueueLimit

When this limit is exceeded it can present the error above in the server logs and prevent the spotfire server from starting. There is also a MessageQueueLimit which controls message back-pressure for server nodes. The slowClientQueueLimit should be set to a lower value than MessageQueueLimit

To resolve this issue, the following configuration changes need to be implemented:

1. Export the server configuration:

config.sh export-config

2.) Set the slowClientQueueLimit to 1999 (this is arbitrarily chosen as an estimate of a sensible value):
config.sh set-config-prop -c configuration.xml -n clustering.apacheignite.slow-client-queue-limit -v 1999

3.) Set the MessageQueueLimit to a greater value:

config.sh set-config-prop -c configuration.xml -n clustering.apacheignite.message-queue-limit -v 2000

4. Import the configuration back into Spotfire:

config.sh import-config -c "Set Ignite queue limits"

5. Restart the Spotfire service.

If the issue persists, please contact Revvity Signals Support for further assistance.

Further reading:
https://ignite.apache.org/releases/latest/dotnetdoc/api/Apache.Ignite.Core.Communication.Tcp.TcpCommunicationSpi.html