Spotfire Server: Cluster instability or Server restarts with warnings: Consider increasing the nodemanager.core-pool-size setting

Spotfire Server cluster instability or sporadic service restarts observed, with server.log WARN message "Consider increasing the nodemanager.core-pool-size setting."

There can be situations where multiple Spotfire Servers in a cluster and multiple Node Manager nodes (running multiple Web Player services) occasionally fail or restart under heavy work loads.

The Spotfire Server log (server.log) shows this warning:
--------------------------------------------------------------
WARN 2019-06-17T06:04:13,884-0300 [monitoring@SPOTFIRESYSTEM, #52307, #19748547] nodemanager.executor.ExecutorServiceFactory: The pool size 75, max 100 cannot add more tasks. 100 active tasks. Consider increasing the nodemanager.core-pool-size setting.
---------------------------------------------------------------

Solution:

- export the active Spotfire server configuration. Using "export-config" command.
- open the exported "configuration.xml" in edit mode.
- find the below tag:

<nodemanager>
<capacity>xx</capacity>
<worker-threads>xx</worker-threads>
</nodemanager>

- add the below lines. One may tweak the values that best suits the environment.

<core-pool-size>250</core-pool-size>
<maximum-pool-size>250</maximum-pool-size>

- ensure that the "Node Manager" setting in the Spotfire Server configuration looks something similar as below:

<nodemanager>
<capacity>xx</capacity>
<worker-threads>xx</worker-threads>
<core-pool-size>250</core-pool-size>
<maximum-pool-size>250</maximum-pool-size>
</nodemanager>

- save the file and import the configuration. Using "import-config" command.
- restart the Spotfire server for the changes to take effect.