Troubleshooting R and TERR Services: Resolving Service Instability and Resource Limitations

Product:

Spotfire Service for R, Spotfire Enterprise Runtime for R (TERR)

Keywords:

R service error, TERR service error, Spotfire service instability, JavaOptions, JVM size, Node Manager configuration, performance issues, troubleshooting R, TERR Services

Description:

This article addresses common issues related to the instability and unexpected shutdowns of R and TERR services in Spotfire environments. These issues often arise due to inadequate node configuration, insufficient memory allocation, or excessive resource usage, especially in multi-service node setups.

Sample Error:

WARN 2024-08-06T06:13:37,205-0400 [EQH-engineInitialization] topology.engine.AbstractEngineController: Marking terrsrv-8 in state INIT_START_EXECUTION_ENGINE as a failure: Failed on startup.

ERROR 2024-08-06T06:13:37,264-0400 [EQH-engineInitialization] topology.engine.EngineQueueHandler: An exception happened when creating a new engine: java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached

Resolution:

1. Ensure One Node per Service:

• One Node, One Service: To ensure optimal performance, it is recommended to have one dedicated node per service, especially when using R, TERR, or Python services. Multiple services on a single node can cause instability, even if some services are stopped.
• Steps:
  1. Remove all other services from the Node Manager via the Admin UI Console.
  2. Delete the removed service folders from the installation directory in the Node Manager.
  3. Restart the server to apply the changes.

2. Define JVM Size:

• JVM Memory Allocation: If JVM options are not explicitly set, Java will calculate defaults based on system memory and CPU cores. It is critical to define appropriate JVM sizes to ensure the services run efficiently.
• Steps:
  1. Define the JVM sizes using the JavaOptions parameter, e.g., -Xms2g,-Xmx4g
     JavaOptions:-Xms2g,-Xmx4g
  2. Export the custom.properties using the config export-service-config command.
  3. Modify the configuration and re-import it to apply changes to the TERR or R service.
• Documentation Reference: Service JVM Options

3. Performance Monitoring:

• Check System Resources: If the service continues to experience issues, monitor the system's memory and CPU usage to determine if resource limitations are the cause.
• Recommendation: Ensure the Node Manager server has at least 8 cores. Increase the number of cores and configure additional engines as needed based on usage.

Note:

If issues persist after applying the above resolutions, additional performance tuning or hardware upgrades may be necessary.

Conclusion:

Service instability in Spotfire Service for R and TERR can be mitigated by ensuring proper configuration of nodes, JVM settings, and resource allocation. Following the recommended guidelines helps maintain service stability and prevents unexpected shutdowns.