Troubleshooting TERR/R Service Shutdown Due to OutOfMemoryError and Native Thread Creation Issues

Keywords:
TERR service error, OutOfMemoryError, native thread, resource limits, Java options, memory management, JVM, TIBCO, TERR Server Edition, TERR/R Service shutdown, system cores, Node Manager.

Products:
TIBCO Enterprise Runtime for R - Server Edition, Open source R (Spotfire Service for R Server Edition).

Description:
Customers have encountered an issue where the TERR/R Service is shutting down unexpectedly with an OutOfMemoryError due to an inability to create native threads. This error indicates that the service may have reached process or resource limits, or is possibly running out of memory. The issue typically occurs in environments with a multiple TERR/R Services with other Services like Webplayer and Automation Services.

Resolution:

1. Optimize Node and Service Configuration:

   • Remove all other services from the node, ensuring that only 1 Node and 1 Service/Instance are maintained.
   • Use the Admin UI Console to remove the services from the Node Manager.
   • Delete the corresponding service folders from the installation directory within the Node Manager.
   • Restart the server.
     
     

2. Review System Core Requirements:

   • TERR/R Services are highly dependent on the number of cores available in the system. Ensure that your system meets the required number of cores based on your usage.
   • If necessary, increase the number of cores assigned to the Node Manager to improve performance and stability.
   • Refer to the Spotfire Enterprise Runtime for R System Requirements for detailed guidance on core requirements.
     
     

3. Address Java OutOfMemoryError:

   • If the error persists, it may be due to Java running out of memory. Define the Java Options in the configuration to optimize memory usage:
     • If no JVM options (especially memory settings) are configured, Java calculates defaults based on the machine's memory and the number of engines.
     • Set appropriate JVM sizes since the engines require minimal JVM memory as most work is done off-heap.
     • Define Java Options by adding the line JavaOptions:-Xms2g,-Xmx4g to the service configuration.

       Steps to follow:
       Export and Apply Configuration:

       • Export the current service configuration using the command config export-service-config.
       • Modify the custom.properties file to include the Java Options line.
       • Import and apply the updated configuration to the TERR service.

4. Next Steps:

   • If the issue persists after implementing these changes, contact Revvity support to investigate potential performance issues related to memory and CPU usage.

Additional Documentation:

• Managing Java Options for TERR
• Configuring the TERR Service
• Spotfire Enterprise Runtime for R System Requirements