Spotfire: newly added Node Manager is offline: PKIX path validation failed: java.security.cert.CertPathValidatorException: OCSP response error: TRY_LATER

Problem:

A newly added Node Manager node is failing to join Spotfire cluster, first getting successfully auto-trusted, but then appearing with "Offline" status in the Spotfire web admin console and reporting a certificate error in nm.log file:

WARN 2025-07-22T20:03:19,974+0000 [Initialization] security.trust.KeyStoreUtil: The certificate chain is missing.
INFO 2025-07-22T20:03:20,177+0000 [Initialization] server.nodemanager.NodeManagerInitializer: Started
INFO 2025-07-22T20:03:20,177+0000 [AcquireAuthorization] nodemanager.trust.DelayUntilAuthorizedGate: Checking to see if we (10.255.8.75 3e3dd1ce-931e-4f07-98c6-9b79421ca454) are trusted. Using 10.255.8.17:9443
WARN 2025-07-22T20:03:20,536+0000 [ocsp-request-1] web.controller.RemoteTrustController: #1 Failed to use MFUwUzBRME8wTTAJBgUrDgMCGgUABBQChGE5r%2B39eZU60pa5ClB1%2Fh0xEQQUTAEvXGmBD7Ongin8ehkXMgbulIkCFCwkvx%2B1U84J172oAsKqI50eiSc4:10.255.8.17 for ocsp request '9080', continuing trying other nodes
org.springframework.web.client.HttpClientErrorException$BadRequest: 400 : "{"errorMessage":"The request was invalid or incomplete","httpErrorCode":400}"
at org.springframework.web.client.HttpClientErrorException.create(HttpClientErrorException.java:103) ~[spring-web-6.1.14.jar:6.1.14]
at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:183) ~[spring-web-6.1.14.jar:6.1.14]
at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:137) ~[spring-web-6.1.14.jar:6.1.14]
at org.springframework.web.client.ResponseErrorHandler.handleError(ResponseErrorHandler.java:63) ~[spring-web-6.1.14.jar:6.1.14]
at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:942) ~[spring-web-6.1.14.jar:6.1.14]
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:891) ~[spring-web-6.1.14.jar:6.1.14]
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:830) ~[spring-web-6.1.14.jar:6.1.14]
at org.springframework.web.client.RestTemplate.getForEntity(RestTemplate.java:447) ~[spring-web-6.1.14.jar:6.1.14]
at com.spotfire.server.nodemanager.remote.web.controller.RemoteTrustController.performOcspGetRequest(RemoteTrustController.java:431) ~[classes/:?]
...
WARN 2025-07-22T20:03:20,552+0000 [ocsp-request-1] web.controller.RemoteTrustController: #1 No servers responded correctly, try later
ERROR 2025-07-22T20:03:20,583+0000 [AcquireAuthorization] nodemanager.trust.DelayUntilAuthorizedGate: Could not connect to supervisor Node [serverName=10.255.8.17, port=9443]: PKIX path validation failed: java.security.cert.CertPathValidatorException: OCSP response error: TRY_LATER

This appears to correspond to server.log error:

WARN 2025-07-22T20:03:20,502+0000 [unknown, #-, #46] web.controller.IllegalArgumentExceptionHandler: 
Exception for /spotfire/nodemanager/trust/ocsp/MFUwUzBRME8wTTAJBgUrDgMCGgUABBQChGE5r%252B39eZU60pa5ClB1%252Fh0xEQQUTAEvXGmBD7Ongin8ehkXMgbulIkCFCwkvx%252B1U84J172oAsKqI50eiSc4 
from 10.255.8.75. Message Illegal base64 character 25

While troubleshooting this issue, one might find that:

- There were no recent network or security changes in this environment.

- Both machines, Spotfire Server and Node Manager, can connect to each other on ports 9080 and 9443 by testing connections using Windows PowerShell command Test-NetConnection -ComputerName "[IP-address]" -Port "[9080 or 9443]".

- Uninstalling Node Manager, removing its directory, untrusting it, restarting the machine and reinstalling Node Manager doesn't resolve the issue.

- Switching to using IP addresses (instead of hostnames) in nodemanager.properties doesn't resolve the issue.
 

Solution:

To resolve this issue, the following steps need to be performed on the Node Manager machine:
 
1. Navigate to the directory <Node Manager home>\nodemanager\<version>\nm\.
2. Open file winsw.xml for Editing:
3. Add the following system property in the <services> section in the winsw.xml file:
 

<service>

   ...
   <env name="JAVA_OPTS" value="-Dspotfire.ocsp.use-local-proxy=false"/>

</service>
 
4. Save the winsw.xml file, restart Node Manager and see if it's able to reconnect to Spotfire Server.

 
This problem might be caused by the use of a local OCSP proxy, which doesn't properly URL-encode Base64 OCSP requests. This workaround ensures that requests are sent directly to the OCSP responder, bypassing local OCSP proxy. The fix for this issue has already been implemented in more recent releases of Spotfire Server.