A trusted Spotfire Node Manager node may not be able to communicate with Spotfire Server and appears offline

A trusted Spotfire Node Manager node may not be able to communicate with Spotfire Server and appears offline.

In server.log file on Spotfire Server, the following message may appear:
   
    server.http.HttpCallable: The trust relationship has been revoked.

Followed by the WARN message listed below:

The "403 Forbidden" messages may be due to the following:

1. There may be a network issue and hence the Spotfire Server assumes that the certificate may have been revoked. 
2. The Spotfire Server is unable to copy the renewed keystore file to the Node Manager machine. 

In both the scenarios listed above, we need to make sure to resolve any network issues between the Spotfire Server and the Node Manager.

Also, follow the steps given below to clean up the old keystore from both the Spotfire Server and the Node Manager machines, as this can help resolve the issue:

1. Stop the Node Manager service.
2. Login to Spotfire Administration web console UI.
3. From within the Nodes & Services section, revoke trust for the Node.
4. Go to untrusted nodes tab and delete the node from there. Note: Please repeat the first 3 steps for every Node Manager machine in the cluster.
5. Now stop the Spotfire Server service. 
6. On the Spotfire Server, go to <Spotfire server installation directory>\nm\trust" and delete the keystore here.
7. On the Node Manager server, go to "<Spotfire Node Manager installation directory>\nm\trust" and delete the certificate here. Note: Repeat the previous 2 steps for every Spotfire Server in the cluster.
8. Start the Spotfire Server service. This will create the keystore file on the Spotfire Server machine.
9. Start the Node Manager service.
10. Log in to Spotfire Administrator UI, go to "Nodes & Services > Untrusted nodes" and wait for the node to show up.
11. Once the node shows up in the "Untrusted Nodes", trust the node.

Doc: Revoking trust of a node

• https://docs.tibco.com/pub/spotfire_server/14.0.6/doc/html/TIB_sfire_server_tsas_admin_help/server/topics/revoking_trust_of_node_managers.html

Doc: Trusting a node

• https://docs.tibco.com/pub/spotfire_server/14.0.6/doc/html/TIB_sfire_server_tsas_admin_help/server/topics/trusting_a_node.html