TIBCO Spotfire Server fails to start after enabling Hazelcast clustering with error "Cluster members count is not equal online servers count."

TIBCO Spotfire Server fails to start after enabling Hazelcast clustering with error "Cluster members count is not equal online servers count."

When setting up the 7.6 and higher TIBCO Spotfire Server for clustering with Hazelcast you may see an issue where the first TIBCO Spotfire Server starts, but subsequent servers in the cluster fail to start. You will see the following in the server.log:

INFO 2017-03-02T22:27:39,685-0600 [*Initialization*] server.cluster.HazelcastClusterBase: Cluster instance 'spotfire' is created.
WARN 2017-03-02T22:27:39,685-0600 [*Initialization*] server.cluster.HazelcastInstanceHolder: Cluster members count is not equal online servers count./nCurrent cluster members: Member [TSSname1]:5701 this ||
Current online servers: [TSSname2, Version=7.7.0 , isPrimus=true, ServerId=b5a7dc52-770b-4b52-8c24-2fde78c78f15, Addresses=[TSSname2, 192.168.0.93, 10.20.64.85]] [TSSname1, Version=7.7.0 HF-002, isPrimus=false, ServerId=6a5ae436-d62d-409b-b979-b322d7492999, Addresses=[TSSname1, 192.168.0.92, 10.20.64.83]]
ERROR 2017-03-02T22:27:39,685-0600 [*Initialization*] server.cluster.RetryHelper: Function execution failed. Retry count: 1
java.lang.IllegalStateException: failed to validate cluster consistency
at com.spotfire.server.cluster.HazelcastInstanceHolder.lambda$init$8(HazelcastInstanceHolder.java:112)
at com.spotfire.server.cluster.RetryHelper.doWithRetry(RetryHelper.java:59)
at com.spotfire.server.cluster.HazelcastInstanceHolder.init(HazelcastInstanceHolder.java:98)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:349)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:300)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:133)
...

  On the TIBCO Spotfire Server machine, check if you have multiple network interfaces (Control Panel). If you do, will need to apply the latest Spotfire server hotfix. Then configure Hazelcast to Bind to Any Network Interface: 

In server hotfix TSS 7.7.0 HF-001 and TSS 7.6.0 HF-006, the option to configure Hazelcast to bind to any network interface was introduced. By default, the Hazelcast clustering component of Spotfire Server binds to the configured backend interface. If there are multiple interfaces and the Spotfire Server fails with an exception similar to "Cluster members count is not equal online servers count" it may be necessary to configure Hazelcast to bind to any network interface, using the following command (see the referenced documentation for detailed instructions on manually updating the configuration file): 

   config set-config-prop --name=clustering.hazelcast.bind-on-any-interface --value=true

Ensure the TIBCO Spotfire Servers are started one at a time (sequentially) Doc: Executing commands on the command line

• https://docs.tibco.com/pub/spotfire_server/7.11.1/doc/html/TIB_sfire_server_tsas_admin_help/GUID-6E14757E-1CD6-4FBD-9B48-2D54AFAD34EC.html