"Service communication error" message in the Automation job activity is seen while executing a job on TIBCO Spotfire Server

"Service communication error" message in the Automation job activity is seen while executing a job on TIBCO Spotfire Server

A "Service communication error" is seen in the activity message when the job is not executed due to the server not being able to communicate with TIBCO Spotfire Automation service due to (temporary) connectivity errors. Below is the screenshot for your reference:


You may see the following error in server.log:

ERROR 2020-03-31T11:00:44,067-0400 [automation-services-job-933cf742-8405-4bc3-92b3-9fb6fc0c6b62] scheduler.service.ASJobSender: An error occurred while executing job task a07b1a11-bce6-43cb-97bd-02279d3e9d24
org.springframework.web.client.ResourceAccessException: I/O error on POST request for "https://123.234.345.456:9502/1188f7a8-50fc-4948-bd5f-60c90c266101/AutomationService/StartJob": Connect to 123.234.345.456:9502 [/123.234.345.456] failed: connect timed out; nested exception is org.apache.http.conn.ConnectTimeoutException: Connect to 123.234.345.456:9502 [/123.234.345.456] failed: connect timed out
    at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:751) ~[spring-web.jar:5.2.3.RELEASE]
    at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:677) ~[spring-web.jar:5.2.3.RELEASE]
    at org.springframework.web.client.RestTemplate.postForObject(RestTemplate.java:421) ~[spring-web.jar:5.2.3.RELEASE]
    at com.spotfire.server.scheduler.service.ASJobSender.sendJob(ASJobSender.java:100) ~[spotfire-common-services.jar:?]
    at com.spotfire.server.scheduler.service.ScheduledJobsQueueProcessorManager.processQueuedTask(ScheduledJobsQueueProcessorManager.java:95) ~[spotfire-common-services.jar:?]
    at com.spotfire.server.scheduler.service.ScheduledJobsQueueProcessorManager.processQueuedTask(ScheduledJobsQueueProcessorManager.java:45) ~[spotfire-common-services.jar:?]
    at com.spotfire.server.scheduler.service.AbstractQueueManager.lambda$processQueuedItems$22(AbstractQueueManager.java:719) ~[spotfire-common-services.jar:?]
    at java.lang.Iterable.forEach(Iterable.java:75) ~[?:1.8.0_241]
    at com.spotfire.server.scheduler.service.AbstractQueueManager.processQueuedItems(AbstractQueueManager.java:717) ~[spotfire-common-services.jar:?]
    at com.spotfire.server.scheduler.service.AbstractQueueManager.processQueuedItemsWithinLock(AbstractQueueManager.java:338) ~[spotfire-common-services.jar:?]
    at com.spotfire.server.scheduler.service.AbstractQueueManager.internalProcessQueuedItems(AbstractQueueManager.java:311) ~[spotfire-common-services.jar:?]
    at com.spotfire.server.scheduler.service.AbstractQueueManager.lambda$processQueuedItems$2(AbstractQueueManager.java:288) ~[spotfire-common-services.jar:?]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_241]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_241]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_241]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[?:1.8.0_241]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_241]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_241]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_241]
Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to 123.234.345.456:9502 [/10.220.181.61] failed: connect timed out
    at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151) ~[httpclient.jar:4.5.10]
    at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:374) ~[httpclient.jar:4.5.10]
    at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient.jar:4.5.10]
    at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient.jar:4.5.10]
    at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient.jar:4.5.10]
    at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient.jar:4.5.10]
    at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient.jar:4.5.10]
    at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient.jar:4.5.10]
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient.jar:4.5.10]
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[httpclient.jar:4.5.10]
    at org.springframework.http.client.HttpComponentsClientHttpRequest.executeInternal(HttpComponentsClientHttpRequest.java:87) ~[spring-web.jar:5.2.3.RELEASE]
    at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48) ~[spring-web.jar:5.2.3.RELEASE]
    at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:53) ~[spring-web.jar:5.2.3.RELEASE]
    at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:742) ~[spring-web.jar:5.2.3.RELEASE]
    ... 18 more
Caused by: java.net.SocketTimeoutException: connect timed out
    at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method) ~[?:1.8.0_241]
    at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:85) ~[?:1.8.0_241]
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_241]
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_241]
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_241]
    at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172) ~[?:1.8.0_241]
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_241]
    at java.net.Socket.connect(Socket.java:606) ~[?:1.8.0_241]
    at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:368) ~[httpclient.jar:4.5.10]
    at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient.jar:4.5.10]
    at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:374) ~[httpclient.jar:4.5.10]
    at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient.jar:4.5.10]
    at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient.jar:4.5.10]
    at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient.jar:4.5.10]
    at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient.jar:4.5.10]
    at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient.jar:4.5.10]
    at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient.jar:4.5.10]
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient.jar:4.5.10]
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[httpclient.jar:4.5.10]
    at org.springframework.http.client.HttpComponentsClientHttpRequest.executeInternal(HttpComponentsClientHttpRequest.java:87) ~[spring-web.jar:5.2.3.RELEASE]
    at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48) ~[spring-web.jar:5.2.3.RELEASE]
    at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:53) ~[spring-web.jar:5.2.3.RELEASE]
    at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:742) ~[spring-web.jar:5.2.3.RELEASE]
    ... 18 more Verify if the TIBCO Spotfire Server and Node managers machines can communicate:

• Check if the TIBCO Spotfire Server can ping and Telnet to the Node Manager server on the ports 9501 or 9502. You can refer to the log snippet and verify the connectivity based on what port it is failing. For example, based on the above error message you can run the syntax "telnet 10.220.181.61 9502" from Spotfire Server machine and verify if you are able to connect. See below screenshot for your reference:

If it fails with "Connect failed" message, please verify if the port is open and that you are able to communicate:

•  Check if there is any firewall rules that is blocking the communication between Spotfire Server and Node Manager. 

• https://docs.tibco.com/pub/spotfire_server/10.3.0/doc/html/TIB_sfire_server_tsas_admin_help/GUID-1891A737-803F-4680-9F44-66D988F06614.html