Product: TIBCO Spotfire®
Spotfire Server hangs or responds slowly, with the error "com.spotfire.ws.dat.DatException: Too many jobs"
Description:
Error: "com.spotfire.ws.dat.DatException: Too many jobs".
Symptoms:
TIBCO Spotfire Server hangs or responds slowly while executing information links. The following warning message can be found in the logs:
-------------------------------------------------------
WARN 2015-06-03T06:47:20,647-0600 [brockrr, #10279] api.common.InformationModelServiceCommon: Server too busy to process job.
com.spotfire.ws.api.common.InformationModelWebServiceException: Server busy.
.....
.....
Caused by: com.spotfire.ws.dat.DatException: Too many jobs.
------------------------------------------------------
Cause:
This issue may be due to the max-jobs settings on the server. The maximum number of simultaneous information links running or waiting in line is limited by a max-jobs setting which is set to 40 by default. If all jobs are taken, when another user tries to run an information link they will see the "too many jobs" error. The max-job setting value needs to be increased. There is also another parameter called the thread-pool-size and you will have to increase that as well to maintain a balance between jobs and threads.
The thread-pool-size setting specifies how many threads will be available to handle the running information links. If there are more jobs than threads, the jobs will be waiting in line for an available thread. If you set max-jobs to a larger number than threads, all users may suffer, as it means that they will all have to compete for available threads, so that it takes longer to finish running an information link. If you increase the thread-pool-size too much, you could overload the server and receive an OutOfMemory Exception. The best way is to increase the value gradually until a balance is found.
When creating a data source in Information Designer, specify the maximum number of connections to the database. You should set the max-jobs and thread-pool-size settings to the same value as the total max number of connections across all data sources. If you set the max-jobs value larger than the max number of connections, users could see timeouts awaiting available connections to the database, as described in KB 000020991(
Error: "Timeout while waiting for database connection after 10 seconds)). Instructions to increase the max-job and max-thread values:
------------------------------------------------------------------------------------
1). Open Spotfire Server's Configuration Tool and export the current running configuration to file (Config Tool -> System Status tab -> Export configuration from database).
2). Search for the following section in the exported file.
<!-- Maximum number of concurrent jobs. -->
<max-jobs>40</max-jobs>
<!-- Maximum number of worker threads. -->
<thread-pool-size>40</thread-pool-size>
Increase both values. It is recommended to use the same value for both settings. It is also recommended to use a value equal to the total number of max-connections across all Data Sources.
3). Save the file and import it into database (Config Tool -> System Status tab -> Load configuration from file -> Save Configuration).
4). Restart the server.
-----------------------------------------------------------------------------------
Note:
This may lead to Java OutOfMemory Exception events, as increasing the number of threads will require more memory. Consider increasing the JVM size as described in KB 000019076 (Tomcat Crash or Hang - OutOfMemoryError: GC Overhead limit exceeded)