Spotfire: How Disk Caching works in TIBCO Spotfire Web Player

This article provides an overview of how disk caching works in the TIBCO Spotfire Web Player.

Details:

This article provides an overview of how disk caching works in the TIBCO Spotfire Web Player. Disk caching can be enabled in addition to, and only in conjunction with, the use of Scheduled Updates

Note: disk caching is currently only available for Web Players running on Windows - not on Linux.

The main benefit of using disk caching on top of the scheduled update cache is to speed the process of re-caching an analysis file in Scheduled Updates after a Web Player restarts. Disk caching acts as a recovery cache (a backup of what is cached by Scheduled Updates in RAM) when TIBCO Spotfire Web player restarts and the schedule window is active. The disk cache is retained until expiration or until the schedule window ends.

For example, say you have set a schedule for an analysis file from 10am - 7pm and disk caching was enabled (with the cache file created on disk at the 10am load time). If the Web Player restarts at 2pm, when the Web Player comes back online shortly after 2pm, the schedule window of the analysis file is still considered active (as it will end at 7pm) and the Web Player will quickly load the analysis from the disk cache back into RAM instead of loading the data from source again.

In Spotfire.Dxp.Worker.Web.config, there is a setting for scheduled updates cache which is used to enable disk caching. See "cacheSettings" here:

<scheduledUpdates concurrentUpdates="2" updateIntervalSeconds="60">         
<forcedUpdate enabled="true" maximumRejectedUpdates="2"/>         
<cacheSettings enabled="false" path="" maxDiskSizeMb="0" maxAgeMinutes="1440"/> 
</scheduledUpdates>

For more details, check out this document on enabling disk caching

By default, "Allow caching" option for disk caching is checked for all analysis while creating rules (Create Rule > File > Additional Properties at extreme bottom left as shown in screenshot).

Specific analysis can be excluded from disk caching by unchecking "Allow caching" option. See Disallowing cached and precomputed data in individual scheduled update files for more details.

Settings from Spotfire.Dxp.Worker.Web.config file:

enabled: By default, disk caching is disabled. It can be enabled by setting this to true

path: Specify the path on disk where data is to be stored.
If you have multiple Node Manager machines and want to share disk cache across all Web Players, you can provide the path for shared file system. If only one Node Manager is used, local file system path of that Node Manager can be provided

maxDiskSizeMb: Specify the maximum disk space used for the cached data. Set this to “0” (zero) to cache data without an upper limit.
When the max disk size is reached, cache entries will be deleted until it is below the limit again, starting with the entries that have not been accessed for a while.

maxAgeMinutes: Specify how long a cache entry should be kept on disk if it has not been reloaded by scheduled updates.
If maxAgeMinutes is reached, disk cache will be considered stale.
The way disk cache works is that scheduled updates calculates a "normalized time stamp" based on the schedules. The cache keeps entries alive
maxAgeMinutes number of minutes from that normalized time stamp.
Normalized time stamps works as follows. Let's say you have a schedule that reloads data every two hours, so 12.00, 14.00, 16.00, etc. If for some reason the analysis is loaded at 15.35 (e.g., because the Web Player was restarted) the normalized time stamp is 14.00 because that is the most recent time data would have been loaded under normal circumstances. Since you saved data to disk at 14.00, you can use that data instead of going to the external source. Note that if the time was 16.05 instead and the data on disk was from 14.00, you would not use that data since the normalized time stamps do not match. The schedule says the data is too old.
When schedule updates save data for a new time stamp (e.g., the 16.00 data), it removes older versions of the cached data so manual intervention is not needed

Account Access:
When using multiple Node Managers to access disk cache, Node Manager machines or accounts running Node Manager machine and scheduled update user account should have access to shared File System.

Synchronization:
When using multiple Web Players (multiple Node Managers) to refresh same cache in scheduled update, say for Node Managers in different time zones, then it uses lock files to synchronize access, so if two Web Players reload the same analysis at the same time, one of them will perform the update and the second will wait for the cache entry to be available.

Important Notes:

• Disk caching is not used outside of the scheduled hours or while a scheduled update is in progress (as that disk cache will be considered as stale/old)
• Disk caching does not decrease RAM requirement for Scheduled updates.

Daily Schedule:
If you want users to have cached copy of analysis even when schedule update is executing during its normal schedule, you can setup 24 hours schedule so analysis will never go out of cache. 

For example, Daily Update Schedule -  Repeat: Monday - Sunday, Start: 10:00, End: 10:00, Check for updates every: 24 hours


If you have enabled schedule as mentioned above and disk caching is enabled, then Disk cache will become stale at 10:00 everyday since Spotfire knows it needs to get data from 10:00 specifically. So if the Web player restarts at 10:30 (and the update is not complete yet) then the disk cache will not be used since data for the last scheduled update is stale i.e. the priority is data accuracy, not data load speed in this situation. Disk cache is considered stale when the cache age time specified in Spotfire.Dxp.Worker.Web.config file is met or a schedule start time is met.

Weekly Schedule:
You can setup weekly cache as shown in this KB article: How to create a weekly Scheduled Updates file rule which refreshes the analysis file only once a week

Whenever the Web Player restarts, all schedules set to refresh automatically will trigger refresh (if analysis takes 2 hours to refresh, the cached copy would not be available for users during that time unless disk cache is enabled). For weekly schedule, if you enable disk cache, you will need to increase cache age time as well. The default cache age time is 1440 minutes (24 hours).