Date Posted:
Product: TIBCO Spotfire®
Product: TIBCO Spotfire®
Problem:
How to minimize the frequency of a Scheduled Updates job without compromising on fresh data
Solution:
When new data is added to your data source every 30 minutes (for example), and reloading all data takes more than 30 minutes, how frequently should you update the report in Scheduled Updates? Even if it is run every couple of hours, you are consuming a lot of machine resources to continually reload all data, and in a worst case scenario the data is 2 hours old. The ideal scenario is to maintain the older data which has not changed and instead only load the new data. Scheduled Updates currently does not directly support incremental updates, but this article explains a workaround solution.
High level Steps:
- Spotfire does not have an incremental refresh/reload feature. Every time you run a Scheduled Updates job, it will always fetch everything. A workaround is to split your data into two parts:
- Part 1: Everything before today.
- Part 2: Today's data.
- Run Scheduled Updates once a day to ensure that you always have all the data before today.
- Exclude Part 2 from the Scheduled Updates job, so it fetches fresh data every time. This way you avoid loading entire datasets multiple times in a day, and at the same time, you get the latest data every time you load the report.
Detailed Steps:
-
Create an information link with a filter on date, where date < today. Named for example IL1.
Example SQL for the information link should look similar to the one below:select * from table where date < today
- Create an information link with a filter on date, where date = today. Named for example IL2.
Example SQL for the information link should look similar to the one below:select * from table where date = today
-
Add a new data table from IL1.
-
Add rows to this data table from IL2.
-
Go to data table properties and exclude the second information link (IL2) from Schedule Updates.
-
Edit > Data Table Properties > Schedule Updates > Reload the following data for each User
Note: You can change the splitting (<today, today) depending on the total size of the data and how frequently you get new data.
Now you can add that report to Scheduled Updates so that it is cached, and only the data from IL2 will be updated when users access the report.
Now you can add that report to Scheduled Updates so that it is cached, and only the data from IL2 will be updated when users access the report.