How to enable logging of long lived threads and data engine queries

How to enable logging of long lived threads and data engine queries

This has helped to find memory leaks related to Python scripts and provide information regarding memory related issues.

Items will be logged if they have been alive for more than 50 seconds (with settings example below). For most jobs, this is never the case. The logging will contain the call stack for the creation to show what kind if job it is. The percentage value is added because the logging uses some resources, so it is possible to decide that only 10% (by random) of the items should be logged, resulting in much less of a performance penalty than 100%.

LogLiveQueriesOnlyOnce and LogLiveWorkItemsOnlyOnce can be used to avoid logging the same item over and over again if it lives for very long time.

IMPORTANT: The setting should be completely disabled by default and nothing should be visible in the config files. This is important because of the performance penalty caused by the additional logging. Only implement this change if instructed to do so by TIBCO Support for investigation purposes and revert the changes when the testing is complete. 1). Open a command-line interface on the TIBCO Spotfire Server and export the service configuration by using the export-service-config command. Specify the Web Player capability and the deployment area:

   export-service-config --capability=WEB_PLAYER --deployment-area=Production
  
By default, the output files are located in the <install dir>\tomcat\bin\config\root directory if you do not specify a source directory.

2). Create a back up of the Spotfire.Dxp.Worker.Host.exe.config file.

3). Open the Spotfire.Dxp.Worker.Host.exe.config file in a text or XML editor.

4). Add the following in the "</Spotfire.Dxp.Data.Properties.Settings>" and <Spotfire.Dxp.Framework.Properties.Settings> sections. If the Settings section is not present, add it.

    <Spotfire.Dxp.Data.Cxx.Properties.Settings>
      <setting name="LogLiveQueries" serializeAs="String">
        <value>True</value>
      </setting>      
      <setting name="LogLiveQueriesPercentage" serializeAs="String">
        <value>100</value>
      </setting>
      <setting name="LogLiveQueriesMinAgeSecondsToLog" serializeAs="String">
        <value>50</value>
      </setting>
      <setting name="LogLiveQueriesOnlyOnce" serializeAs="String">
        <value>True</value>
      </setting>
    </Spotfire.Dxp.Data.Cxx.Properties.Settings>

    <Spotfire.Dxp.Framework.Properties.Settings>
      <setting name="LogLiveWorkItems" serializeAs="String">
        <value>True</value>
      </setting>
      <setting name="LogLiveWorkItemsPercentage" serializeAs="String">
        <value>100</value>
      </setting>
      <setting name="LogLiveWorkItemsMinAgeSecondsToLog" serializeAs="String">
        <value>50</value>
      </setting>
      <setting name="LogLiveWorkItemsOnlyOnce" serializeAs="String">
        <value>True</value>
      </setting>
    </Spotfire.Dxp.Framework.Properties.Settings>

Verify that the following line is present in <sectionGroup>:
<section name="Spotfire.Dxp.Data.Cxx.Properties.Settings" type="System.Configuration.ClientSettingsSection, System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" requirePermission="false" />

5). Save.

6). Create a back up of the log4net.config file.

7). Open the log4net.config in a text or XML editor.

8). Add a log4net appender for the WebLiveJobsLog and add following. This can be done at the bottom file prior to </log4net> .

    <appender name="WebLiveJobsLog" type="Spotfire.Dxp.Internal.Diagnostics.WebLoggerFileAppender">
      <PreserveLogFileNameExtension value="true" />
      <file type="log4net.Util.PatternString" value="..\..\logs\WebLiveJobsLog%property{serviceIdWithPeriod}.txt"/>
      <appendToFile value="true"/>
      <rollingStyle value="Size" />
      <maxSizeRollBackups value="4" />
      <maximumFileSize value="500MB" />
      <staticLogFileName value="false" />
      <layout type="log4net.Layout.PatternLayout">
        <header value="Level;TimeStamp1;TimeStamp2;InstanceId;AgeSeconds;Expression;Callstack " />
        <conversionPattern value="%message%newline"/>
      </layout>
    </appender>
    <logger name="WebLogger.WebLiveJobsLog" additivity="false">
      <appender-ref ref="WebLiveJobsLog"/>
      <level value="DEBUG" />
    </logger>

9). Save.

10). In the command-line interface, import the configuration file back into Spotfire Server and name the configuration by using the import-service-config command.

   import-service-config --config-name=WebPlayerConfiguration

11). Assign the created configuration to your service.

Option 1:  In the command-line interface, assign the created Spotfire Web Player configuration to the Spotfire Server to make it possible to use for the service:
set-service-config --service-id=value --config-name=WebPlayerConfiguration

Option 2: In the Spotfire Server web interface:
- Nodes & Services > Nodes > Select your service on the left > Click Edit
- In the "Edit service" dialog, select your imported configuration in the "Configuration" dropdown
- Click Update (Note: This will trigger a restart of the instances).   Doc: Manually editing the service configuration files

• https://docs.tibco.com/pub/spotfire_server/7.11.1/doc/html/TIB_sfire_server_tsas_admin_help/GUID-958A3AB5-7243-4837-9AEA-78AA274A6F7E.html