Spotfire: Python Service fails to start with error "ValueError('numpy.dtype size changed, may indicate binary incompatibility."

Summary:

Spotfire Python Service on a Spotfire Node Manager machine no longer starts successfully. The service was working normally until a service restart. No Spotfire application changes were made, but there were updates to Python custom packages and some new packages were also added.

The Python Service repeatedly fails during canary execution-engine initialization and is restarted by the Node Manager. The Python Service log is reporting this error:

# A fatal error has been detected by the Java Runtime Environment:
#
#  EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x00007fff407e2bbb, pid=8600, tid=13320
#
# JRE version: OpenJDK Runtime Environment Temurin-17.0.8+7 (17.0.8+7) (build 17.0.8+7)
# Java VM: OpenJDK 64-Bit Server VM Temurin-17.0.8+7 (17.0.8+7, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, windows-amd64)
# Problematic frame:
# C  [pythonJava.dll+0x2bbb]
#
# CreateCoredumpOnCrash turned off, no core file dumped
#
# An error report file with more information is saved as:
# C:\Users\spotfire\AppData\Local\Temp\work\pythonsrv-62003\hs_err_pid8600.log
#
# If you would like to submit a bug report, please visit:
#   https://github.com/adoptium/adoptium-support/issues
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
>>> import os
>>> import traceback
>>> os.chdir(work_dir)
>>> from spotfire.data_function import *
There was a problem evaluating the code object.
Error: 
  Error Type: <class 'ValueError'>
  Error Value: ValueError('numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject')
  Error Traceback: <traceback object at 0x000002F1E78D9640>
>>> import threading

Resolution:

Based on the error "numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject", the most likely root cause of the issue is a compatibility issue between NumPy and one of the compiled Python packages used during Spotfire Python Service initialization. 
 
In Spotfire 14.x, this error is commonly associated with a NumPy 2.x version being loaded alongside packages compiled against NumPy 1.x. The next step is a review of all custom and updated Python packages that are installed in the non-working environment. This can be done as follows:

 - Run command .\python.exe -m pip list --user from the Python Service's installation directory.
 - Check the creation and last-modified dates of the user-level `site-packages` directory and confirm whether any Python packages were installed or updated recently.
 - Run the following command from the Python Service's installation directory to verify whether the bundled packages load successfully when user-level packages are excluded:

.\python.exe -s -c "import numpy; print(numpy.__version__); print(numpy.__file__); from spotfire.data_function import *; print('Import successful')"

The `-s` option temporarily disables user-level site-packages for this execution without modifying the Python environment. If this test succeeds, we can proceed with isolating the conflicting user-level packages. One option is to rename the user-level `site-packages` directory rather than deleting it, provided no other applications depend on those packages.

For example, in Spotfire 14.x, if a failing Python Service node is loading a NumPy 2.4.6 from the service account's user-level directory (C:\Users\svc-spotfire\AppData\Roaming\Python\Python311\site-packages), this is very likely the cause of the reported `numpy.dtype size changed` binary incompatibility error, as the user-level NumPy installation is taking precedence over the bundled version of this package that comes with Spotfire Python Service.