Initializing the pyspark on Jupyter Python Notebooks in Team Studio fails with an error

Created
October 08, 2021 23:18

Date Posted: 2019-10-17
Product: TIBCO Spotfire®

Problem:

Solution:

Initializing the pyspark on Jupyter Python Notebooks in Team Studio fails with the following error:

===============================================
An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext: java.lang.IllegalStateException: Spark context stopped while waiting for backend at org.apache.spark.scheduler.TaskSchedulerImpl.waitBackendReady(TaskSchedulerImpl.scala:818)
================================================ Make sure that the java version on the Cloudera datasource is greater than 1.7. By default, the Cloudera datasource is shipped with Java 1.7 so we have to manually update the java version to 1.8 or higher.

NOTE: TIBCO Data Science Team Studio supports Oracle jdk and doesn't support Open jdk yet.