pyspark.SparkContext.setJobDescription¶
-
SparkContext.
setJobDescription
(value)[source]¶ Set a human readable description of the current job.
Notes
Currently, setting a job description (set to local properties) with multiple threads does not properly work. Internally threads on PVM and JVM are not synced, and JVM thread can be reused for multiple threads on PVM, which fails to isolate local properties for each thread on PVM.
To avoid this, enable the pinned thread mode by setting
PYSPARK_PIN_THREAD
environment variable totrue
and usespyspark.InheritableThread
.