Python Forum

the spark submit job is creating large files in the temp folder when i run the dataload job , and failing when the /tmp folder gets full.
can i divert this temp files creation to another folder ? if yes how ?

thanks

ok I found the solution , for those who do not know its just a matter of setting "spark.local.dir" in sparksession as below. I am surprised no one knew this on this forum .

spark = SparkSession.builder \
.master('local[*]') \
.config("spark.driver.memory", "25g") \
.config("spark.local.dir", "/u01/spark-temp") \
.appName('read_from_RS_write_to_oracle demo') \
.getOrCreate()