+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: General Coding Help (https://python-forum.io/forum-8.html)
+--- Thread: pyspark creating temp files in /tmp folder (/thread-35230.html)
pyspark creating temp files in /tmp folder - aliyesami - Oct-11-2021
the spark submit job is creating large files in the temp folder when i run the dataload job , and failing when the /tmp folder gets full.
can i divert this temp files creation to another folder ? if yes how ?
thanks
RE: pyspark creating temp files in /tmp folder - aliyesami - Oct-16-2021
ok I found the solution , for those who do not know its just a matter of setting "spark.local.dir" in sparksession as below. I am surprised no one knew this on this forum .
spark = SparkSession.builder \
.master('local[*]') \
.config("spark.driver.memory", "25g") \
.config("spark.local.dir", "/u01/spark-temp") \
.appName('read_from_RS_write_to_oracle demo') \
.getOrCreate()