Python Forum
pyspark creating temp files in /tmp folder
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
pyspark creating temp files in /tmp folder
#1
the spark submit job is creating large files in the temp folder when i run the dataload job , and failing when the /tmp folder gets full.
can i divert this temp files creation to another folder ? if yes how ?

thanks
Reply
#2
ok I found the solution , for those who do not know its just a matter of setting "spark.local.dir" in sparksession as below. I am surprised no one knew this on this forum .

spark = SparkSession.builder \
.master('local[*]') \
.config("spark.driver.memory", "25g") \
.config("spark.local.dir", "/u01/spark-temp") \
.appName('read_from_RS_write_to_oracle demo') \
.getOrCreate()
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  PySpark Equivalent Code cpatte7372 0 138 Jan-14-2022, 08:59 PM
Last Post: cpatte7372
  Compare filename with folder name and copy matching files into a particular folder shantanu97 2 310 Dec-18-2021, 09:32 PM
Last Post: Larz60+
  Pyspark - my code works but I want to make it better Kevin 1 329 Dec-01-2021, 05:04 AM
Last Post: Kevin
  pyspark parallel write operation not working aliyesami 1 401 Oct-16-2021, 05:18 PM
Last Post: aliyesami
  Help with storing temp data for each day then recording min/max in app. trthskr4 3 714 Sep-10-2021, 10:51 PM
Last Post: trthskr4
  How to save Matplot chart to temp file? Morkus 2 1,365 Jun-12-2021, 10:52 AM
Last Post: Morkus
  Move file from one folder to another folder with timestamp added end of file shantanu97 0 938 Mar-22-2021, 10:59 AM
Last Post: shantanu97
Photo Integration of apache spark and Kafka on eclipse pyspark aupres 1 1,568 Feb-27-2021, 08:38 AM
Last Post: Serafim
  KafkaUtils module not found on spark 3 pyspark aupres 2 2,393 Feb-17-2021, 09:40 AM
Last Post: Larz60+
  PySpark Coding Challenge cpatte7372 3 1,963 Feb-14-2021, 04:49 PM
Last Post: ndc85430

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020