Python Forum
pyspark creating temp files in /tmp folder
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
pyspark creating temp files in /tmp folder
#1
the spark submit job is creating large files in the temp folder when i run the dataload job , and failing when the /tmp folder gets full.
can i divert this temp files creation to another folder ? if yes how ?

thanks
Reply
#2
ok I found the solution , for those who do not know its just a matter of setting "spark.local.dir" in sparksession as below. I am surprised no one knew this on this forum .

spark = SparkSession.builder \
.master('local[*]') \
.config("spark.driver.memory", "25g") \
.config("spark.local.dir", "/u01/spark-temp") \
.appName('read_from_RS_write_to_oracle demo') \
.getOrCreate()
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Help with creating folder and "virtual environment" AudunNilsen 1 171 Mar-21-2024, 04:41 AM
Last Post: deanhystad
  Compare folder A and subfolder B and display files that are in folder A but not in su Melcu54 3 466 Jan-05-2024, 05:16 PM
Last Post: Pedroski55
  Rename files in a folder named using windows explorer hitoxman 3 693 Aug-02-2023, 04:08 PM
Last Post: deanhystad
  Rename all files in a folder hitoxman 9 1,384 Jun-30-2023, 12:19 AM
Last Post: Pedroski55
  PySpark Coding Challenge cpatte7372 4 5,961 Jun-25-2023, 12:56 PM
Last Post: prajwal_0078
  Pyspark dataframe siddhi1919 3 1,183 Apr-25-2023, 12:39 PM
Last Post: snippsat
  pyspark help lokesh 0 736 Jan-03-2023, 04:34 PM
Last Post: lokesh
  How to loop through all excel files and sheets in folder jadelola 1 4,331 Dec-01-2022, 06:12 PM
Last Post: deanhystad
  Creating csv files from Excel file azizrasul 40 5,326 Nov-03-2022, 08:33 PM
Last Post: azizrasul
  python gzip all files from a folder mg24 3 3,812 Oct-28-2022, 03:59 PM
Last Post: mg24

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020