Oct-16-2021, 05:15 PM
ok I found the solution , for those who do not know its just a matter of setting "spark.local.dir" in sparksession as below. I am surprised no one knew this on this forum .
spark = SparkSession.builder \
.master('local[*]') \
.config("spark.driver.memory", "25g") \
.config("spark.local.dir", "/u01/spark-temp") \
.appName('read_from_RS_write_to_oracle demo') \
.getOrCreate()
spark = SparkSession.builder \
.master('local[*]') \
.config("spark.driver.memory", "25g") \
.config("spark.local.dir", "/u01/spark-temp") \
.appName('read_from_RS_write_to_oracle demo') \
.getOrCreate()