Welcome to python-forum.io

Results

Search Results

Post

Author

Forum

Replies

Views

Posted [asc]

Thread: pyspark parallel write operation not working
Post: RE: pyspark parallel write operation not working

There must be many people who are writing to the database from python , no one ever wanted to use more than one session to do this?

aliyesami

General Coding Help

1,684

Oct-16-2021, 05:18 PM

Thread: pyspark creating temp files in /tmp folder
Post: RE: pyspark creating temp files in /tmp folder

ok I found the solution , for those who do not know its just a matter of setting "spark.local.dir" in sparksession as below. I am surprised no one knew this on this forum . spark = SparkSession.bu...

aliyesami

General Coding Help

4,970

Oct-16-2021, 05:15 PM

Thread: pyspark creating temp files in /tmp folder
Post: pyspark creating temp files in /tmp folder

the spark submit job is creating large files in the temp folder when i run the dataload job , and failing when the /tmp folder gets full. can i divert this temp files creation to another folder ? if ...

aliyesami

General Coding Help

4,970

Oct-11-2021, 04:14 AM

Thread: pyspark parallel write operation not working
Post: pyspark parallel write operation not working

I want pyspark code to use parallel threads when connecting to the database when i am inserting into a table but its not. I have tried splitting the DF , also used numPartitions atribute in the writ...

aliyesami

General Coding Help

1,684

Oct-11-2021, 04:04 AM

Thread: problem writing dataframe to oracle
Post: RE: problem writing dataframe to oracle

to any moderator: I think the thing i am trying to do is very common and simple , i am facing issue because I am novice. So I am surprised why I am not getting response . Did i post this in a wrong...

aliyesami

General Coding Help

2,650

Sep-25-2021, 04:25 PM

Thread: problem writing dataframe to oracle
Post: RE: problem writing dataframe to oracle

thanks but the link you posted has no relation to my issue. as I stated I don't have any issue in connection or loading as long as I keep the header . I want to know how to load the data when the cs...

aliyesami

General Coding Help

2,650

Sep-25-2021, 01:52 PM

Thread: problem writing dataframe to oracle
Post: problem writing dataframe to oracle

i have a simple csv file as follows : Output:1,'sami' 2.'john', 3,'mary' 4,'peter'and i have a table test(id, name) the code is as follows minus DB connection details since connection is fine : ...

aliyesami

General Coding Help

2,650

Sep-25-2021, 02:01 AM

Results

User Panel Messages

Announcements