Sep-25-2022, 10:58 PM
(This post was last modified: Sep-26-2022, 09:09 AM by Larz60+.
Edit Reason: Fixed closing python tag
)
Hi Team,
How to write SQL Table data to parquet file. and compress to gzip.
SQL Table data size is 60gb.
or from writing sql data to csv first and then writing to parquet.
Pandas is slow or Cant handle data of 60 gb I think.
below is my attempted code.
---------Attempt to parquet sql table-----------------
How to write SQL Table data to parquet file. and compress to gzip.
SQL Table data size is 60gb.
or from writing sql data to csv first and then writing to parquet.
Pandas is slow or Cant handle data of 60 gb I think.
below is my attempted code.
---------Attempt to parquet sql table-----------------
1 2 3 4 5 6 7 8 9 |
for chunk in pd.read_sql_table( 'employee' ,connection,chunksize = 1000 ): mylist.append(chunk) df = pd.concat(mylist) df.to_parquet( '{path}.gzip' , compression = 'gzip' , index = False ) def to_parquet(data, path): df = pd.DataFrame(data) df.to_parquet( '{path}.gzip' , compression = 'gzip' , index = False ) |