Python Forum
Write sql data or CSV Data into parquet file
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Write sql data or CSV Data into parquet file
#1
Hi Team,

How to write SQL Table data to parquet file. and compress to gzip.
SQL Table data size is 60gb.

or from writing sql data to csv first and then writing to parquet.

Pandas is slow or Cant handle data of 60 gb I think.
below is my attempted code.

---------Attempt to parquet sql table-----------------
for chunk in pd.read_sql_table('employee',connection,chunksize=1000):
	mylist.append(chunk)
df = pd.concat(mylist)
df.to_parquet('{path}.gzip', compression='gzip', index=False)


def to_parquet(data, path):
    df = pd.DataFrame(data)
    df.to_parquet('{path}.gzip', compression='gzip', index=False)
Reply


Messages In This Thread
Write sql data or CSV Data into parquet file - by mg24 - Sep-25-2022, 10:58 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
  What does .flush do? How can I change this to write to the file? Pedroski55 3 268 Apr-22-2024, 01:15 PM
Last Post: snippsat
  encrypt data in json file help jacksfrustration 1 264 Mar-28-2024, 05:16 PM
Last Post: deanhystad
  Help with to check an Input list data with a data read from an external source sacharyya 3 437 Mar-09-2024, 12:33 PM
Last Post: Pedroski55
  Last record in file doesn't write to newline gonksoup 3 466 Jan-22-2024, 12:56 PM
Last Post: deanhystad
  write to csv file problem jacksfrustration 11 1,588 Nov-09-2023, 01:56 PM
Last Post: deanhystad
  python Read each xlsx file and write it into csv with pipe delimiter mg24 4 1,517 Nov-09-2023, 10:56 AM
Last Post: mg24
  Input network device connection info from data file edroche3rd 6 1,090 Oct-12-2023, 02:18 AM
Last Post: edroche3rd
  Convert File to Data URL michaelnicol 3 1,228 Jul-08-2023, 11:35 AM
Last Post: DeaD_EyE
  How do I read and write a binary file in Python? blackears 6 6,854 Jun-06-2023, 06:37 PM
Last Post: rajeshgk
Video doing data treatment on a file import-parsing a variable EmBeck87 15 2,971 Apr-17-2023, 06:54 PM
Last Post: EmBeck87

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020