Python Forum
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Exporting a huge dataFrame
#1
Hey guys I am trying to export a pretty big data frame. I am using this code

df = pd.DataFrame(historicalData)
df.to_csv(r'dataframe.csv')
Now my dataframe has 5 million rows in it.

When I open the CSV file, it says "not loaded completely" and only 1 million rows are loaded.

If I use this code

df = pd.DataFrame(historicalData)
df.to_excel(r'dataframe.xlsx', header=True)
I get the ValueError : This sheet is too large! Your sheet size is: 5218863, 8 Max sheet size is: 1048576, 16384

Anyone know how to solve this problem?
Reply


Messages In This Thread
Exporting a huge dataFrame - by stylingpat - Mar-21-2021, 02:34 PM
RE: Exporting a huge dataFrame - by perfringo - Mar-21-2021, 04:10 PM
RE: Exporting a huge dataFrame - by stylingpat - Mar-21-2021, 08:22 PM
RE: Exporting a huge dataFrame - by perfringo - Mar-21-2021, 11:17 PM
RE: Exporting a huge dataFrame - by supuflounder - Mar-22-2021, 09:57 AM
RE: Exporting a huge dataFrame - by stylingpat - Mar-23-2021, 12:13 AM

Possibly Related Threads…
Thread Author Replies Views Last Post
  Exporting Stock Fundamental Data to a CSV file with yahoo_fin DustinKlent 2 4,738 Aug-01-2022, 06:08 PM
Last Post: paulyan
  Exporting dataframes to excel without loosing datetime format Rafa 0 1,240 Oct-27-2021, 10:42 AM
Last Post: Rafa
  exporting all lines YazeedbnMohmmed 2 2,131 Feb-24-2021, 03:29 AM
Last Post: YazeedbnMohmmed
  Pip prints huge error when installing p5 hayden2s 1 2,255 Aug-08-2020, 02:30 PM
Last Post: snippsat
  Huge CSV Reading and Sorting for comparison akshaynimkar 3 2,489 Aug-04-2020, 11:20 AM
Last Post: Larz60+
  Python code for exporting table using Selenium gj31980 4 3,009 Aug-04-2020, 01:29 AM
Last Post: gj31980
  Exporting data from python into excel Zankawah 5 3,406 Jun-02-2020, 03:17 AM
Last Post: buran
  How to scan huge files and make it in chunks ampai 2 2,608 May-28-2020, 08:20 PM
Last Post: micseydel
  convert huge xml to csv using python srikanta_p 2 2,041 Feb-08-2020, 07:16 PM
Last Post: srikanta_p
  Exporting list with dictionary to Excel veromi22 0 3,051 Oct-15-2019, 12:54 AM
Last Post: veromi22

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020