Python Forum
how to load large data into dataframe.
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
how to load large data into dataframe.
#1
Hi All,

I am working on 1 GB Data pulling from database .I want to bring this data into dataframe and do some analysis. Due to huge data ,python not able to execute it.
I read some notes and found chunk command to load data into chunks.
for chunk in
pd.read_sql(query_string ,conn_bmg,chunksize=1000000): this command is executing but how to send this data into data frame..

if I ran this df = pd.read_sql(query_string ,conn_bmg,chunksize=1000000)
df I am getting this message (<generator object SQLiteDatabase._query_iterator at 0x000001BA83B44480>)

please can any one help me on this .. correct my syntax..


thanks in advance
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Load data in Oracle muzokh 0 257 Mar-08-2024, 11:19 PM
Last Post: muzokh
  Filter data into new dataframe as main dataframe is being populated cubangt 8 1,007 Oct-23-2023, 12:43 AM
Last Post: cubangt
  Using pyodbc&pandas to load a Table data to df tester_V 3 827 Sep-09-2023, 08:55 PM
Last Post: tester_V
  Load multiple Jason data in one Data Frame vijays3 6 1,558 Aug-12-2022, 05:17 PM
Last Post: vijays3
  Seeing al the data in a dataframe or numpy.array Led_Zeppelin 1 1,150 Jul-11-2022, 08:54 PM
Last Post: Larz60+
  Need help formatting dataframe data before saving to CSV cubangt 16 5,888 Jul-01-2022, 12:54 PM
Last Post: cubangt
  Reading data to python: turn into list or dataframe hhchenfx 2 5,394 Jun-01-2021, 10:28 AM
Last Post: Larz60+
  Load the data from multiple source files to one table amy83 2 2,597 Apr-27-2021, 12:33 AM
Last Post: Pedroski55
  How to save json data in a dataframe shantanu97 1 2,165 Apr-15-2021, 02:44 PM
Last Post: klllmmm
  Pandas Extract data from two dataframe nio74maz 1 2,187 Dec-26-2020, 09:52 PM
Last Post: nio74maz

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020