Python Forum
build pandas dataframe from a for loop
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
build pandas dataframe from a for loop
#1
Hello,

this is my first post about Python, and also this is my first post in English. So... thank you for your patience :)... And sorry for my low english level, I'm practicing to improve.

I want to build a pandas Dataframe but the rows info are coming to me one by one (in a for loop), in form of a dictionary (or json).

In each iteration I receive a dictionary where the keys refer to the columns, and the values are the rows values. For example:

1st Iteration I receive:
d_val = {'key1': 1.1, 'key2':2.2, 'key3':3.3}

And I want to build a dataframe (called df), where column names will be d_val.keys(), and the first row values will be d_val.values()

2nd Iteration I receive:
{'key1': 4.4, 'key2':5.5, 'key3': 6.6}

And I want to append the values to a new row in the datframe.

Example outcome wanted:

key1---key2----key3
1.1-----2.2------3.3
4.4-----5.5------6.6

And each new iteration, will make a new row append. I will be sure that in each iteration the dict keys are the same.

I try to create a empty dataframe or a dataframe with only de columns values, and then use the DataFrame.append method... but never works. I'm lost, because I think that It will not be so difficult.

I hope that I have explained well.

Thanks
Reply
#2
Example below should help you...

import pandas as pd

df = pd.DataFrame({'key1': [], 'key2': [], 'key3': []})
for i in range(10):
    df = df.append({'key1': i, 'key2': i*2, 'key3': i**3}, ignore_index=True)
Reply
#3
Hello scidam,

I've adapted it to my program and it works! Thank you.

But in my first message I have simplified the problem a little... ;) I was afraid of not explaining myself well and that's why I simplified it...

Really, I need to do this, but I don't know how many items will have the dict a priori. Then, I don't know if I will have 'key1', 'key2'... 'keyn'

In each iteration, I will receive a message with the information. When I receive the first message (in the first iteration), then I analize the message and I see the dict/json items. Before that, I don't know how many columns will have the dataframe.

Is there a more generic way?

Thanks
Reply
#4
Pandas is quite smart. Don't worry about new keys, you can even start with an empty data frame, e.g.

import pandas as pd
import random
df = pd.DataFrame()
for j in range(10):
    df = df.append({'key_%s'%random.choice('abcde'): j}, ignore_index=True)
Output:
>>> df key_d key_b key_e key_c 0 0.0 NaN NaN NaN 1 1.0 NaN NaN NaN 2 2.0 NaN NaN NaN 3 NaN 3.0 NaN NaN 4 NaN NaN 4.0 NaN 5 5.0 NaN NaN NaN 6 6.0 NaN NaN NaN 7 NaN NaN 7.0 NaN 8 NaN NaN NaN 8.0 9 NaN NaN NaN 9.0
Reply
#5
Hello scidam,

great! Thank you!
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  HTML Decoder pandas dataframe column mbrown009 3 962 Sep-29-2023, 05:56 PM
Last Post: deanhystad
  Use pandas to obtain cartesian product between a dataframe of int and equations? haihal 0 1,091 Jan-06-2023, 10:53 PM
Last Post: haihal
  Pandas Dataframe Filtering based on rows mvdlm 0 1,396 Apr-02-2022, 06:39 PM
Last Post: mvdlm
  Pandas dataframe: calculate metrics by year mcva 1 2,269 Mar-02-2022, 08:22 AM
Last Post: mcva
  Pandas dataframe comparing anto5 0 1,243 Jan-30-2022, 10:21 AM
Last Post: anto5
  PANDAS: DataFrame | Replace and others questions moduki1 2 1,758 Jan-10-2022, 07:19 PM
Last Post: moduki1
  PANDAS: DataFrame | Saving the wrong value moduki1 0 1,526 Jan-10-2022, 04:42 PM
Last Post: moduki1
  update values in one dataframe based on another dataframe - Pandas iliasb 2 9,100 Aug-14-2021, 12:38 PM
Last Post: jefsummers
  empty row in pandas dataframe rwahdan 3 2,419 Jun-22-2021, 07:57 PM
Last Post: snippsat
Question Pandas - Creating additional column in dataframe from another column Azureaus 2 2,915 Jan-11-2021, 09:53 PM
Last Post: Azureaus

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020