Python Forum
Best way to append data frame?
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Best way to append data frame?
#1
I read by accident, somewhere online, recently that it is highly unsought to append/insert a data frame either row by row, or cell by cell (for iloc). I cannot recall which one exactly, but to the point; What is the best way to append data into a data frame? Here is the snippit I have. I am doing my best to work with PEP8 style guide while making changes at the same time so feel free to mention discrepancies in the style. I have a total of 12 columns and I feel that appending each line singly is not good to do. It looks too separated (for lack of better words to explain it).

                    # Sets the quantity #
                    InvoiceDataFrame.loc[[len(InvoiceDataFrame) - 1],
                                         InvoiceDataFrame.columns[3]]="1"

                    # Sets the class #
                    InvoiceDataFrame.loc[[len(InvoiceDataFrame) - 1],
                                         InvoiceDataFrame.columns[4]]=ClassIndex[LoadedSales.iloc[inx, 1]]

                    # Sets the Job name #
                    InvoiceDataFrame.loc[[len(InvoiceDataFrame) - 1],
                                         InvoiceDataFrame.columns[5]]=HA

                    # Sets the Terms #
                    InvoiceDataFrame.loc[[len(InvoiceDataFrame) - 1],
                                         InvoiceDataFrame.columns[6]]="Redacted"
Note: With regards to PEP8 Style Guide. I haven't taken time yet to change my variable names to match the style guide.
Reply
#2
(Oct-04-2018, 05:11 PM)WuchaDoin Wrote: I read by accident, somewhere online, recently that it is highly unsought to append/insert a data frame either row by row, or cell by cell (for iloc). I cannot recall which one exactly, but to the point; What is the best way to append data into a data frame?

The best way is to create DataFrame from either a list of lists or lists of dictionaries or dictionary of lists(see Pandas cookbook).

You may also read data from either CSV or Excel file - or from SQL DB. You may load JSON structure. See pandas IO methods But you never build DataFrame row by row. If you want some data processing to create a new data - adding columns is OK.

Let me give you some simple demonstration
Output:
In [17]: df = pd.DataFrame([{'a': 1, 'b':2, 'c': 3}, ...: {'a': 4, 'b': 3, 'c': 10}]) ...: In [18]: df['d'] = df.apply(lambda r: r['a'] ** 3 + r['b'] ** 2 + r['c'], axis=1) In [19]: df Out[19]: a b c d 0 1 2 3 8 1 4 3 10 83 In [20]: df.a**2 + df.b**2 Out[20]: 0 5 1 25 dtype: int64
You may bulk change cell values too - in this example, I replace all odd values with zeroes
Output:
In [23]: df[df % 2 == 1] = 0 In [24]: df Out[24]: a b c d 0 0 2 0 8 1 4 0 10 0
But processing row by row defies the purpose of pandas and is inefficient.
Test everything in a Python shell (iPython, Azure Notebook, etc.)
  • Someone gave you an advice you liked? Test it - maybe the advice was actually bad.
  • Someone gave you an advice you think is bad? Test it before arguing - maybe it was good.
  • You posted a claim that something you did not test works? Be prepared to eat your hat.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Grouping in pandas/multi-index data frame Aleqsie 3 655 Jan-06-2024, 03:55 PM
Last Post: deanhystad
  Filtering Data Frame, with another value NewBiee 9 1,385 Aug-21-2023, 10:53 AM
Last Post: NewBiee
  Exporting data frame to excel dyerlee91 0 1,623 Oct-05-2021, 11:34 AM
Last Post: dyerlee91
  Pandas Data frame column condition check based on length of the value aditi06 1 2,683 Jul-28-2021, 11:08 AM
Last Post: jefsummers
  Adding a new column to a Panda Data Frame rsherry8 2 2,119 Jun-06-2021, 06:49 PM
Last Post: jefsummers
  grouped data frame glitter 0 1,592 Feb-02-2021, 11:22 AM
Last Post: glitter
  how to filter data frame dynamically with the columns psahay 0 2,397 Aug-24-2020, 01:10 PM
Last Post: psahay
  Dropping Rows From A Data Frame Based On A Variable JoeDainton123 1 2,211 Aug-03-2020, 02:05 AM
Last Post: scidam
  How to shift data frame rows of specified column Mekala 0 1,890 Jul-21-2020, 02:42 PM
Last Post: Mekala
  HELP- DATA FRAME INTO TIME SERIES- BASIC bntayfur 0 1,747 Jul-11-2020, 09:04 PM
Last Post: bntayfur

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020