Python Forum
drop duplicate values based on a criteria
Thread Rating:
  • 1 Vote(s) - 3 Average
  • 1
  • 2
  • 3
  • 4
  • 5
drop duplicate values based on a criteria
#1
I have a pandas dataframe that has duplicate data. Similar to below table. I want to drop duplicates keeping only the rows with value close to zero in column2
table1
Cloumn1    column2
1100            -27
1100               0
1100              10
1300               -2
1300               12

Resultant table looks like
Cloumn1    column2
1100             0
1300             -2

Hope someone can help on this


import pandas as pd

table = pd.DataFrame(data = {'Col1':[1100,1100,1100,1300,1300], 'Col2':[-27,0,10,-2,12]})
Reply
#2
It can be done similarly like you could do it with sql with window function (row_number over ordered group and take first).

table['sortcol'] = np.abs(table.Col2)
table.sort('sortcol').groupby('Col1').first()
followed by dropping of sortcol column
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  update values in one dataframe based on another dataframe - Pandas iliasb 2 9,247 Aug-14-2021, 12:38 PM
Last Post: jefsummers
  Counting Criteria in Pandas Question Koenig 1 2,157 Sep-30-2019, 05:16 AM
Last Post: perfringo
  how to get x values based on y-axis values from curvefit function python_newbie09 1 3,270 Sep-19-2019, 02:09 AM
Last Post: scidam
  Splitting values in column in a pandas dataframe based on a condition hey_arnold 1 4,168 Jul-24-2018, 02:18 PM
Last Post: hey_arnold
  Get max values based on unique values in another list - python Antonio 8 8,415 Jun-12-2018, 07:49 PM
Last Post: Mekire
  Select rows based on a criteria klllmmm 8 7,627 May-11-2017, 06:58 PM
Last Post: klllmmm
  Insert values into a column in the same table based on a criteria klllmmm 3 4,209 Apr-13-2017, 10:10 AM
Last Post: zivoni
  Match two data sets based on item values klllmmm 7 6,429 Mar-29-2017, 02:33 PM
Last Post: zivoni

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020