Python Forum
removing duplicate numbers from a list
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
removing duplicate numbers from a list
#1
hi, there! i am new to python and i was trying to remove duplicate numbers in a list using for loop. but the code doesn't quite work

numbers = [3, 3, 20, 6, 10, 6, 5, 5, 7, 10, 7, 19, 19, 20]
for x in numbers:
    if numbers.count(x) > 1:
        numbers.remove(x)
print(numbers)
it doesn't remove all the duplicates.

this is what i get when i run it.

Output:
[3, 6, 6, 5, 10, 7, 19, 20]
Reply
#2
>>> numbers = [3, 3, 20, 6, 10, 6, 5, 5, 7, 10, 7, 19, 19, 20]
>>> list(set(numbers))
[3, 5, 6, 7, 10, 19, 20]
or

>>> from collections import OrderedDict
>>> numbers = [3, 3, 20, 6, 10, 6, 5, 5, 7, 10, 7, 19, 19, 20]
>>> list(OrderedDict.fromkeys(numbers))
[3, 20, 6, 10, 5, 7, 19]
Recommended Tutorials:
Reply
#3
@calonia Modifying a list whilst iterating over it is a very bad idea.
The 2nd example from @metulburr is perfect as it keeps the order of the items in the original list.
If that doesn´t matter his 1st example is faster.
In addition your runtime is O(len(numbers)*len(numbers)) which is bad.
numbers.count(x) runs over the full list and counts each number.
Try to do this in O(len(numbers) which is possible. (not using above examples) :-)
Reply
#4
Also from 3.6 and guaranteed in 3.7 so are dictionaries ordered.
This mean that can drop import of OrderedDict,and just use dict().
>>> numbers = [3, 3, 20, 6, 10, 6, 5, 5, 7, 10, 7, 19, 19, 20]
>>> list(dict.fromkeys(numbers))
[3, 20, 6, 10, 5, 7, 19]
Reply
#5
(Jun-14-2019, 04:10 PM)ThomasL Wrote: @calonia Modifying a list whilst iterating over it is a very bad idea.
The 2nd example from @metulburr is perfect as it keeps the order of the items in the original list.
If that doesn´t matter his 1st example is faster.
In addition your runtime is O(len(numbers)*len(numbers)) which is bad.
numbers.count(x) runs over the full list and counts each number.
Try to do this in O(len(numbers) which is possible. (not using above examples) :-)

thanks for the reply. i really appreciate your help. but i want to know what is wrong with this code. it doesn't seem to work with some of the numbers!
Reply
#6
Have a read here
Reply
#7
(Jun-14-2019, 05:13 PM)ThomasL Wrote: Have a read here

thank you very much. that was so helpful. i now get it. modifying a list while iterating through it(adding or removing items) leads to skipping indexes.
Reply
#8
Printing is 'cheapest' debugging tool. If one adds two print statements everything is quite self-explanatory:

>>> numbers = [3, 3, 20, 6, 10, 6, 5, 5, 7, 10, 7, 19, 19, 20]
>>> for x in numbers:
...     print(f'x is {x}')
...     if numbers.count(x) > 1:
...         numbers.remove(x)
...         print(numbers)
... 
x is 3                                         # first element in numbers
[3, 20, 6, 10, 6, 5, 5, 7, 10, 7, 19, 19, 20]
x is 20                                        # second element in numbers
[3, 6, 10, 6, 5, 5, 7, 10, 7, 19, 19, 20]
x is 10                                        # third element in numbers
[3, 6, 6, 5, 5, 7, 10, 7, 19, 19, 20]
x is 5                                         # fourth element in numbers
[3, 6, 6, 5, 7, 10, 7, 19, 19, 20]
x is 7                                         # fifth element in numbers
[3, 6, 6, 5, 10, 7, 19, 19, 20]
x is 7                                         # sixth element in numbers
x is 19                                        # seventh element in numbers
[3, 6, 6, 5, 10, 7, 19, 20]
x is 20                                        # eight and last element in numbers
I'm not 'in'-sane. Indeed, I am so far 'out' of sane that you appear a tiny blip on the distant coast of sanity. Bucky Katt, Get Fuzzy

Da Bishop: There's a dead bishop on the landing. I don't know who keeps bringing them in here. ....but society is to blame.
Reply
#9
(Jun-14-2019, 08:41 PM)perfringo Wrote: Printing is 'cheapest' debugging tool. If one adds two print statements everything is quite self-explanatory:

>>> numbers = [3, 3, 20, 6, 10, 6, 5, 5, 7, 10, 7, 19, 19, 20]
>>> for x in numbers:
...     print(f'x is {x}')
...     if numbers.count(x) > 1:
...         numbers.remove(x)
...         print(numbers)
... 
x is 3                                         # first element in numbers
[3, 20, 6, 10, 6, 5, 5, 7, 10, 7, 19, 19, 20]
x is 20                                        # second element in numbers
[3, 6, 10, 6, 5, 5, 7, 10, 7, 19, 19, 20]
x is 10                                        # third element in numbers
[3, 6, 6, 5, 5, 7, 10, 7, 19, 19, 20]
x is 5                                         # fourth element in numbers
[3, 6, 6, 5, 7, 10, 7, 19, 19, 20]
x is 7                                         # fifth element in numbers
[3, 6, 6, 5, 10, 7, 19, 19, 20]
x is 7                                         # sixth element in numbers
x is 19                                        # seventh element in numbers
[3, 6, 6, 5, 10, 7, 19, 20]
x is 20                                        # eight and last element in numbers

thanks, that further illustrates my problem.
Reply
#10
(Jun-14-2019, 06:50 PM)calonia Wrote: modifying a list while iterating through it(adding or removing items) leads to skipping indexes.
You should know that you can loop over a copy of a list however
The [:] returns a shallow copy of the list, in which this case doesnt matter because all of the elements are ints
numbers = [3, 3, 20, 6, 10, 6, 5, 5, 7, 10, 7, 19, 19, 20]
for x in numbers[:]:
    if numbers.count(x) > 1:
        numbers.remove(x)
print(numbers)
Output:
[3, 6, 5, 10, 7, 19, 20]
Recommended Tutorials:
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  How do I calculate a ratio from 2 numbers and return an equivalent list of about 1000 Pleiades 8 15,437 Jan-05-2024, 08:30 PM
Last Post: sgrey
  find random numbers that are = to the first 2 number of a list. Frankduc 23 3,012 Apr-05-2023, 07:36 PM
Last Post: Frankduc
  List of random numbers astral_travel 17 2,533 Dec-02-2022, 10:37 PM
Last Post: deanhystad
  Remove numbers from a list menator01 4 1,251 Nov-13-2022, 01:27 AM
Last Post: menator01
  [split] why can't i create a list of numbers (ints) with random.randrange() astral_travel 7 1,428 Oct-23-2022, 11:13 PM
Last Post: Pedroski55
  Divide a number by numbers in a list. Wallen 7 7,926 Feb-12-2022, 01:51 PM
Last Post: deanhystad
  producing numbers out of a list bouraque7878 10 3,621 Nov-12-2021, 09:13 PM
Last Post: jefsummers
  How to change odd to even numbers in the list? plumberpy 8 3,619 Aug-08-2021, 11:07 AM
Last Post: plumberpy
  convert numbers into list lokesh 1 2,346 Jun-03-2021, 06:37 AM
Last Post: menator01
  adding numbers in a list Nickd12 2 2,147 Jan-15-2021, 12:46 PM
Last Post: Serafim

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020