Python Forum
removing duplicate numbers from a list - Printable Version

+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: General Coding Help (https://python-forum.io/forum-8.html)
+--- Thread: removing duplicate numbers from a list (/thread-19134.html)

Pages: 1 2

removing duplicate numbers from a list - calonia - Jun-14-2019

hi, there! i am new to python and i was trying to remove duplicate numbers in a list using for loop. but the code doesn't quite work

numbers = [3, 3, 20, 6, 10, 6, 5, 5, 7, 10, 7, 19, 19, 20]
for x in numbers:
    if numbers.count(x) > 1:
        numbers.remove(x)
print(numbers)
it doesn't remove all the duplicates.

this is what i get when i run it.

Output:
[3, 6, 6, 5, 10, 7, 19, 20]


RE: removing duplicate numbers from a list - metulburr - Jun-14-2019 

>>> numbers = [3, 3, 20, 6, 10, 6, 5, 5, 7, 10, 7, 19, 19, 20]
>>> list(set(numbers))
[3, 5, 6, 7, 10, 19, 20]
or

>>> from collections import OrderedDict
>>> numbers = [3, 3, 20, 6, 10, 6, 5, 5, 7, 10, 7, 19, 19, 20]
>>> list(OrderedDict.fromkeys(numbers))
[3, 20, 6, 10, 5, 7, 19]


RE: removing duplicate numbers from a list - ThomasL - Jun-14-2019

@calonia Modifying a list whilst iterating over it is a very bad idea.
The 2nd example from @metulburr is perfect as it keeps the order of the items in the original list.
If that doesn´t matter his 1st example is faster.
In addition your runtime is O(len(numbers)*len(numbers)) which is bad.
numbers.count(x) runs over the full list and counts each number.
Try to do this in O(len(numbers) which is possible. (not using above examples) :-)

RE: removing duplicate numbers from a list - snippsat - Jun-14-2019

Also from 3.6 and guaranteed in 3.7 so are dictionaries ordered.
This mean that can drop import of OrderedDict,and just use dict().
>>> numbers = [3, 3, 20, 6, 10, 6, 5, 5, 7, 10, 7, 19, 19, 20]
>>> list(dict.fromkeys(numbers))
[3, 20, 6, 10, 5, 7, 19]


RE: removing duplicate numbers from a list - calonia - Jun-14-2019

(Jun-14-2019, 04:10 PM)ThomasL Wrote: @calonia Modifying a list whilst iterating over it is a very bad idea.
The 2nd example from @metulburr is perfect as it keeps the order of the items in the original list.
If that doesn´t matter his 1st example is faster.
In addition your runtime is O(len(numbers)*len(numbers)) which is bad.
numbers.count(x) runs over the full list and counts each number.
Try to do this in O(len(numbers) which is possible. (not using above examples) :-)

thanks for the reply. i really appreciate your help. but i want to know what is wrong with this code. it doesn't seem to work with some of the numbers!

RE: removing duplicate numbers from a list - ThomasL - Jun-14-2019

Have a read here

RE: removing duplicate numbers from a list - calonia - Jun-14-2019

(Jun-14-2019, 05:13 PM)ThomasL Wrote: Have a read here

thank you very much. that was so helpful. i now get it. modifying a list while iterating through it(adding or removing items) leads to skipping indexes.

RE: removing duplicate numbers from a list - perfringo - Jun-14-2019

Printing is 'cheapest' debugging tool. If one adds two print statements everything is quite self-explanatory:

>>> numbers = [3, 3, 20, 6, 10, 6, 5, 5, 7, 10, 7, 19, 19, 20]
>>> for x in numbers:
...     print(f'x is {x}')
...     if numbers.count(x) > 1:
...         numbers.remove(x)
...         print(numbers)
... 
x is 3                                         # first element in numbers
[3, 20, 6, 10, 6, 5, 5, 7, 10, 7, 19, 19, 20]
x is 20                                        # second element in numbers
[3, 6, 10, 6, 5, 5, 7, 10, 7, 19, 19, 20]
x is 10                                        # third element in numbers
[3, 6, 6, 5, 5, 7, 10, 7, 19, 19, 20]
x is 5                                         # fourth element in numbers
[3, 6, 6, 5, 7, 10, 7, 19, 19, 20]
x is 7                                         # fifth element in numbers
[3, 6, 6, 5, 10, 7, 19, 19, 20]
x is 7                                         # sixth element in numbers
x is 19                                        # seventh element in numbers
[3, 6, 6, 5, 10, 7, 19, 20]
x is 20                                        # eight and last element in numbers


RE: removing duplicate numbers from a list - calonia - Jun-14-2019

(Jun-14-2019, 08:41 PM)perfringo Wrote: Printing is 'cheapest' debugging tool. If one adds two print statements everything is quite self-explanatory:

>>> numbers = [3, 3, 20, 6, 10, 6, 5, 5, 7, 10, 7, 19, 19, 20]
>>> for x in numbers:
...     print(f'x is {x}')
...     if numbers.count(x) > 1:
...         numbers.remove(x)
...         print(numbers)
... 
x is 3                                         # first element in numbers
[3, 20, 6, 10, 6, 5, 5, 7, 10, 7, 19, 19, 20]
x is 20                                        # second element in numbers
[3, 6, 10, 6, 5, 5, 7, 10, 7, 19, 19, 20]
x is 10                                        # third element in numbers
[3, 6, 6, 5, 5, 7, 10, 7, 19, 19, 20]
x is 5                                         # fourth element in numbers
[3, 6, 6, 5, 7, 10, 7, 19, 19, 20]
x is 7                                         # fifth element in numbers
[3, 6, 6, 5, 10, 7, 19, 19, 20]
x is 7                                         # sixth element in numbers
x is 19                                        # seventh element in numbers
[3, 6, 6, 5, 10, 7, 19, 20]
x is 20                                        # eight and last element in numbers

thanks, that further illustrates my problem.

RE: removing duplicate numbers from a list - metulburr - Jun-14-2019

(Jun-14-2019, 06:50 PM)calonia Wrote: modifying a list while iterating through it(adding or removing items) leads to skipping indexes.
You should know that you can loop over a copy of a list however
The [:] returns a shallow copy of the list, in which this case doesnt matter because all of the elements are ints
numbers = [3, 3, 20, 6, 10, 6, 5, 5, 7, 10, 7, 19, 19, 20]
for x in numbers[:]:
    if numbers.count(x) > 1:
        numbers.remove(x)
print(numbers)
Output:
[3, 6, 5, 10, 7, 19, 20]