Python Forum

I have a sorted list of integers that has about 4000 elements. I'm trying to remove every element that is above 550. I used remove() function for the task but it doesn't seem to work. I also looked into pop, tried it and it doesn't seem to work to because pop I think is for storing single data while also removing it from the original list. I also tried del() function. Since my list is sorted, I could use the del to remove the numbers above 550 just by figuring out from what index the element starts.

Here is my current code using the remove() function:

sorted_MD_list = sorted(MD_list)

for i in sorted_MD_list:
    if i > 550:
        sorted_MD_list.remove(i)

print(sorted_MD_list)

The result when i print the sorted list is the same list including the values that are above 550.

Please use bbtags when posting code.

Here is one way

# Create a list
alist = [i for i in range(50)]
sorted(alist)

print(alist)
print()
# Create a copy of the list.
# Should not try to edit a list that you are iterating over
alist_copy = alist.copy()

# Iterate over the copy and remove anything above chosen number
# from original list
for index, number in enumerate(alist_copy):
    if number > 20:
        alist.remove(number)

print(alist)

Output

Output:
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49]

[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20]

This is because you're altering the list object as you iterate over it. The way that I would do this is to have your .remove() loop compile a list of values that need to be removed, than use that list to find and remove the values from the sorted_MD_list.

(Nov-28-2023, 12:17 AM)menator01 Wrote: [ -> ]Please use bbtags when posting code.

Here is one way

# Create a list
alist = [i for i in range(50)]
sorted(alist)

print(alist)
print()
# Create a copy of the list.
# Should not try to edit a list that you are iterating over
alist_copy = alist.copy()

# Iterate over the copy and remove anything above chosen number
# from original list
for index, number in enumerate(alist_copy):
    if number > 20:
        alist.remove(number)

print(alist)

Output

Output:
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49]

[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20]

Hello, sorry but I don't know how to do the bbtags. As for the code. Wow! It works. Why is that? Why can't I iterate over the original list?

(Nov-28-2023, 01:54 AM)gohanhango Wrote: [ -> ]Why can't I iterate over the original list?

If you think about it, if you have a list of 1,2,3,4,5 and you iterate over that with a for n in lst: loop, n will return each element in turn, but if you remove one of the elements mid loop (say when n == 3) the next element becomes 5, not 4, because each element is shifted one index to the left to fill the now vacated index position: the 4 to where the 3 was and the 5 to where the 4 was.

(Nov-28-2023, 12:18 AM)rob101 Wrote: [ -> ]This is because you're altering the list object as you iterate over it. The way that I would do this is to have your .remove() loop compile a list of values that need to be removed, than use that list to find and remove the values from the sorted_MD_list.

Thank you.

I would delete a slice.

values = list(range(1, 4001))
for index, value in enumerate(values):
    if value >= 550:
        del values[index:]
        break
print(len(values))

Output:
549

If it is ok to reassign the list, I would make a slice.

values = list(range(1, 4001))
for index, value in enumerate(values):
    if value >= 550:
        values = values[:index]
        break
print(len(values))

For a tiny list like 4000 values the difference between deleting a slice and using remove is zero time to maybe 1/10th of a second. For a list with hundreds of millions of values the difference becomes significant.

I would have used numpy and np.where + np.delete (it has been fully detailled here after to see how it works); actually only lines 9-11-12-(22) are necessary.

import numpy as np

# a list is created first using numpy
N = 5_000
M = np.random.randint(-2, 10_000, size=(N)).tolist()

# it starts here
MaxNumber = 4_000
Mat = np.asarray(M)

Ind = np.where(Mat >= MaxNumber)
n = np.shape(Ind)[1]
print(f"There's {n} elements to remove")

Mat = np.delete(Mat, np.s_[Ind])

# you can of course sort the array if it's needed
Indexes = Mat.argsort()
MatSorted = Mat[Indexes]

# now you can go back to a list
M_new = MatSorted.tolist()

Others have already explained why you should not remove elements from a list while iterating over it.
Given that you have sorted list, this is very nice use case for bisect module from Standard Library

import random
import bisect

# create sorted list of 10 elements
spam = sorted([random.randint(1, 100) for _ in range(10)])
print(spam)

# remove all elements equal or greater than 42
cutoff = bisect.bisect_left(spam, 42)
eggs = spam[:cutoff]
print(eggs)

Output:
[19, 31, 32, 32, 44, 47, 61, 68, 70, 85]
[19, 31, 32, 32]

You can explore also other module level functions

If the list is sorted, you can simply use bisect which performs a logarithmic search

import random
import bisect

# create sorted list of numbers
alist = sorted(random.choices(range(10000), k=4000))

# find insertion point to the right of a given value
index = bisect.bisect_right(alist, 550)
# keep only elements lower or equal to the given value
alist[:] = alist[:index]

print(alist)