+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: General Coding Help (https://python-forum.io/forum-8.html)
+--- Thread: can itertools compact a list removing all of some value? (/thread-20776.html)
can itertools compact a list removing all of some value? - Skaperen - Aug-29-2019
i have a huge list (approaching a billion items) which is mostly strings and/or numbers, but there are a lot of
None values in there, too. i would like to compact the list in place by removing all the None values. can a method in some module do this or do i need to figure this out? the import part is to not duplicate the list since that would cause more swapping delays. but something that copies the remainder of the list each time it finds a None is also a bad idea since it bumps it up to O(n**2). something that morphs the list into a tuple is OK, since it is not modified after this.i have looked over
itertools but nothing seems obvious.
RE: can itertools compact a list removing all of some value? - Larz60+ - Aug-30-2019
>>> startlist = [4,None,'harry',67,None,67,'sss'] >>> startlist = [x for x in startlist if x is not None] >>> startlist [4, 'harry', 67, 67, 'sss'] >>>
RE: can itertools compact a list removing all of some value? - Skaperen - Aug-30-2019
that makes a new list while the old one exists, before it is garbage collected. it would swap like crazy with a huge list as the new one will be in a new memory location.
RE: can itertools compact a list removing all of some value? - perfringo - Aug-30-2019
It's usually dangerous to mutate the list while iterating over it. But if you know what are you doing then...
Simple code to remove elements in place (starting from end in order not to mess indices):
>>> test = [None, 'b', None, 'a', None] >>> for i in range(len(test) - 1, -1, -1): ... if test[i] == None: ... del test[i] ... >>> test ['b', 'a']
RE: can itertools compact a list removing all of some value? - Skaperen - Aug-30-2019
my first thought was:
n = 0
for i in range(len(test)):
if test[i] is not None:
test[n] = test[i]
n += 1
test[n:] = []but i don't know how safe or fast that is.
RE: can itertools compact a list removing all of some value? - Larz60+ - Aug-31-2019
if you remove data from the list starting from the end and moving towards the beginning, then there will be no index discrepancies during the purge.
RE: can itertools compact a list removing all of some value? - Skaperen - Sep-02-2019
but, would my way work? that's how i would it it in C (then store the new length where the length is kept). but i can understand the way working from the end. it would just seem to be a slower way because the list is shortened so many times.