Python Forum
can itertools compact a list removing all of some value?
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
can itertools compact a list removing all of some value?
#1
i have a huge list (approaching a billion items) which is mostly strings and/or numbers, but there are a lot of None values in there, too. i would like to compact the list in place by removing all the None values. can a method in some module do this or do i need to figure this out? the import part is to not duplicate the list since that would cause more swapping delays. but something that copies the remainder of the list each time it finds a None is also a bad idea since it bumps it up to O(n**2). something that morphs the list into a tuple is OK, since it is not modified after this.

i have looked over itertools but nothing seems obvious.
Tradition is peer pressure from dead people

What do you call someone who speaks three languages? Trilingual. Two languages? Bilingual. One language? American.
Reply
#2
>>> startlist = [4,None,'harry',67,None,67,'sss']
>>> startlist = [x for x in startlist if x is not None]
>>> startlist
[4, 'harry', 67, 67, 'sss']
>>>
Reply
#3
that makes a new list while the old one exists, before it is garbage collected. it would swap like crazy with a huge list as the new one will be in a new memory location.
Tradition is peer pressure from dead people

What do you call someone who speaks three languages? Trilingual. Two languages? Bilingual. One language? American.
Reply
#4
It's usually dangerous to mutate the list while iterating over it. But if you know what are you doing then...

Simple code to remove elements in place (starting from end in order not to mess indices):

>>> test = [None, 'b', None, 'a', None]
>>> for i in range(len(test) - 1, -1, -1):
...     if test[i] == None:
...         del test[i]
...
>>> test
['b', 'a']
I'm not 'in'-sane. Indeed, I am so far 'out' of sane that you appear a tiny blip on the distant coast of sanity. Bucky Katt, Get Fuzzy

Da Bishop: There's a dead bishop on the landing. I don't know who keeps bringing them in here. ....but society is to blame.
Reply
#5
my first thought was:
n = 0
for i in range(len(test)):
    if test[i] is not None:
        test[n] = test[i]
        n += 1
test[n:] = []
but i don't know how safe or fast that is.
Tradition is peer pressure from dead people

What do you call someone who speaks three languages? Trilingual. Two languages? Bilingual. One language? American.
Reply
#6
if you remove data from the list starting from the end and moving towards the beginning, then there will be no index discrepancies during the purge.
Reply
#7
but, would my way work? that's how i would it it in C (then store the new length where the length is kept). but i can understand the way working from the end. it would just seem to be a slower way because the list is shortened so many times.
Tradition is peer pressure from dead people

What do you call someone who speaks three languages? Trilingual. Two languages? Bilingual. One language? American.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  itertools and amazing speed Pedroski55 8 1,984 Nov-11-2022, 01:20 PM
Last Post: Gribouillis
  What happens to a <itertools.permutations object at 0x7fe3cc66af68> after it is read? Pedroski55 3 2,354 Nov-29-2020, 08:35 AM
Last Post: DeaD_EyE
  Removing element from list squall 6 3,023 Nov-22-2020, 09:34 PM
Last Post: jefsummers
  Making lists using itertools and eliminating duplicates. mike3891 2 2,189 Oct-26-2020, 05:39 PM
Last Post: bowlofred
  Removing items from list if containing a substring pythonnewbie138 2 2,150 Aug-27-2020, 10:20 PM
Last Post: pythonnewbie138
  Python3 itertools.groupby printing the key august 1 2,043 Aug-17-2020, 05:46 AM
Last Post: bowlofred
  removing dictionary element in list using (key, value) MelonMusk 3 2,231 Jun-13-2020, 02:37 PM
Last Post: buran
  Generate Cartesian Products with Itertools Incrementally CoderMan 2 1,810 Jun-04-2020, 04:51 PM
Last Post: CoderMan
  Help removing asterisk item in a nested list. bmcguire 3 2,549 Apr-06-2020, 02:35 PM
Last Post: snippsat
  itertools.zip_shortest() fo unequal iterators Skaperen 10 6,580 Dec-27-2019, 12:17 AM
Last Post: Skaperen

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020