Python Forum
Code golfing: splitting a list
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Code golfing: splitting a list
#1
Question 
So I have a list L. To do some parallel processing, I want to split it into N sublists of equivalent length. The contents of the sublists are indifferent. What is your shortest and/or most pythonic code? Can you avoid using len(L) explicitly? 

python python python
Unless noted otherwise, code in my posts should be understood as "coding suggestions", and its use may require more neurones than the two necessary for Ctrl-C/Ctrl-V.
Your one-stop place for all your GIMP needs: gimp-forum.net
Reply
#2

Assumes N divides evenly into len(L). If that's a concern you could put a second index into the slice double zip it to account for that.
Craig "Ichabod" O'Brien - xenomind.com
I wish you happiness.
Recommended Tutorials: BBCode, functions, classes, text adventures
Reply
#3
So you want to chunk?
Lots of good answers here:
http://stackoverflow.com/questions/31244...zed-chunks

The standard idiom people generally use is:
chunks = (L[i:i + N] for i in range(0, len(L), N))
This assumes the whole sequence fits in memory which it sounds like you are trying to avoid.

Alternatives use iter and zip and look like this:
chunks_zip = zip(*[iter(L)]*N)
The second version above ignores groups that are not full.  But can handle generators that wouldn't otherwise fit in memory:
Python3
L = range(10**25)
N = 3

chunks_zip = zip(*[iter(L)]*N)

for i in range(10):
    print(next(chunks_zip))
Output:
(0, 1, 2) (3, 4, 5) (6, 7, 8) (9, 10, 11) (12, 13, 14) (15, 16, 17) (18, 19, 20) (21, 22, 23) (24, 25, 26) (27, 28, 29)
Use zip_longest with a fill value if you don't want to lose incomplete groups.
Reply
#4
(Jan-12-2017, 01:26 AM)ichabod801 Wrote:

Assumes N divides evenly into len(L). If that's a concern you could put a second index into the slice double zip it to account for that.

This is more or less what my code does, but with slightly clumsier Python. And as far as I can tell this works even if N is doesn't divide len(N).

(Jan-12-2017, 02:41 AM)Mekire Wrote: So you want to chunk?
Lots of good answers here:
http://stackoverflow.com/questions/31244...zed-chunks

The standard idiom people generally use is:
chunks = (L[i:i + N] for i in range(0, len(L), N))
This assumes the whole sequence fits in memory which it sounds like you are trying to avoid.
No, in the real-life problem I am handling rather small lists, it just that each item ends up used in a webservice call so to speed things up I create N threads and give each a part of the original list.

(Jan-12-2017, 02:41 AM)Mekire Wrote: Alternatives use iter and zip and look like this:
chunks_zip = zip(*[iter(L)]*N)
The second version above ignores groups that are not full.  But can handle generators that wouldn't otherwise fit in memory:
Python3
L = range(10**25)
N = 3

chunks_zip = zip(*[iter(L)]*N)

for i in range(10):
    print(next(chunks_zip))
Output:
(0, 1, 2) (3, 4, 5) (6, 7, 8) (9, 10, 11) (12, 13, 14) (15, 16, 17) (18, 19, 20) (21, 22, 23) (24, 25, 26) (27, 28, 29)
Use zip_longest with a fill value if you don't want to lose incomplete groups.
Clever, but if you are unlucky, one of the chunks produced by the idioms above can have a length of 1 so the maximum size difference is N-1. With the 1 every N sampling you get more uniform sizes (but possibly using more CPU/memory, which isn't really a concern for me but could be for someone else).
Unless noted otherwise, code in my posts should be understood as "coding suggestions", and its use may require more neurones than the two necessary for Ctrl-C/Ctrl-V.
Your one-stop place for all your GIMP needs: gimp-forum.net
Reply
#5
There is an example in the Python documentation: https://docs.python.org/3.5/library/iter...ls-recipes

def grouper(iterable, n, fillvalue=None):
    "Collect data into fixed-length chunks or blocks"
    # grouper('ABCDEFG', 3, 'x') --> ABC DEF Gxx"
    args = [iter(iterable)] * n
    return zip_longest(*args, fillvalue=fillvalue)
"As they say in Mexico 'dosvidaniya'. That makes two vidaniyas."
https://freedns.afraid.org
Reply
#6
(Jan-12-2017, 09:13 AM)wavic Wrote: There is an example in the Python documentation: https://docs.python.org/3.5/library/iter...ls-recipes

def grouper(iterable, n, fillvalue=None):
    "Collect data into fixed-length chunks or blocks"
    # grouper('ABCDEFG', 3, 'x') --> ABC DEF Gxx"
    args = [iter(iterable)] * n
    return zip_longest(*args, fillvalue=fillvalue)

Yes, this is Mekire's suggestion...
Unless noted otherwise, code in my posts should be understood as "coding suggestions", and its use may require more neurones than the two necessary for Ctrl-C/Ctrl-V.
Your one-stop place for all your GIMP needs: gimp-forum.net
Reply
#7
(Jan-12-2017, 08:10 AM)Ofnuts Wrote: This is more or less what my code does, but with slightly clumsier Python. And as far as I can tell this works even if N is doesn't divide len(N).

You said you wanted lists of equivalent length. I took that to mean equal length. If N does not divide evenly into len(L), then some of the lists will be one item longer. The fixes I proposed would make them all the same length, but they would drop some items.
Craig "Ichabod" O'Brien - xenomind.com
I wish you happiness.
Recommended Tutorials: BBCode, functions, classes, text adventures
Reply
#8
How about this

It looks simple enough for everyone.
"As they say in Mexico 'dosvidaniya'. That makes two vidaniyas."
https://freedns.afraid.org
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  "pythonic" way of splitting up my code? liggisten 4 611 Dec-30-2023, 08:23 PM
Last Post: Gribouillis
  Splitting code into several files TLammert 4 1,446 Jun-26-2022, 02:33 PM
Last Post: TLammert
  Splitting strings in list of strings jesse68 3 1,704 Mar-02-2022, 05:15 PM
Last Post: DeaD_EyE
  Splitting String into 2d list cclark135 2 2,757 Aug-26-2019, 01:46 PM
Last Post: ThomasL
  splitting numeric list based on condition python_newbie09 7 9,281 May-27-2019, 03:58 PM
Last Post: python_newbie09
  Need help | splitting list into list Vinci141 3 2,551 Mar-13-2019, 09:09 PM
Last Post: Vinci141
  some ideas for intelligent list splitting? wardancer84 4 3,180 Nov-20-2018, 02:47 PM
Last Post: DeaD_EyE
  Splitting a list help johiah 5 4,648 Mar-21-2017, 04:31 PM
Last Post: metulburr

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020