Hello guest, if you read this it means you are not registered. Click here to register in a few simple steps, you will enjoy all features of our Forum.
Bottom Page

Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Code golfing: splitting a list
#1
Question 
So I have a list L. To do some parallel processing, I want to split it into N sublists of equivalent length. The contents of the sublists are indifferent. What is your shortest and/or most pythonic code? Can you avoid using len(L) explicitly? 

python python python
Unless noted otherwise, code in my posts should be understood as "coding suggestions", and its use may require more neurones than the two necessary for Ctrl-C/Ctrl-V.
Your one-stop place for all your GIMP needs: gimp-forum.net
Quote
#2

Assumes N divides evenly into len(L). If that's a concern you could put a second index into the slice double zip it to account for that.
Ofnuts likes this post
Craig "Ichabod" O'Brien - xenomind.com
Buddhist, biker, poet, coder, theist
Recommended Tutorials: BBCode, functions, classes, text adventures

Quote
#3
So you want to chunk?
Lots of good answers here:
http://stackoverflow.com/questions/31244...zed-chunks

The standard idiom people generally use is:
chunks = (L[i:i + N] for i in range(0, len(L), N))
This assumes the whole sequence fits in memory which it sounds like you are trying to avoid.

Alternatives use iter and zip and look like this:
chunks_zip = zip(*[iter(L)]*N)
The second version above ignores groups that are not full.  But can handle generators that wouldn't otherwise fit in memory:
Python3
L = range(10**25)
N = 3

chunks_zip = zip(*[iter(L)]*N)

for i in range(10):
    print(next(chunks_zip))
Output:
(0, 1, 2) (3, 4, 5) (6, 7, 8) (9, 10, 11) (12, 13, 14) (15, 16, 17) (18, 19, 20) (21, 22, 23) (24, 25, 26) (27, 28, 29)
Use zip_longest with a fill value if you don't want to lose incomplete groups.
Quote
#4
(Jan-12-2017, 01:26 AM)ichabod801 Wrote:

Assumes N divides evenly into len(L). If that's a concern you could put a second index into the slice double zip it to account for that.

This is more or less what my code does, but with slightly clumsier Python. And as far as I can tell this works even if N is doesn't divide len(N).

(Jan-12-2017, 02:41 AM)Mekire Wrote: So you want to chunk?
Lots of good answers here:
http://stackoverflow.com/questions/31244...zed-chunks

The standard idiom people generally use is:
chunks = (L[i:i + N] for i in range(0, len(L), N))
This assumes the whole sequence fits in memory which it sounds like you are trying to avoid.
No, in the real-life problem I am handling rather small lists, it just that each item ends up used in a webservice call so to speed things up I create N threads and give each a part of the original list.

(Jan-12-2017, 02:41 AM)Mekire Wrote: Alternatives use iter and zip and look like this:
chunks_zip = zip(*[iter(L)]*N)
The second version above ignores groups that are not full.  But can handle generators that wouldn't otherwise fit in memory:
Python3
L = range(10**25)
N = 3

chunks_zip = zip(*[iter(L)]*N)

for i in range(10):
    print(next(chunks_zip))
Output:
(0, 1, 2) (3, 4, 5) (6, 7, 8) (9, 10, 11) (12, 13, 14) (15, 16, 17) (18, 19, 20) (21, 22, 23) (24, 25, 26) (27, 28, 29)
Use zip_longest with a fill value if you don't want to lose incomplete groups.
Clever, but if you are unlucky, one of the chunks produced by the idioms above can have a length of 1 so the maximum size difference is N-1. With the 1 every N sampling you get more uniform sizes (but possibly using more CPU/memory, which isn't really a concern for me but could be for someone else).
Unless noted otherwise, code in my posts should be understood as "coding suggestions", and its use may require more neurones than the two necessary for Ctrl-C/Ctrl-V.
Your one-stop place for all your GIMP needs: gimp-forum.net
Quote
#5
There is an example in the Python documentation: https://docs.python.org/3.5/library/iter...ls-recipes

def grouper(iterable, n, fillvalue=None):
    "Collect data into fixed-length chunks or blocks"
    # grouper('ABCDEFG', 3, 'x') --> ABC DEF Gxx"
    args = [iter(iterable)] * n
    return zip_longest(*args, fillvalue=fillvalue)
"As they say in Mexico 'dosvidaniya'. That makes two vidaniyas."
https://freedns.afraid.org
Quote
#6
(Jan-12-2017, 09:13 AM)wavic Wrote: There is an example in the Python documentation: https://docs.python.org/3.5/library/iter...ls-recipes

def grouper(iterable, n, fillvalue=None):
    "Collect data into fixed-length chunks or blocks"
    # grouper('ABCDEFG', 3, 'x') --> ABC DEF Gxx"
    args = [iter(iterable)] * n
    return zip_longest(*args, fillvalue=fillvalue)

Yes, this is Mekire's suggestion...
Mekire likes this post
Unless noted otherwise, code in my posts should be understood as "coding suggestions", and its use may require more neurones than the two necessary for Ctrl-C/Ctrl-V.
Your one-stop place for all your GIMP needs: gimp-forum.net
Quote
#7
(Jan-12-2017, 08:10 AM)Ofnuts Wrote: This is more or less what my code does, but with slightly clumsier Python. And as far as I can tell this works even if N is doesn't divide len(N).

You said you wanted lists of equivalent length. I took that to mean equal length. If N does not divide evenly into len(L), then some of the lists will be one item longer. The fixes I proposed would make them all the same length, but they would drop some items.
Craig "Ichabod" O'Brien - xenomind.com
Buddhist, biker, poet, coder, theist
Recommended Tutorials: BBCode, functions, classes, text adventures

Quote
#8
How about this

It looks simple enough for everyone.
"As they say in Mexico 'dosvidaniya'. That makes two vidaniyas."
https://freedns.afraid.org
Quote

Top Page

Possibly Related Threads...
Thread Author Replies Views Last Post
  splitting or parsing control characters Skaperen 19 291 Yesterday, 07:27 PM
Last Post: zivoni
  Help with code/URGENT/Adding position to register list Laxman 3 131 Apr-23-2017, 04:48 PM
Last Post: micseydel
  Splitting a list help johiah 5 206 Mar-21-2017, 04:31 PM
Last Post: metulburr
  splitting a string by different splits Skaperen 9 423 Feb-09-2017, 08:59 AM
Last Post: Ofnuts
  splitting a string by 2 characters Skaperen 8 478 Dec-27-2016, 06:14 AM
Last Post: wavic
  splitting up long number iFunKtion 7 593 Dec-07-2016, 05:58 PM
Last Post: Larz60+
  splitting a string numeically Skaperen 14 973 Oct-10-2016, 03:39 PM
Last Post: micseydel

Forum Jump:


Users browsing this thread: 1 Guest(s)