parallel iterables

[Skaperen]

is there a tool (an iterable) that can iterate over a list of given iterables in parallel?

[Gribouillis]

zip?

[Skaperen]

in the mean time i wrote my own. it has an option pad= that allows specifying a pad for shorter iterables (if not specified then the shortest iterable stops the whole thing). i also has a last= option that allows specifying a final yield when all iterables are done. if it is given an int instead of a real iterable, it passes that int to range(that_int) and uses that. likewise if what is passed is literally a list or tuple of len() in (1,2,3) then those ints are passed as args to range() to become the iterable that is used. iow, cheap ways to do ranges.

since zip() did not match my project's need, i guess i didn't think about it when i coded my own variant (still in pre-alpha ... i haven't even coded a test for it, yet)

[Yoriz]

There is also

itertools.zip_longest(*iterables, fillvalue=None)
Make an iterator that aggregates elements from each of the iterables. If the iterables are of uneven length, missing values are filled-in with fillvalue. Iteration continues until the longest iterable is exhausted

[Gribouillis]

Also have a look at itertools.zip_longest() and various flavours of zip() in module more_itertools.

[Skaperen]

yes, i saw that from your first zip() link. that's what made me think of my code. it would have been more handy if the original built-in zip() had an option to do that. then we would not have had any need for a separate itertools.zip_longest(). i guess whoever implement itertools.zip_longest() was not in a position to add the option to zip() or missed the idea of that concept.

it's the way i think when i design something. Python made this thinking even easier with key word arguments. where the mere existence can mean something. for example, pad=None and last=None can be used in my generator to do padding of short iterables and add an extra yield at the end if desired. in a lot of other cases, the value None is taken as if not specified.

[buran]

i guess whoever implement itertools.zip_longest() was not in a position to add the option to zip() or missed the idea of that concept.

At the moment zip() acts as zip_shortest. If they had added pad=None to zip(), instead of adding itertools.zip_longest() this behavior would have been lost - zip_shortest and zip_longest with default value (i.e. None) are not the same thing (i.e. default value means that you want to stop when shortest is exhauset or you indeed want to pad shortest with None and iterate till longest is exhausted?). So I guess it is well thought and intentional not to change zip(), but add zip_longest instead. One more point they considered, I guess, is it would have break any existing code that rely on zip stop when shortest iterable is exhausted.

[Gribouillis]

it would have break any existing code that rely on zip stop when shortest iterable is exhausted.

They could have added an option longest=False to the existing zip. If I remember well, the story is rather that zip() returned initially a list and functions in module itertools return iterables, so they created izip and izip_longest separately from zip. At some point it was decided that many functions from the standard library that returned lists should now return iterables and izip became unnecessary.

The question remains of whether we should alter the behavior of functions by adding more and more keyword arguments, in the same way that linux commands have many command line switches, or if we must write separate functions with similar behaviors. In Martin Fowler's Refactoring book, he favours the use of different names for the sake of clarity and explicitness, so zip_longest(a, b) is better code than a zip(a, b, longest=True) would be. On the other hand in Python we have curryfication, so if our library functions have keyword arguments, we can easily write zip_longest = partial(zip, longest=True) for the sake of clarity.

[buran]

f I remember well, the story is rather that zip() returned initially a list and functions in module itertools return iterables, so they created izip and izip_longest separately from zip.

yeah, I completely forgot there was zip and the lazy izip

[Skaperen]

this behavior would have been lost

i don't see how it would be lost. if called without option pad= then it would act shortest. i called with option pad= then it would act longest (which needs to know what to pad shorter iterables with). so if called with pad=None this is saying to use the value None as the pad. so there needs to be a difference between using the pad=None option and not using the pad= option at all. i'm not inclined to post untested code where newbies might see it so i'm not going to show the whole thing, right now. but part of it has:

    do_stop = True
    if 'pad' in kwargs:
        do_stop = False
        pad = kwargs.pop('pad')