Problem about yield, please help!!

[cls0724]

I am still new to Python and i come across the below sample from my textbook that confuses me quite a lot.

[u]def flatten(nested, indent=""):
  try:
    print(indent, "Going to iterate over", nested)
    for sublist in nested:
      print(indent, "Going to iterate over flattening of",sublist)
      for element in flatten(sublist, indent+"  "):
        print(indent, "Yielding", element)
        yield element
  except TypeError:
    print(indent, "Type Error!  Yielding", nested)
    yield nested[/u]

The above code is to flatten a list. I understand how a list is flattened till you reach a number.
Then you reach Type Error.

However, i dun understand the next step when the

yield nested

then jump to

for element in flatten

.

I can comprehend the logic in between. I thought once you reached TypeError then it's the end of this loop. Why it will lead back to

for element in flatten

.

Thanks!

[deanhystad]

Yield returns a value and then freezes until the function is called again. It is really pretty cool. Functions like this are called generators if you want to read up on them.

For this example the code is almost equivalent to:

def flatten(nested, indent=""):
    flattened = []
    try:
        print(indent, "Going to iterate over", nested)
        for sublist in nested:
            print(indent, "Going to iterate over flattening of",sublist)
            for element in flatten(sublist, indent+"  "):
                print(indent, "Yielding", element)
                flattened.append(element)
        return flattened
    except TypeError:
        print(indent, "Type Error!  Yielding", nested)
        yield nested

However being generator there is no need to flatten the entire list and build the entire flattened list. The generator can just return the next element. If I wanted the flattened list for some reason I could always use a list comprehension.

flattened_list = [element for element in flatten(source_list)]

Generators are better than functions that return lists because generators don't make you wait while the list is built (which can be significant) and they use very little memory.

[cls0724]

sorry, but i still dun get how

yield nested

can link to

yield element

here.
I thought

yield nested

just returns a value then that's it? So why would we need

yield element

Sorry, this recursive generator is very confusing....

[deanhystad]

The easiest way to understand this code is generate a list with an error and see what it does.

def flatten(nested):
    try:
        for sublist in nested:
            for element in flatten(sublist):
                if element == 5:  # Force a type error
                    raise TypeError()
                yield element
    except TypeError:
        yield nested

src = (1, 2, (3, (4, 5, 6, 7), (9, 10)))
flattened = [element for element in flatten(src)]
print(flattened)

Output:
[1, 2, 3, 4, (4, 5, 6, 7), 9, 10]

As you can see, the generator is returning elements one at a time until it encounters 5 which forces a type error. At this time we are three layers deep in the recursion.
flatten((1, 2, (3, (4, 5, 6, 7), (9, 10))))
.flatten((3, (4, 5, 6, 7)))
..flatten((4, 5, 6, 7))

According to the exception code, the flatten function yields the offending list. The list is the original sublist and is not altered in any way, even though the exception didn't occur until processing the second element.

Unlike return, yield does not have to walk back up through the recursion. The value is returned (yielded) and the code freezes until the next iteration. For our faulted list, that is at the end of the function and the next time though we will wind back and begin processing the next element.

Does that help at all?

[cls0724]

Thanks for your help!
While i am trying my very best to understand the logic behind...but my question remains there.

Perhaps let me try to elaborate my concern in a more detailed way.

My understanding of the code is that the below recursion will keep going until i reach the 'flat' layer, i.e. 1

for sublist in nested:
            for element in flatten(sublist):

At this point, the 'flat' layer cannot be used in for again because it is an integer. So a TypeError takes places.

In order to capture it, we have the below code

except TypeError:
        yield nested

In this case, yield nested will return 1.

Then i dun understand what happens next. As you suggested, i thought the code freezes right there and gives us 1. I have to use next() to return the next 'flat' layer.

Nonetheless, it seems to me that it is the

yield element

that gives us 1 instead of

yield nested

.

Sorry for bothering you!

[deanhystad]

My earlier description is wrong and you are correct. The exception code is how elements are selected for return. When you call flatten with an element instead of a list, the "sublist in nested" inside the recursion generates an exception. The exception yields a value back to the caller and that becomes "element" for the caller's "for element in flatten).

So this is not a generator, but a generator inside of a generator, or a generator inside a generator inside a generator.... Each generator yields values to the "calling" generator. Pretty cool.

Things became a lot clearer for me when I wrote the code like this:

def flatten(nested, indent=""):
    try:
        for sublist in nested:
            for element in flatten(sublist, indent+"+"):
                yield indent
                # yield element
    except TypeError:
        pass
        # print(indent, nested)
        yield nested
 
for e in flatten((1, 2, (3, 4, (5, 6), 7), 8, (9, 10)), '-'):
    # pass
    print(e)

When I switch the comments around to print from inside flatten and pass the return value I see the list with indents. This makes it obvious the exception code is called for each individual element and not for any sublists. Then comment out the print in flatten and print the return value and it is all '-' for each element, clearly demonstrating that the values returned to the caller come from the for loop yield, and that the values are yielded to the caller which yields to the caller all the way up to the first call (thus no "+"). So it kind of works like normal recursion in that values are all returned by the first/top function, but it also does the yield/freeze thing, freezing the entire recursion chain right after yielding the value (kind of a yield/unchain/yield/unchain...).

Whew! That is a bit of a brain twister.