Python Forum
Problem about yield, please help!!
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Problem about yield, please help!!
#1
I am still new to Python and i come across the below sample from my textbook that confuses me quite a lot.

[u]def flatten(nested, indent=""):
  try:
    print(indent, "Going to iterate over", nested)
    for sublist in nested:
      print(indent, "Going to iterate over flattening of",sublist)
      for element in flatten(sublist, indent+"  "):
        print(indent, "Yielding", element)
        yield element
  except TypeError:
    print(indent, "Type Error!  Yielding", nested)
    yield nested[/u]
The above code is to flatten a list. I understand how a list is flattened till you reach a number.
Then you reach Type Error.

However, i dun understand the next step when the
yield nested
then jump to
for element in flatten
.

I can comprehend the logic in between. I thought once you reached TypeError then it's the end of this loop. Why it will lead back to
for element in flatten
.

Thanks!
Reply
#2
Yield returns a value and then freezes until the function is called again. It is really pretty cool. Functions like this are called generators if you want to read up on them.

For this example the code is almost equivalent to:
def flatten(nested, indent=""):
    flattened = []
    try:
        print(indent, "Going to iterate over", nested)
        for sublist in nested:
            print(indent, "Going to iterate over flattening of",sublist)
            for element in flatten(sublist, indent+"  "):
                print(indent, "Yielding", element)
                flattened.append(element)
        return flattened
    except TypeError:
        print(indent, "Type Error!  Yielding", nested)
        yield nested
However being generator there is no need to flatten the entire list and build the entire flattened list. The generator can just return the next element. If I wanted the flattened list for some reason I could always use a list comprehension.
flattened_list = [element for element in flatten(source_list)]
Generators are better than functions that return lists because generators don't make you wait while the list is built (which can be significant) and they use very little memory.
Reply
#3
sorry, but i still dun get how
yield nested
can link to
yield element
here.
I thought
yield nested
just returns a value then that's it? So why would we need
yield element
Sorry, this recursive generator is very confusing....
Reply
#4
The easiest way to understand this code is generate a list with an error and see what it does.
def flatten(nested):
    try:
        for sublist in nested:
            for element in flatten(sublist):
                if element == 5:  # Force a type error
                    raise TypeError()
                yield element
    except TypeError:
        yield nested

src = (1, 2, (3, (4, 5, 6, 7), (9, 10)))
flattened = [element for element in flatten(src)]
print(flattened)
Output:
[1, 2, 3, 4, (4, 5, 6, 7), 9, 10]
As you can see, the generator is returning elements one at a time until it encounters 5 which forces a type error. At this time we are three layers deep in the recursion.
flatten((1, 2, (3, (4, 5, 6, 7), (9, 10))))
.flatten((3, (4, 5, 6, 7)))
..flatten((4, 5, 6, 7))

According to the exception code, the flatten function yields the offending list. The list is the original sublist and is not altered in any way, even though the exception didn't occur until processing the second element.

Unlike return, yield does not have to walk back up through the recursion. The value is returned (yielded) and the code freezes until the next iteration. For our faulted list, that is at the end of the function and the next time though we will wind back and begin processing the next element.

Does that help at all?
Reply
#5
Thanks for your help!
While i am trying my very best to understand the logic behind...but my question remains there.

Perhaps let me try to elaborate my concern in a more detailed way.

My understanding of the code is that the below recursion will keep going until i reach the 'flat' layer, i.e. 1

for sublist in nested:
            for element in flatten(sublist):
At this point, the 'flat' layer cannot be used in for again because it is an integer. So a TypeError takes places.

In order to capture it, we have the below code
except TypeError:
        yield nested
In this case, yield nested will return 1.

Then i dun understand what happens next. As you suggested, i thought the code freezes right there and gives us 1. I have to use next() to return the next 'flat' layer.

Nonetheless, it seems to me that it is the
yield element
that gives us 1 instead of
yield nested
.

Sorry for bothering you!
Reply
#6
My earlier description is wrong and you are correct. The exception code is how elements are selected for return. When you call flatten with an element instead of a list, the "sublist in nested" inside the recursion generates an exception. The exception yields a value back to the caller and that becomes "element" for the caller's "for element in flatten).

So this is not a generator, but a generator inside of a generator, or a generator inside a generator inside a generator.... Each generator yields values to the "calling" generator. Pretty cool.

Things became a lot clearer for me when I wrote the code like this:
def flatten(nested, indent=""):
    try:
        for sublist in nested:
            for element in flatten(sublist, indent+"+"):
                yield indent
                # yield element
    except TypeError:
        pass
        # print(indent, nested)
        yield nested
 
for e in flatten((1, 2, (3, 4, (5, 6), 7), 8, (9, 10)), '-'):
    # pass
    print(e)
When I switch the comments around to print from inside flatten and pass the return value I see the list with indents. This makes it obvious the exception code is called for each individual element and not for any sublists. Then comment out the print in flatten and print the return value and it is all '-' for each element, clearly demonstrating that the values returned to the caller come from the for loop yield, and that the values are yielded to the caller which yields to the caller all the way up to the first call (thus no "+"). So it kind of works like normal recursion in that values are all returned by the first/top function, but it also does the yield/freeze thing, freezing the entire recursion chain right after yielding the value (kind of a yield/unchain/yield/unchain...).

Whew! That is a bit of a brain twister.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  yield usage as statement or expression akbarza 5 752 Oct-23-2023, 11:43 AM
Last Post: Gribouillis
  Using list comprehension with 'yield' in function tester_V 5 1,176 Apr-02-2023, 06:31 PM
Last Post: tester_V
  Yield generator weird output Vidar567 8 3,198 Nov-23-2020, 10:59 PM
Last Post: deanhystad
  Trying to access next element using Generator(yield) in a Class omm 2 1,935 Oct-19-2020, 03:36 PM
Last Post: omm
  Yield statement question DPaul 6 2,449 Sep-26-2020, 05:18 PM
Last Post: DPaul
  does yield support variable args? Skaperen 0 1,643 Mar-03-2020, 02:44 AM
Last Post: Skaperen
  generator function that yield from a list buran 9 4,106 Jun-04-2019, 10:26 PM
Last Post: snippsat
  yield help chakox 5 3,208 Apr-13-2019, 09:42 PM
Last Post: chakox
  Multiple calls to Python interpreter embedded in C++ application yield segmentation f mmoelle1 0 2,797 Mar-21-2019, 08:54 PM
Last Post: mmoelle1
  1 == 1 seems to yield false zatlas1 3 2,547 Jan-07-2019, 01:49 PM
Last Post: Gribouillis

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020