Async for loop

[wavic]

Well, I am not a programmer and lots of programming concepts are unknown for me be but this is a process after all. Slowly but I have some progress.

It seems like Asyncio is becoming the new religion which could make a huge difference how we write code. Sort of...
I admit that I still don't understand how it works. It is not a small library and even getting the basics of all of that is not easy for me.

Half a year ago I wrote a script to get a bunch of emails from a web site. It was relatively easy but I've wanted to make some improvements. So I have decided to scrape all pages concurrently. It was not an easy for me ( first time doing that ) and I ended up with concurrent.futures.ProcessPoolExecutor.

Now, I am going to try a different approach. Using the asyncio standard library.
What is basically I need is to open each URL from a list of some web pages address and get what I want.

After this prelude lets see how all this works. I got it finally

Asyncio can be used in for loop which sounds wonderful but I have missed something when I tried it the first time.
The sintax:

async for element in iterable:
    print(element)

Pretty awesome, huh?  Simple enough.
Not exactly.

async for char in "So Long, and Thanks for All the Fish":
    print(char)

just doesn't work SyntaxError... Realy?

In order to be possible, the iterable have to be an Asynchronous Iterable. According to PEP492 it must implement two methods. __aiter__ which returns asynchronous iterator object and __anext__ which returns awaitable object. Alright! Awaitable?

 

• A native coroutine object returned from a native coroutine function.
  
• A generator-based coroutine object returned from a function decorated with types.coroutine().
  
• An object with an __await__ method returning an iterator.
  

 

Still don't get it but let's give it a try.

class Aiter :
    def __init__(self, iterable):
        self.iter_ = iter(iterable)

    async def __aiter__(self):
        return self

    async def __anext__(self):
        await asyncio .sleep(0)
        try:
            object = next(self.iter_)
        except StopIteration:
            raise StopAsyncIteration # :-) PEP492 - "To stop iteration __anext__ must raise a StopAsyncIteration exception"

        return object

This should create an Async iterable. It has the two important methods - __aiter__ and __anext__ - and it raises the proper exception to stop the iteration.
It's time to try again.

async for char in Aiter ('So Long, and Thanks for All the Fish'):
    print(char)

SyntaxError! Again... It should work. Back in the time, this made me give up.

What I have learned today and made it work? Well, it appears that you can't use async for statement outside of a async defined function. So it must be implemented into something like that:

async def foo(text):
    await asyncio .sleep(0) # this is needed in order to made it coroutine function
    async for char in Aiter ('So Long, and Thanks for All the Fish'):
        print(char)

But if you run it as it is, it still doesn't work.

 >>> foo('So Long, and Thanks for All the Fish')
<coroutine object foo at 0x7f0886affaf0>

One more thing you have to do is to create an event loop and run it. Because of all these await statements.

>>> text = 'So Long, and Thanks for All the Fish'
>>> loop = asyncio.get_event_loop()

>>> loop.run_until_complete(foo(text)) 

Finally, we get some something.

Output:
S
o
 
L
o
n
g
,
 
a
n
d
 
T
h
a
n
k
s
 
f
o
r
 
A
l
l
 
t
h
e
 
F
i
s
h

There was no BeautifulSoup and web pages scrapping but simplified example how all of this works. There is a much more but I will try it when I  need it. This could be so powerful and I can't even imagine how much. You take the results you need as they are available without blocking the executing of the other part of the code. I am assuming that all code has to be async/await and run the event loop after if __name__ == '__main__' statement.

[nilamo]

The event loop is the better way to run async functions, but it isn't the only way... https://python-forum.io/Thread-Exploring...ad-of-time

[micseydel]

Thanks for sharing this.

As a note about debugging problems, the fact that it was a syntax error initially meant that adding dunder methods wouldn't solve the problem, because those are semantic errors. Syntax errors happen before any code actually runs, and code would have to run to define those methods.

[programmerguy]

Great thread on loops, if anyone else is interested I helped write a tutorial on python for loops some time ago. Check it out if interested :)

[DeaD_EyE]

You can replace this code:

class Aiter :
    def __init__(self, iterable):
        self.iter_ = iter(iterable)
 
    async def __aiter__(self):
        return self
 
    async def __anext__(self):
        await asyncio .sleep(0)
        try:
            object = next(self.iter_)
        except StopIteration:
            raise StopAsyncIteration # :-) PEP492 - "To stop iteration __anext__ must raise a StopAsyncIteration exception"
 
        return object

With this code:

async def aiter(msg):
    for char in msg:
        yield char

You can't use yield from in async function, but you can use yield to use async generator, which could be used with async for.