Well, I am not a programmer and lots of programming concepts are unknown for me be but this is a process after all. Slowly but I have some progress.
It seems like Asyncio is becoming the new religion which could make a huge difference how we write code. Sort of...
I admit that I still don't understand how it works. It is not a small library and even getting the basics of all of that is not easy for me.
Half a year ago I wrote a script to get a bunch of emails from a web site. It was relatively easy but I've wanted to make some improvements. So I have decided to scrape all pages concurrently. It was not an easy for me ( first time doing that ) and I ended up with concurrent.futures.ProcessPoolExecutor.
Now, I am going to try a different approach. Using the asyncio standard library.
What is basically I need is to open each URL from a list of some web pages address and get what I want.
After this prelude lets see how all this works. I got it finally
Asyncio can be used in for loop which sounds wonderful but I have missed something when I tried it the first time.
The sintax:
Not exactly.
In order to be possible, the iterable have to be an Asynchronous Iterable. According to PEP492 it must implement two methods. __aiter__ which returns asynchronous iterator object and __anext__ which returns awaitable object. Alright! Awaitable?
It's time to try again.
What I have learned today and made it work? Well, it appears that you can't use async for statement outside of a async defined function. So it must be implemented into something like that:
It seems like Asyncio is becoming the new religion which could make a huge difference how we write code. Sort of...
I admit that I still don't understand how it works. It is not a small library and even getting the basics of all of that is not easy for me.
Half a year ago I wrote a script to get a bunch of emails from a web site. It was relatively easy but I've wanted to make some improvements. So I have decided to scrape all pages concurrently. It was not an easy for me ( first time doing that ) and I ended up with concurrent.futures.ProcessPoolExecutor.
Now, I am going to try a different approach. Using the asyncio standard library.
What is basically I need is to open each URL from a list of some web pages address and get what I want.
After this prelude lets see how all this works. I got it finally
Asyncio can be used in for loop which sounds wonderful but I have missed something when I tried it the first time.
The sintax:
async for element in iterable: print(element)Pretty awesome, huh? Simple enough.
Not exactly.
async for char in "So Long, and Thanks for All the Fish": print(char)just doesn't work SyntaxError... Realy?
In order to be possible, the iterable have to be an Asynchronous Iterable. According to PEP492 it must implement two methods. __aiter__ which returns asynchronous iterator object and __anext__ which returns awaitable object. Alright! Awaitable?
Quote:Still don't get it but let's give it a try.
- A native coroutine object returned from a native coroutine function.
- A generator-based coroutine object returned from a function decorated with types.coroutine().
- An object with an __await__ method returning an iterator.
class Aiter : def __init__(self, iterable): self.iter_ = iter(iterable) async def __aiter__(self): return self async def __anext__(self): await asyncio .sleep(0) try: object = next(self.iter_) except StopIteration: raise StopAsyncIteration # :-) PEP492 - "To stop iteration __anext__ must raise a StopAsyncIteration exception" return objectThis should create an Async iterable. It has the two important methods - __aiter__ and __anext__ - and it raises the proper exception to stop the iteration.
It's time to try again.
async for char in Aiter ('So Long, and Thanks for All the Fish'): print(char)SyntaxError! Again... It should work. Back in the time, this made me give up.
What I have learned today and made it work? Well, it appears that you can't use async for statement outside of a async defined function. So it must be implemented into something like that:
async def foo(text): await asyncio .sleep(0) # this is needed in order to made it coroutine function async for char in Aiter ('So Long, and Thanks for All the Fish'): print(char)But if you run it as it is, it still doesn't work.
>>> foo('So Long, and Thanks for All the Fish') <coroutine object foo at 0x7f0886affaf0>One more thing you have to do is to create an event loop and run it. Because of all these await statements.
>>> text = 'So Long, and Thanks for All the Fish' >>> loop = asyncio.get_event_loop() >>> loop.run_until_complete(foo(text))Finally, we get some something.
Output:S
o
L
o
n
g
,
a
n
d
T
h
a
n
k
s
f
o
r
A
l
l
t
h
e
F
i
s
h
There was no BeautifulSoup and web pages scrapping but simplified example how all of this works. There is a much more but I will try it when I need it. This could be so powerful and I can't even imagine how much. You take the results you need as they are available without blocking the executing of the other part of the code. I am assuming that all code has to be async/await and run the event loop after if __name__ == '__main__' statement.