May-21-2017, 08:22 AM
Well I have a list of URLs and I want to iterate over it so do get the pages as fast as it's possible.
I know that there is async for loop but can't get it how it works
Basically this is what I want
I have changed the code so many times and now it gives me an error and I don't even know what caused it.
Can't get this async stuff very well yet
I've tried to subclass the list as I saw it in some web pages so to get an object with __aiter__
Didn't work
I've tried to yielding each list element.
I know that there is async for loop but can't get it how it works
Basically this is what I want
# urls async for link in urls: print('{},{}'.format(await get_email(link))) # this is simplified. I am doing something else # get_email async def get_email(link): page = await fetch(link) soup = BeautifulSoup(page, 'lxml') name = soup.find('div', class_='MProwD').text.strip().lower().title() try: email = soup.find('div', class_='MPinfo').find_all('a')[-1]['href'].split(':')[1].strip() except: email = 'Unknown' return name, email #fetch async def fetch(url): async with aiohttp.ClientSession() as session: async with session.get(url) as response: return response.read()Until there wasn't an error I have not noticed performance difference from the regular program.
I have changed the code so many times and now it gives me an error and I don't even know what caused it.
Can't get this async stuff very well yet
I've tried to subclass the list as I saw it in some web pages so to get an object with __aiter__
Didn't work
I've tried to yielding each list element.
def list_gen(l): i = 0 try: yield l[i] i += 1 except StopIteration: return