Python Forum

Full Version: How to make a for loop asynchronous?
You're currently viewing a stripped down version of our content. View the full version with proper formatting.
Basically I have a BeautifulSoup 4 for loop that scans over all g classes in a website, so it looks like this:
for g in soup.find_all(class_='g'):
For each g inside the loop basically I scan it for certain keywords, and if none are found I open up the link inside of the class and then scan that, then move to the next one. Now this process takes a while, since the for loop waits until the one before it is done before moving on to the next one. What I want to do is have them scan each g class at the same time, and not wait for the one before it. Does anyone know how to do this? I have played around with asyncio but I do not really understand it. I have also heard about aiohttp, but am unclear about that too.

If it helps, here is the full for loop:

for g in soup.find_all(class_='g'):
    webBlock = str(g).lower()
    ans1Tally = webBlock.count(ans1)
    ans2Tally = webBlock.count(ans2)
    ans3Tally = webBlock.count(ans3)

    ans1Found = True
    ans2Found = True
    ans3Found = True

    if 	ans1 in webBlock:

        ans1Score += ans1Tally


        ans1Found = False
    if ans2 in webBlock:

        ans2Score += ans2Tally


        ans2Found = False

    if ans3 in webBlock:

        ans3Score += ans3Tally


        ans3Found = False
    if (ans1Found == False) and (ans2Found == False) and (ans3Found == False):

        print("Searching Link!")


            searchLink = str(links[0])

            if searchLink.endswith('pdf'):


                response2 = requests.get(searchLink)
                soup2 = BeautifulSoup(response2.text, 'lxml')

                for p in soup2.find_all('p'):

                    extraBlock = str(p)

                    extraAns1Tally = extraBlock.count(ans1)
                    extraAns2tally = extraBlock.count(ans2)
                    extraAns3Tally = extraBlock.count(ans3)

                    if ans1.lower() in extraBlock.lower():

                        ans1Score += extraAns1Tally

                    if ans2.lower() in extraBlock.lower():

                        ans2Score += extraAns2Tally

                    if ans3.lower() in extraBlock.lower():

                        ans3Score += extraAns3Tally



    if len(links) > 0:



The line
searchLink = str(links[0])
is pulling the first link out of the list, the links are added to it before the for loop is called. Thank you for reading!
You maybe looking for multithreading programming.

In this book you can find some guidelines to start with Multithreading in Python - - Chapter 35: Multithreading

Hope it helps.. good luck!