Python Forum

Basically I have a BeautifulSoup 4 for loop that scans over all g classes in a website, so it looks like this:

for g in soup.find_all(class_='g'):

For each g inside the loop basically I scan it for certain keywords, and if none are found I open up the link inside of the class and then scan that, then move to the next one. Now this process takes a while, since the for loop waits until the one before it is done before moving on to the next one. What I want to do is have them scan each g class at the same time, and not wait for the one before it. Does anyone know how to do this? I have played around with asyncio but I do not really understand it. I have also heard about aiohttp, but am unclear about that too.

If it helps, here is the full for loop:

for g in soup.find_all(class_='g'):
    
    webBlock = str(g).lower()
    
    ans1Tally = webBlock.count(ans1)
    ans2Tally = webBlock.count(ans2)
    ans3Tally = webBlock.count(ans3)

    ans1Found = True
    ans2Found = True
    ans3Found = True

    if 	ans1 in webBlock:

        ans1Score += ans1Tally

    else:

        ans1Found = False
        
    if ans2 in webBlock:

        ans2Score += ans2Tally

    else:

        ans2Found = False

    if ans3 in webBlock:

        ans3Score += ans3Tally

    else:

        ans3Found = False
        
    if (ans1Found == False) and (ans2Found == False) and (ans3Found == False):

        print("Searching Link!")

        try:

            searchLink = str(links[0])

            if searchLink.endswith('pdf'):
                pass

            else:

                response2 = requests.get(searchLink)
                soup2 = BeautifulSoup(response2.text, 'lxml')

                for p in soup2.find_all('p'):

                    extraBlock = str(p)

                    extraAns1Tally = extraBlock.count(ans1)
                    extraAns2tally = extraBlock.count(ans2)
                    extraAns3Tally = extraBlock.count(ans3)

                    if ans1.lower() in extraBlock.lower():

                        ans1Score += extraAns1Tally

                    if ans2.lower() in extraBlock.lower():

                        ans2Score += extraAns2Tally

                    if ans3.lower() in extraBlock.lower():

                        ans3Score += extraAns3Tally

        except:

            pass

    if len(links) > 0:

        links.pop(0)

    else:

        pass

The line

searchLink = str(links[0])

is pulling the first link out of the list, the links are added to it before the for loop is called. Thank you for reading!

You maybe looking for multithreading programming.

In this book you can find some guidelines to start with Multithreading in Python - http://books.goalkicker.com/PythonBook/ - Chapter 35: Multithreading

Hope it helps.. good luck!

See if this can help: https://python-forum.io/Thread-Async-for-loop