Basically I have a BeautifulSoup 4 for loop that scans over all g classes in a website, so it looks like this:
If it helps, here is the full for loop:
for g in soup.find_all(class_='g'):For each g inside the loop basically I scan it for certain keywords, and if none are found I open up the link inside of the class and then scan that, then move to the next one. Now this process takes a while, since the for loop waits until the one before it is done before moving on to the next one. What I want to do is have them scan each g class at the same time, and not wait for the one before it. Does anyone know how to do this? I have played around with asyncio but I do not really understand it. I have also heard about aiohttp, but am unclear about that too.
If it helps, here is the full for loop:
for g in soup.find_all(class_='g'): webBlock = str(g).lower() ans1Tally = webBlock.count(ans1) ans2Tally = webBlock.count(ans2) ans3Tally = webBlock.count(ans3) ans1Found = True ans2Found = True ans3Found = True if ans1 in webBlock: ans1Score += ans1Tally else: ans1Found = False if ans2 in webBlock: ans2Score += ans2Tally else: ans2Found = False if ans3 in webBlock: ans3Score += ans3Tally else: ans3Found = False if (ans1Found == False) and (ans2Found == False) and (ans3Found == False): print("Searching Link!") try: searchLink = str(links[0]) if searchLink.endswith('pdf'): pass else: response2 = requests.get(searchLink) soup2 = BeautifulSoup(response2.text, 'lxml') for p in soup2.find_all('p'): extraBlock = str(p) extraAns1Tally = extraBlock.count(ans1) extraAns2tally = extraBlock.count(ans2) extraAns3Tally = extraBlock.count(ans3) if ans1.lower() in extraBlock.lower(): ans1Score += extraAns1Tally if ans2.lower() in extraBlock.lower(): ans2Score += extraAns2Tally if ans3.lower() in extraBlock.lower(): ans3Score += extraAns3Tally except: pass if len(links) > 0: links.pop(0) else: passThe line
searchLink = str(links[0])is pulling the first link out of the list, the links are added to it before the for loop is called. Thank you for reading!