Basically I have a BeautifulSoup 4 for loop that scans over all g classes in a website, so it looks like this:
If it helps, here is the full for loop:
for g in soup.find_all(class_='g'):For each g inside the loop basically I scan it for certain keywords, and if none are found I open up the link inside of the class and then scan that, then move to the next one. Now this process takes a while, since the for loop waits until the one before it is done before moving on to the next one. What I want to do is have them scan each g class at the same time, and not wait for the one before it. Does anyone know how to do this? I have played around with asyncio but I do not really understand it. I have also heard about aiohttp, but am unclear about that too.
If it helps, here is the full for loop:
for g in soup.find_all(class_='g'):
webBlock = str(g).lower()
ans1Tally = webBlock.count(ans1)
ans2Tally = webBlock.count(ans2)
ans3Tally = webBlock.count(ans3)
ans1Found = True
ans2Found = True
ans3Found = True
if ans1 in webBlock:
ans1Score += ans1Tally
else:
ans1Found = False
if ans2 in webBlock:
ans2Score += ans2Tally
else:
ans2Found = False
if ans3 in webBlock:
ans3Score += ans3Tally
else:
ans3Found = False
if (ans1Found == False) and (ans2Found == False) and (ans3Found == False):
print("Searching Link!")
try:
searchLink = str(links[0])
if searchLink.endswith('pdf'):
pass
else:
response2 = requests.get(searchLink)
soup2 = BeautifulSoup(response2.text, 'lxml')
for p in soup2.find_all('p'):
extraBlock = str(p)
extraAns1Tally = extraBlock.count(ans1)
extraAns2tally = extraBlock.count(ans2)
extraAns3Tally = extraBlock.count(ans3)
if ans1.lower() in extraBlock.lower():
ans1Score += extraAns1Tally
if ans2.lower() in extraBlock.lower():
ans2Score += extraAns2Tally
if ans3.lower() in extraBlock.lower():
ans3Score += extraAns3Tally
except:
pass
if len(links) > 0:
links.pop(0)
else:
passThe line searchLink = str(links[0])is pulling the first link out of the list, the links are added to it before the for loop is called. Thank you for reading!