Looping problem using Selenium - Printable Version +- Python Forum (https://python-forum.io) +-- Forum: Python Coding (https://python-forum.io/forum-7.html) +--- Forum: Web Scraping & Web Development (https://python-forum.io/forum-13.html) +--- Thread: Looping problem using Selenium (/thread-12942.html) |
Looping problem using Selenium - smk90 - Sep-19-2018 Hi, I am trying to use Python with Selenium recursively loop through a list and provide a cumulative value of a metric for each 'match' in a list. Although my issue is I am getting the same (incorrect) value for each of the items in the list. Any tips where this is going wrong? I am pretty sure it is the for data in soup.findAll('div',{'class':'ml1-SoccerStatsBar '}): line but not sure why. #use Selenium & Beautiful Soup from selenium import webdriver import time from bs4 import BeautifulSoup #define URL/driver my_url = "https://www.bet365.com/#/IP/" driver = webdriver.Edge() driver.get(my_url) #allow a sleep of 10 seconds time.sleep(10) #parse the page pSource= driver.page_source soup = BeautifulSoup(pSource, "html.parser") #containers tag - per match containers = soup.findAll("div", {"class": "ipn-TeamStack "}) for container in containers: match = container.find_all('div') texts = [div.text for div in match] #Total Match Shots cumul_match_shots = 0 for data in soup.findAll('div',{'class':'ml1-SoccerStatsBar '}): for result in data.find_all('span'): a = result.text if len(a) > 0: cumul_match_shots += int(a) #print out values print(texts) print(cumul_match_shots) #close the webpage driver.close() |