Apr-22-2019, 08:34 PM
I use a combination of Selenium and BeautifulSoup for a couple of reasons.
In this particular instance, once you have expanded the JavaScript, there's no need for selenium anymore.
Beautiful Soup is the best way to traverse the DOM and scrape the data, so after all JavaScript has been expanded, I use Beautiful Soup to grab the desired data. It speeds up the process (which, in some instances, can be a considerable amount of time), and makes it easier to grab other data if needed later on.
In this particular instance, once you have expanded the JavaScript, there's no need for selenium anymore.
Beautiful Soup is the best way to traverse the DOM and scrape the data, so after all JavaScript has been expanded, I use Beautiful Soup to grab the desired data. It speeds up the process (which, in some instances, can be a considerable amount of time), and makes it easier to grab other data if needed later on.