Oct-19-2019, 12:39 PM
Well i did some debugging and found when executing from heroku , the page is not getting scrolled. Added new method to scroll. Can somebody please suggest.
def scrollDownUp(driver,elm): last_height = driver.execute_script("return document.body.scrollHeight") while True: # Scroll down to almost the bottom of the page driver.execute_script("window.scrollTo(0, (document.body.scrollHeight-600));") # Time Taken to Load the page time.sleep(7) # Scrolling Up & Down to load more Data elm.send_keys(Keys.HOME) time.sleep(4) elm.send_keys(Keys.END) time.sleep(3) print('last height ', last_height) # Calculate the new scrolling height and then compare it to old height new_height = driver.execute_script("return document.body.scrollHeight") print('new height',new_height) if new_height == last_height: break last_height = new_heightlocal machine
Output:last height 3531
new height 6601
last height 6601
new height 7825
last height 7825
new height 7825
38 - 38 - 38 - 38 - 38
from heroku consoleOutput:D:\Scapper\webscapper>heroku run python FlightScapper.py
Running python FlightScapper.py on ⬢ webscapper... up, run.3818 (Free)
.......
last height 2606
new height 2606
0 - 0 - 0 - 0 - 0
...............best flight...............
Empty DataFrame
Columns: []
Index: []
called made