Python Forum
Scrape script when deployed on heroku not giving expected outcomes
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Scrape script when deployed on heroku not giving expected outcomes
#2
Well i did some debugging and found when executing from heroku , the page is not getting scrolled. Added new method to scroll. Can somebody please suggest.

def scrollDownUp(driver,elm):
    last_height = driver.execute_script("return document.body.scrollHeight")

    while True:
        # Scroll down to almost the bottom of the page
        driver.execute_script("window.scrollTo(0, (document.body.scrollHeight-600));")

        # Time Taken to Load the page
        time.sleep(7)

        # Scrolling Up & Down to load more Data
        elm.send_keys(Keys.HOME)
        time.sleep(4)
        elm.send_keys(Keys.END)
        time.sleep(3)
        print('last height ', last_height)
        # Calculate the new scrolling height and then compare it to old height
        new_height = driver.execute_script("return document.body.scrollHeight")
        print('new height',new_height)
        if new_height == last_height:
            break
        last_height = new_height
local machine
Output:
last height 3531 new height 6601 last height 6601 new height 7825 last height 7825 new height 7825 38 - 38 - 38 - 38 - 38
from heroku console
Output:
D:\Scapper\webscapper>heroku run python FlightScapper.py Running python FlightScapper.py on ⬢ webscapper... up, run.3818 (Free) ....... last height 2606 new height 2606 0 - 0 - 0 - 0 - 0 ...............best flight............... Empty DataFrame Columns: [] Index: [] called made
Reply


Messages In This Thread
RE: Scrape script when deployed on heroku not giving expected outcomes - by alok001 - Oct-19-2019, 12:39 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
Star Leapcell: The Python-Friendly Alternative to Heroku + Airtable Hybrid IssacChan 1 477 Feb-01-2024, 06:00 AM
Last Post: Athi
  Deployed Spider on Heroku: How do I email downloaded files? JaneTan 2 1,576 Mar-24-2022, 08:31 AM
Last Post: JaneTan
  Heroku Error H10 jamesaarr 1 2,037 Oct-21-2021, 03:43 PM
Last Post: jamesaarr
  Importing Postgres Heroku from AWS S3 Drone4four 0 1,816 May-27-2021, 01:09 PM
Last Post: Drone4four
  Django project deployed to Heroku: Postgres security Drone4four 0 1,946 Mar-26-2021, 10:17 AM
Last Post: Drone4four
  scrape data 1 go to next page scrape data 2 and so on alkaline3 6 5,219 Mar-13-2020, 07:59 PM
Last Post: alkaline3
  flask app to save images locally when deployed on heroku not working Prince_Bhatia 1 5,286 Feb-20-2019, 11:59 PM
Last Post: snippsat
  Deploy flask app on Heroku Prince_Bhatia 0 4,044 Feb-20-2019, 09:05 AM
Last Post: Prince_Bhatia
  Hosting statistic tool on heroku with flask secure? Zoja 1 2,887 Oct-29-2018, 10:07 AM
Last Post: Zoja
  Scrape java script web site PythonHunger 6 4,144 Oct-25-2018, 05:59 AM
Last Post: PythonHunger

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020