Web Crawler help

[metulburr]

It would be something along the lines of....

#import requests
from selenium import webdriver
#from selenium.webdriver.common.keys import Keys
from bs4 import BeautifulSoup
#open('output.csv', 'w').close()
import re
import time

#browser = webdriver.Firefox()
browser = webdriver.Chrome('/home/metulburr/chromedriver')
browser.set_window_position(0,0)
  
def fundaSpider(max_pages):
    page = 1
    while page <= max_pages:

        url = 'http://www.funda.nl/koop/rotterdam/p{}'.format(page)
        browser.get(url)
        time.sleep(1)#normal delay to allow browser to load content
        raw_input('Press Enter after bypassing Captcha')
        soup = BeautifulSoup(browser.page_source, 'html.parser')
        ads = soup.find_all('li', {'class': 'search-result'})
        for ad in ads:
            title = ad.find('h3')
            print(title)
        page += 1
        
fundaSpider(2)

 couple of things.....
- i used chrome as i dont have the firefox driver on my system.
- i repositioned the window to the top left, because i have dual monitors and it puts it on my TV when i run it if i dont put it there
- you can use PhantomJS to keep it in the background instead of popping up a browser.
- i kept trying to bring up the captcha, and this time i didnt get one, so i didnt know exactly what occurs after the captcha is entered, or how often it occurs.....ITs just an input to stop the program until the captcha is entered, currently is placed for every page. This is assuming the captchas gets triggered on every page. If you only get the captcha on the first time, you can move the input out of the while loop, however you are going to need to a do a captcha trigger get()
such as....

browser.get(url) #trigger captcha by going to the first page
raw_input() #halts program until after captcha is entered
...
while page <= max_pages:
    ...
    browser.get(url) # now go to true page as captcha will not be triggered
    ...