Python Forum
pagination for non standarded pages
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
pagination for non standarded pages
#1
hi guys,

i am learning scraping and i currently i ve stopped in point of doing pagination in script below:

from bs4 import BeautifulSoup
import requests
import pandas as pd
import re
import urllib

headers = {
    'Sec-Fetch-Mode': 'cors',
    'Referer': 'https://www.pararius.com/apartments/amsterdam',
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36',
    'Content-Type': 'text/plain',
}

data = '{"tags":[{"sizes":[{"width":728,"height":90},{"width":970,"height":250}],"primary_size":{"width":728,"height":90},"ad_types":["banner"],"uuid":"5f5a2718d3aa6d","id":11247563,"allow_smaller_sizes":false,"use_pmt_rule":false,"prebid":true,"disable_psa":true},{"sizes":[{"width":728,"height":90},{"width":970,"height":250}],"primary_size":{"width":728,"height":90},"ad_types":["banner"],"uuid":"66526a063a1a8c","id":11247564,"allow_smaller_sizes":false,"use_pmt_rule":false,"prebid":true,"disable_psa":true}],"sdk":{"source":"pbjs","version":"2.19.0-pre"},"gdpr_consent":{"consent_string":"BOmDsv2OmDsv2BQABBENCN-AAAAmd7_______9______5uz_Ov_v_f__33e8__9v_l_7_-___u_-3zd4-_1vf99yfm1-7etr3tp_87ues2_Xur__59__3z3_9phPrsk89ryw","consent_required":true},"referrer_detection":{"rd_ref":"https%3A%2F%2Fwww.pararius.com%2Fapartments%2Famsterdam","rd_top":true,"rd_ifs":1,"rd_stk":"https%3A%2F%2Fwww.pararius.com%2Fapartments%2Famsterdam,https%3A%2F%2Fwww.pararius.com%2Fapartments%2Famsterdam"}}'

page_number = 2
page = 'https://www.pararius.com/apartments/amsterdam/page-' + str(page_number)
    
r = requests.get(page, headers=headers, data=data)
content = (r.text)
soup = BeautifulSoup(content, 'html.parser')



for section in soup.find_all(class_='property-list-item-container'):
    dlink = section.find('a').get('href')
    type = section.find('span', {'class': 'type'}).text
    neighborhood = section.find('a').text.strip().split()[1]
    size = section.find('li', {'class': 'surface'}).text.strip().split()[0]
    bedrooms = section.find('li', {'class': 'surface'}).text.strip().split()[2]
    furniture = section.find('li', {'class': 'surface'}).text.strip().split()[4]
    if furniture == 'upholstered':
        furniture = "Unfurnished"
    elif furniture == 'furnished or upholstered':
        furniture = "Furnished & Unfurnished"
    availablefrom = size = section.find('li', {'class': 'surface'}).text.strip().split()[6]
    price = section.find('p', {'class': 'price '}).text.strip().split()[0]
    curr = "EUR" if "€" in price else "other"
    print(curr)
    break
I have to add that it might happend that result from the site has let's say 50 pages, and it can happen that it has 30 only... how to deal with it?
what should be my next step?

I would appreciate any kind of help/tip!
Reply


Messages In This Thread
pagination for non standarded pages - by zarize - Aug-30-2019, 12:53 PM
RE: pagination for non standarded pages - by buran - Aug-30-2019, 01:10 PM
RE: pagination for non standarded pages - by zarize - Aug-30-2019, 01:55 PM
RE: pagination for non standarded pages - by buran - Aug-30-2019, 02:23 PM
RE: pagination for non standarded pages - by zarize - Aug-30-2019, 04:04 PM
RE: pagination for non standarded pages - by buran - Aug-30-2019, 04:40 PM
RE: pagination for non standarded pages - by zarize - Sep-02-2019, 10:46 AM
RE: pagination for non standarded pages - by buran - Sep-02-2019, 10:52 AM
RE: pagination for non standarded pages - by zarize - Sep-02-2019, 12:15 PM
RE: pagination for non standarded pages - by buran - Sep-02-2019, 12:18 PM
RE: pagination for non standarded pages - by zarize - Sep-02-2019, 12:23 PM
RE: pagination for non standarded pages - by buran - Sep-02-2019, 12:24 PM
RE: pagination for non standarded pages - by zarize - Sep-02-2019, 12:35 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
  BeautifulSoup pagination using href rhat398 1 2,442 Jun-30-2021, 10:55 AM
Last Post: snippsat
  Python beautifulsoup pagination error The61 5 3,515 Apr-09-2020, 09:17 PM
Last Post: Larz60+
  Pagination prejni 2 2,429 Nov-18-2019, 10:45 AM
Last Post: alekson
  Scrapy Javascript Pagination (next_page) nazmulfinance 2 3,074 Nov-18-2019, 01:01 AM
Last Post: nazmulfinance
  Python - Scrapy Javascript Pagination (next_page) Baggelhsk95 3 10,053 Oct-08-2018, 01:20 PM
Last Post: stranac
  Filtering and pagination garynobles 0 38,508 Jun-14-2018, 08:11 PM
Last Post: garynobles
  BeautifulSoup and pagination. Mike Ru 1 7,933 Sep-22-2017, 10:15 AM
Last Post: snippsat

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020