Python Forum
pagination for non standarded pages
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
pagination for non standarded pages
#7
thank you :)

#edit

can i ask you how to structure it properly so python can read 'num_pages'? In case i am placing range 1, 12 it is working i mean it returns direct link from 1 to 11.

from bs4 import BeautifulSoup
import requests
import pandas as pd
import re
import urllib

headers = {
    'Sec-Fetch-Mode': 'cors',
    'Referer': 'https://www.pararius.com/apartments/amsterdam',
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36',
    'Content-Type': 'text/plain',
}

data = '{"tags":[{"sizes":[{"width":728,"height":90},{"width":970,"height":250}],"primary_size":{"width":728,"height":90},"ad_types":["banner"],"uuid":"5f5a2718d3aa6d","id":11247563,"allow_smaller_sizes":false,"use_pmt_rule":false,"prebid":true,"disable_psa":true},{"sizes":[{"width":728,"height":90},{"width":970,"height":250}],"primary_size":{"width":728,"height":90},"ad_types":["banner"],"uuid":"66526a063a1a8c","id":11247564,"allow_smaller_sizes":false,"use_pmt_rule":false,"prebid":true,"disable_psa":true}],"sdk":{"source":"pbjs","version":"2.19.0-pre"},"gdpr_consent":{"consent_string":"BOmDsv2OmDsv2BQABBENCN-AAAAmd7_______9______5uz_Ov_v_f__33e8__9v_l_7_-___u_-3zd4-_1vf99yfm1-7etr3tp_87ues2_Xur__59__3z3_9phPrsk89ryw","consent_required":true},"referrer_detection":{"rd_ref":"https%3A%2F%2Fwww.pararius.com%2Fapartments%2Famsterdam","rd_top":true,"rd_ifs":1,"rd_stk":"https%3A%2F%2Fwww.pararius.com%2Fapartments%2Famsterdam,https%3A%2F%2Fwww.pararius.com%2Fapartments%2Famsterdam"}}'


for n in range(1, 12):
    page = 'https://www.pararius.com/apartments/amsterdam/page-' + str(n)
    print(num_pages)
    
r = requests.get(page, headers=headers, data=data)
content = (r.text)
soup = BeautifulSoup(content, 'html.parser')


#pagination- find max pages

page1 = soup.find('ul', {'class': 'pagination'})
pages = page1.find_all('li')
last_page = pages[-3]
num_pages = last_page.find('a').text
#print(num_pages)


for section in soup.find_all(class_='property-list-item-container'):
    dlink = section.find('a').get('href')
    type = section.find('span', {'class': 'type'}).text
    neighborhood = section.find('a').text.strip().split()[1]
    size = section.find('li', {'class': 'surface'}).text.strip().split()[0]
    bedrooms = section.find('li', {'class': 'surface'}).text.strip().split()[2]
    furniture = section.find('li', {'class': 'surface'}).text.strip().split()[4]
    if furniture == 'upholstered':
        furniture = "Unfurnished"
    elif furniture == 'furnished or upholstered':
        furniture = "Furnished & Unfurnished"
    availablefrom = size = section.find('li', {'class': 'surface'}).text.strip().split()[6]
    price = section.find('p', {'class': 'price '}).text.strip().split()[0]
    curr = "EUR" if "€" in price else "other"

    break
What i am doing wrong?
I wanted to do
for n in range(1, num_pages):
but it doesnt work

I was trying to replace code upper/lower but i am doing something wrong..
Reply


Messages In This Thread
pagination for non standarded pages - by zarize - Aug-30-2019, 12:53 PM
RE: pagination for non standarded pages - by buran - Aug-30-2019, 01:10 PM
RE: pagination for non standarded pages - by zarize - Aug-30-2019, 01:55 PM
RE: pagination for non standarded pages - by buran - Aug-30-2019, 02:23 PM
RE: pagination for non standarded pages - by zarize - Aug-30-2019, 04:04 PM
RE: pagination for non standarded pages - by buran - Aug-30-2019, 04:40 PM
RE: pagination for non standarded pages - by zarize - Sep-02-2019, 10:46 AM
RE: pagination for non standarded pages - by buran - Sep-02-2019, 10:52 AM
RE: pagination for non standarded pages - by zarize - Sep-02-2019, 12:15 PM
RE: pagination for non standarded pages - by buran - Sep-02-2019, 12:18 PM
RE: pagination for non standarded pages - by zarize - Sep-02-2019, 12:23 PM
RE: pagination for non standarded pages - by buran - Sep-02-2019, 12:24 PM
RE: pagination for non standarded pages - by zarize - Sep-02-2019, 12:35 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
  BeautifulSoup pagination using href rhat398 1 2,505 Jun-30-2021, 10:55 AM
Last Post: snippsat
  Python beautifulsoup pagination error The61 5 3,593 Apr-09-2020, 09:17 PM
Last Post: Larz60+
  Pagination prejni 2 2,508 Nov-18-2019, 10:45 AM
Last Post: alekson
  Scrapy Javascript Pagination (next_page) nazmulfinance 2 3,183 Nov-18-2019, 01:01 AM
Last Post: nazmulfinance
  Python - Scrapy Javascript Pagination (next_page) Baggelhsk95 3 10,203 Oct-08-2018, 01:20 PM
Last Post: stranac
  Filtering and pagination garynobles 0 44,358 Jun-14-2018, 08:11 PM
Last Post: garynobles
  BeautifulSoup and pagination. Mike Ru 1 7,982 Sep-22-2017, 10:15 AM
Last Post: snippsat

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020