Python Forum
Python beautifulsoup pagination error
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Python beautifulsoup pagination error
#1
I want to make a pagination but page is not changed
import requests 
from bs4 import BeautifulSoup as bs
from colorama import init 

init(autoreset=True) 

#Header parametreleri
header_param = {"User-Agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.162 Safari/537.36"}

#Site Urlsi
url = "https://eksisozluk.com/"

#Aranacak entry 
word = input("Aranacak kelime:")


r = requests.get(url+word,headers=header_param)

soup = bs(r.content,"lxml")

#Aranan konuya ait sayfa sayisini almak 
pages = soup.find("div",attrs={"class":"pager"}).get("data-pagecount")

#Sayfa sayisi 
num  = 1
sayfa = "?p="

for num in range(1,int(pages)+1):
	#Her sayfa icin ayri bir istek gonderiyoruz
	pageRequest = requests.get("https://eksisozluk.com/"+word+"?p="+str(num),headers=header_param)
	
	pageSource = bs(pageRequest.content,"lxml")
	
	print(pageRequest.url)
	#Entrylerin bulundugu ul tagindaki tum li elementlerini aliyoruz
	
	entryler = pageSource.find(id="entry-item-list").find_all("li")
	
	for entry in entryler:
		#Entry paylasan kisi
		name = entry.find(class_='entry-author').get_text(strip=True)
		#Entry icerigi
		content = entry.find(class_='content').get_text(strip=True)
		#Entry Paylasilma Zamani
		publish = entry.find(class_='entry-date').get_text(strip=True)
		
		#Bilgileri Ekrana Yaziyoruz
		print(name,content,publish,sep="\n")
		print("\n")
		
		
		
Reply
#2
what would your search word be?
comment out line 30, then
after line 29, add:
    newurl = f"https://eksisozluk.com/{word}?p={str(num)}"
    print(newurl)
    pageRequest = requests.get(newurl, headers=header_param)
what do you get?
does the newurl look proper?
Reply
#3
(Apr-08-2020, 03:06 AM)Larz60+ Wrote: what would your search word be?
comment out line 30, then
after line 29, add:
    newurl = f"https://eksisozluk.com/{word}?p={str(num)}"
    print(newurl)
    pageRequest = requests.get(newurl, headers=header_param)
what do you get?
does the newurl look proper?

Page number is changed but page content is not changed
Reply
#4
what would your search word be?
Reply
#5
(Apr-08-2020, 09:29 AM)Larz60+ Wrote: what would your search word be?
Thank you i solve the problem
Reply
#6
Please share how solved with the forum, so others may benefit.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Strange ModuleNotFound Error on BeautifulSoup for Python 3.11 Gaberson19 1 920 Jul-13-2023, 10:38 AM
Last Post: Gaurav_Kumar
  [Solved]Help with BeautifulSoup.getText() Error Extra 5 3,642 Jan-19-2023, 02:03 PM
Last Post: prvncpa
  Python BeautifulSoup gives unusable text? dggo666 0 1,405 Oct-29-2021, 05:12 AM
Last Post: dggo666
  BeautifulSoup pagination using href rhat398 1 2,355 Jun-30-2021, 10:55 AM
Last Post: snippsat
  Python BeautifulSoup IndexError: list index out of range rhat398 1 6,163 May-28-2021, 09:09 PM
Last Post: Daring_T
  Python 3.9 : BeautifulSoup: 'NoneType' object has no attribute 'text' fudgemasterultra 1 8,816 Mar-03-2021, 09:40 AM
Last Post: Larz60+
  Error with NumPy, BeautifulSoup when using pip tsurubaso 7 5,161 Oct-20-2020, 04:34 PM
Last Post: tsurubaso
  Beautifulsoup doesn't scrape page (python 2.7) Hikki 0 1,951 Aug-01-2020, 05:54 PM
Last Post: Hikki
  Pagination prejni 2 2,361 Nov-18-2019, 10:45 AM
Last Post: alekson
  Scrapy Javascript Pagination (next_page) nazmulfinance 2 2,987 Nov-18-2019, 01:01 AM
Last Post: nazmulfinance

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020