Python beautifulsoup pagination error - Printable Version +- Python Forum (https://python-forum.io) +-- Forum: Python Coding (https://python-forum.io/forum-7.html) +--- Forum: Web Scraping & Web Development (https://python-forum.io/forum-13.html) +--- Thread: Python beautifulsoup pagination error (/thread-25682.html) |
Python beautifulsoup pagination error - The61 - Apr-08-2020 I want to make a pagination but page is not changed import requests from bs4 import BeautifulSoup as bs from colorama import init init(autoreset=True) #Header parametreleri header_param = {"User-Agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.162 Safari/537.36"} #Site Urlsi url = "https://eksisozluk.com/" #Aranacak entry word = input("Aranacak kelime:") r = requests.get(url+word,headers=header_param) soup = bs(r.content,"lxml") #Aranan konuya ait sayfa sayisini almak pages = soup.find("div",attrs={"class":"pager"}).get("data-pagecount") #Sayfa sayisi num = 1 sayfa = "?p=" for num in range(1,int(pages)+1): #Her sayfa icin ayri bir istek gonderiyoruz pageRequest = requests.get("https://eksisozluk.com/"+word+"?p="+str(num),headers=header_param) pageSource = bs(pageRequest.content,"lxml") print(pageRequest.url) #Entrylerin bulundugu ul tagindaki tum li elementlerini aliyoruz entryler = pageSource.find(id="entry-item-list").find_all("li") for entry in entryler: #Entry paylasan kisi name = entry.find(class_='entry-author').get_text(strip=True) #Entry icerigi content = entry.find(class_='content').get_text(strip=True) #Entry Paylasilma Zamani publish = entry.find(class_='entry-date').get_text(strip=True) #Bilgileri Ekrana Yaziyoruz print(name,content,publish,sep="\n") print("\n") RE: Python beautifulsoup pagination error - Larz60+ - Apr-08-2020 what would your search word be? comment out line 30, then after line 29, add: newurl = f"https://eksisozluk.com/{word}?p={str(num)}" print(newurl) pageRequest = requests.get(newurl, headers=header_param)what do you get? does the newurl look proper? RE: Python beautifulsoup pagination error - The61 - Apr-08-2020 (Apr-08-2020, 03:06 AM)Larz60+ Wrote: what would your search word be? Page number is changed but page content is not changed RE: Python beautifulsoup pagination error - Larz60+ - Apr-08-2020 what would your search word be? RE: Python beautifulsoup pagination error - The61 - Apr-09-2020 (Apr-08-2020, 09:29 AM)Larz60+ Wrote: what would your search word be?Thank you i solve the problem RE: Python beautifulsoup pagination error - Larz60+ - Apr-09-2020 Please share how solved with the forum, so others may benefit. |