Python Forum
Scraping not moving to the next pages in a website
Thread Rating:
  • 1 Vote(s) - 5 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Scraping not moving to the next pages in a website
#1
I am trying to scrape a website with all the succeeding pages in it. But after the second page, it is repeating itself indefinitely with output from just the second page. What went wrong here? I am not getting any error message. It is continuously scraping just the second page.

from bs4 import BeautifulSoup
import requests
import pandas as pd
url="https://www.programmableweb.com/category/all/apis"
while True:
    response= requests.get(url)
    response    
    data=response.text
    soup= BeautifulSoup(data,'html.parser') 
    apis=soup.find_all('tr',{"class":["odd","even"]})

    for api in apis:
        name_tag= api.find('td',{"class":"views-field views-field-pw-version-title"})
        name=name_tag.text if name_tag else 'na'
        des_tag=api.find('td',{'class':'views-field views-field-search-api-excerpt views-field-field-api-description hidden-xs visible-md visible-sm col-md-8'})
        des=des_tag.text if des_tag else 'na'
        category_tag=api.find('td',{'class':'views-field views-field-field-article-primary-category'})
        category=category_tag.text if category_tag else 'na'
        link_tag= api.find('a',{'class':'views-field views-field-pw-version-title'})
        link=link_tag.get('href') if link_tag else 'na'
        print('Name:',name,'\nDescription:', des ,'\nCategory:', category ,'\nLink:', link)
    url_tag=soup.find('a',{'title':'Go to next page'})
    if url_tag.get('href'):
        url=url+url_tag.get('href')
    else:
        break
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Need help opening pages when web scraping templeowls 1 273 Feb-29-2024, 06:45 PM
Last Post: snippsat
  web scraping for new additions/modifed website? kingoman123 4 2,184 Apr-14-2022, 04:46 PM
Last Post: snippsat
  Scraping lender data from Ren Ren Dai website using Python. I will pay for that 200$ Hafedh_2021 1 2,724 May-18-2021, 08:41 PM
Last Post: snippsat
  Scraping all website text using Python MKMKMKMK 1 2,052 Nov-26-2020, 10:35 PM
Last Post: Larz60+
  scraping multiple pages from table bandar 1 2,650 Jun-27-2020, 10:43 PM
Last Post: Larz60+
  Scraping a Website (HELP) LearnPython2 1 1,709 May-08-2020, 03:20 PM
Last Post: Larz60+
  Scraping Multiple Pages mbadatanut 1 4,178 May-08-2020, 02:30 AM
Last Post: Larz60+
  scraping from a website that hides source code PIWI_Protein 1 1,938 Mar-27-2020, 05:08 PM
Last Post: Larz60+
  Scrapping javascript website with Selenium where pages randomly fail to load JuanJuan 14 7,062 Dec-27-2019, 12:32 PM
Last Post: JuanJuan
  Random Loss of Control of Website When Scraping bmccollum 0 1,490 Aug-30-2019, 04:04 AM
Last Post: bmccollum

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020