Python Forum
Web Scraping Error : Not getting expected result
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Web Scraping Error : Not getting expected result
#2
from bs4 import BeautifulSoup
import requests
import pandas as pd

url = 'https://www.programmableweb.com/apis/directory'

api_dict = {}
api_no = 0
while True:
    response = requests.get(url)
    data = response.text
    soup = BeautifulSoup(data, 'html.parser')
    apis = soup.find_all('td',{'class':'views-field views-field-title col-md-3'})


    for api in apis:
        name = api.find('a').text
        api_no += 1 
        #print(name)
    url_tag = soup.find('a',{'title':'Go to next page'})
    if url_tag.get('href'):
        url = 'https://www.programmableweb.com' + url_tag.get('href')
        #print(url)
    else:
        break
    
print('Total APIs: ',api_no)
api_dict_df = pd.DataFrame.from_dict(api_dict, orient = 'index', columns = ['API name'])
api_dict_df.head()
api_dict_df.to_csv('api_detail.csv')
Reply


Messages In This Thread
RE: Web Scraping Error : Not getting expected result - by adminravi - Oct-08-2019, 03:46 AM

Possibly Related Threads…
Thread Author Replies Views Last Post
  I am scraping a web page but got an Error Sarmad54 3 1,505 Mar-02-2023, 08:20 PM
Last Post: Sarmad54
  error in code web scraping alexisbrunaux 5 3,905 Aug-19-2020, 02:31 AM
Last Post: alexisbrunaux
  error zomato scraping data syxzetenz 3 3,439 Jun-23-2020, 08:53 PM
Last Post: Gribouillis
  Web scraping error jithin123 0 2,478 Mar-22-2020, 08:13 PM
Last Post: jithin123
  Scraping data saving to DB error with Cursor cubangt 3 2,855 May-20-2019, 08:30 PM
Last Post: Yoriz
  Python Scraping Error ZenWoR 1 2,278 Sep-15-2018, 08:23 PM
Last Post: snippsat
  Error while scraping item price from online store mgtheboss 2 5,292 Jan-12-2018, 06:42 PM
Last Post: snippsat
  Error while scraping links with beautiful soup mgtheboss 4 8,513 Dec-22-2017, 12:41 PM
Last Post: mgtheboss

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020