Posts: 3
Threads: 1
Joined: Oct 2019
Oct-07-2019, 08:36 PM
(This post was last modified: Oct-08-2019, 03:50 AM by Larz60+.)
Hi,
I am a beginner in Python programming. Of late, I have started learning web scraping after completing python programming course from Udemy. I was trying to scrape this website but unfortunately could not get the csv file. Moreover, total number is also not printing. There maybe more errors. Can you please figure it out and help me with simple explanation. I am just a couple of days old in web scraping.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 |
from bs4 import BeautifulSoup
import requests
import pandas as pd
api_dict = {}
api_no = 0
while True:
response = requests.get(url)
data = response.text
soup = BeautifulSoup(data, 'html.parser')
apis = soup.find_all('td',{'class':'views-field views-field-title col-md-3'})
for api in apis:
name = api.find('a').text
api_no += 1
url_tag = soup.find('a',{'title':'Go to next page'})
if url_tag.get('href'):
else:
break
print('Total APIs: ',api_no)
api_dict_df = pd.DataFrame.from_dict(api_dict, orient = 'index', columns = ['API name'])
api_dict_df.head()
api_dict_df.to_csv('api_detail.csv')
|
Posts: 3
Threads: 1
Joined: Oct 2019
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 |
from bs4 import BeautifulSoup
import requests
import pandas as pd
api_dict = {}
api_no = 0
while True:
response = requests.get(url)
data = response.text
soup = BeautifulSoup(data, 'html.parser')
apis = soup.find_all('td',{'class':'views-field views-field-title col-md-3'})
for api in apis:
name = api.find('a').text
api_no += 1
url_tag = soup.find('a',{'title':'Go to next page'})
if url_tag.get('href'):
else:
break
print('Total APIs: ',api_no)
api_dict_df = pd.DataFrame.from_dict(api_dict, orient = 'index', columns = ['API name'])
api_dict_df.head()
api_dict_df.to_csv('api_detail.csv')
|
Posts: 12,030
Threads: 485
Joined: Sep 2016
Oct-08-2019, 03:52 AM
(This post was last modified: Oct-08-2019, 03:53 AM by Larz60+.)
This code will show what's available the group you're interested in (prints out all tr's and td's)
All you have to do is select the elements that you want, and then extract that data.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 |
from bs4 import BeautifulSoup
import requests
import pandas as pd
import sys
def scrape_page():
api_dict = {}
api_no = 0
response = requests.get(url)
if response.status_code == 200:
data = response.text
else:
print(f"Unable to fetch page, bad status: {response.status_code}")
sys.exit(-1)
soup = BeautifulSoup(data, 'html.parser')
tbody = soup.select('.views-table > tbody:nth-child(2)')[0]
trs = tbody.find_all('tr')
for n, tr in enumerate(trs):
tds = tr.find_all('td')
for n1, td in enumerate(tds):
print(f"\n--------------------- tr_{n}, td_{n1} ---------------------")
print(td)
|
Posts: 3
Threads: 1
Joined: Oct 2019
(Oct-08-2019, 03:52 AM)Larz60+ Wrote: This code will show what's available the group you're interested in (prints out all tr's and td's)
All you have to do is select the elements that you want, and then extract that data.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 |
from bs4 import BeautifulSoup
import requests
import pandas as pd
import sys
def scrape_page():
api_dict = {}
api_no = 0
response = requests.get(url)
if response.status_code == 200:
data = response.text
else:
print(f"Unable to fetch page, bad status: {response.status_code}")
sys.exit(-1)
soup = BeautifulSoup(data, 'html.parser')
tbody = soup.select('.views-table > tbody:nth-child(2)')[0]
trs = tbody.find_all('tr')
for n, tr in enumerate(trs):
tds = tr.find_all('td')
for n1, td in enumerate(tds):
print(f"\n--------------------- tr_{n}, td_{n1} ---------------------")
print(td)
|
Thank you very much for the solution. As you can understand, I am just a few days old in web scraping. Can you also suggest one good tutorial which I should learn from?
Regards,
Ravi
Posts: 7,316
Threads: 123
Joined: Sep 2016
(Oct-08-2019, 07:53 AM)adminravi Wrote: I am just a few days old in web scraping. Can you also suggest one good tutorial which I should learn from? Have a couple here about this topic.
Web-Scraping part-1
Web-Scraping part-2
|
