Python Forum
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Scaping (all) of a specific tag
#1
Most pages have multiple h2's for example.

I can manually add [0], [1] and [2] for example which is great - but what if a page had 20 h2's? How do I scrape all of them?

I tried a couple of things... I thought maybe doing [0:] would work, but that's not the case. I'm not really sure what to type into Google, or the forum to get an answer either.

Current code which manually gets as many tags as I specify.
from bs4 import BeautifulSoup
import requests

url= 'https://python.org'
url_get = requests.get(url)
soup = BeautifulSoup(url_get.content, 'lxml')
print(soup.select('h2')[0].text)
print(soup.select('h2')[1].text)
print(soup.select('h2')[2].text)
Reply
#2
select by default returns all of the requested item.
to find the individual items:
from bs4 import BeautifulSoup
import requests
 
url= 'https://python.org'
url_get = requests.get(url)
soup = BeautifulSoup(url_get.content, 'lxml')
all_h2 = soup.select('h2')
for h2 in all_h2:
    print(h2.text)
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Scaping pages created by javascript mbizzl 1 1,482 Jul-17-2022, 10:01 PM
Last Post: Larz60+

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020