Python Forum
Code Help, web scraping non uniform lists(ul)
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Code Help, web scraping non uniform lists(ul)
#4
It's common to get some error when try to scraping a lot of pages.
There can be different way handle it,can make an AttributeError error as a demo.
from bs4 import BeautifulSoup

html = '''\
<tr>
  <td id="BMW">Black color</td>
  <td>2014 model</td>
</tr>'''

soup = BeautifulSoup(html, 'lxml')
for td in soup.find('td', id="BMW").find_next('td'):
        print(td)
Output:
2014 model
So if change id to Lada🚗 get an AttributeError(some would say it's a big error😲),then can catch it with try: except.
from bs4 import BeautifulSoup

html = '''\
<tr>
  <td id="BMW">Black color</td>
  <td>2014 model</td>
</tr>'''

soup = BeautifulSoup(html, 'lxml')
try:
    for td in soup.find('td', id="Lada").find_next('td'):
        print(td)
except AttributeError:
    print('Got an error AttributeError')
    td = 'Dummy value'

print(td)
Output:
Got an error AttributeError Dummy value
Do nothing just skip error would be.
except AttributeError:
    pass
Reply


Messages In This Thread
RE: Code Help, web scraping non uniform lists(ul) - by snippsat - Apr-22-2021, 11:45 AM

Possibly Related Threads…
Thread Author Replies Views Last Post
  scraping code misses listings kolarmi19 0 1,185 Jan-27-2023, 10:00 AM
Last Post: kolarmi19
  scraping code nexuz89 0 1,603 Sep-28-2020, 12:16 PM
Last Post: nexuz89
  In need of web scraping code! kolbyng 1 1,800 Sep-21-2020, 06:02 AM
Last Post: buran
  error in code web scraping alexisbrunaux 5 3,978 Aug-19-2020, 02:31 AM
Last Post: alexisbrunaux
  scraping from a website that hides source code PIWI_Protein 1 2,054 Mar-27-2020, 05:08 PM
Last Post: Larz60+
  Web Scraping, Merging two lists and getting data from various dates? AgileAVS 0 1,925 Feb-07-2020, 04:05 PM
Last Post: AgileAVS

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020