Python Forum
Extracting Elements From A Website List
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Extracting Elements From A Website List
#1
Hi all,

I'm practicing some webscraping and have come up to an obstacle that has me stuck.

I'm trying to use code that will go through a few different pages of the same website and extract certain text from a list that is visible in each page. The challenge is, that the site list can have a different number of elements, so I'm not how to handle that if an element is not available.

Let me demo this to make it clearer...

Example html of one page in the website:

<ul class="nb-type-md nb-list-undecorated undefined">
	<li class=""><span><div class="nb-icon-small nb-inline-block nb-text-gray-200 nb-mr-2xs nb-align-middle"></div>Blue</span></li>
	<li class=""><span><div class="nb-icon-small nb-inline-block nb-text-gray-200 nb-mr-2xs nb-align-middle"></div>Designed in China</span></li>
	<li class=""><span><div class="nb-icon-small nb-inline-block nb-text-gray-200 nb-mr-2xs nb-align-middle"></div>http://www.mysupersite.com</span></li>
</ul> 
And here's an example of another page in the same website:

<ul class="nb-type-md nb-list-undecorated undefined">
	<li class=""><span><div class="nb-icon-small nb-inline-block nb-text-gray-200 nb-mr-2xs nb-align-middle"></div>Green</span></li>
	<li class=""><span><div class="nb-icon-small nb-inline-block nb-text-gray-200 nb-mr-2xs nb-align-middle"></div>Designed in England</span></li>
	<li class=""><span><div class="nb-icon-small nb-inline-block nb-text-gray-200 nb-mr-2xs nb-align-middle"></div>Shadow Chrome Painted</span></li>
	<li class=""><span><div class="nb-icon-small nb-inline-block nb-text-gray-200 nb-mr-2xs nb-align-middle"></div>http://www.mydifferentsite.com</span></li>
</ul>
As you can see, the first page has 3 items, whilst the second page has 4.

So if for example I'm trying to extract the url from these two pages (ie- http://www.mysupersite.com and http://www.mydifferentsite.com), how would I go about doing that?

My latest trial:
    for wa in lists.find_all('li'):
        if wa[3] is KeyError:
            wa[2]
        else:
            wa[3]
I get:
Error:
Traceback (most recent call last): File "C:/Users/testscrape.py", line 28, in <module> if wa[3] is KeyError: File "C:\Users\lib\site-packages\bs4\element.py", line 1406, in __getitem__ return self.attrs[key] KeyError: 3
I thought an IF statement would be what works- something like: IF wa[3] doesn't exist, then use wa[2], else wa[3]- but I don't know how to translate that into code Undecided

Could someone please enlighten me how to handle these sort's of optional indexes?

Thanks a lot.
Reply
#2
(Jul-19-2021, 08:08 AM)knight2000 Wrote: So if for example I'm trying to extract the url from these two pages (ie- http://www.mysupersite.com and http://www.mydifferentsite.com), how would I go about doing that?
Can get exact with CSS selector or find_all() with index.
To give a example.
from bs4 import BeautifulSoup

html = '''\
<ul class="nb-type-md nb-list-undecorated undefined">
    <li class=""><span><div class="nb-icon-small nb-inline-block nb-text-gray-200 nb-mr-2xs nb-align-middle"></div>Green</span></li>
    <li class=""><span><div class="nb-icon-small nb-inline-block nb-text-gray-200 nb-mr-2xs nb-align-middle"></div>Designed in England</span></li>
    <li class=""><span><div class="nb-icon-small nb-inline-block nb-text-gray-200 nb-mr-2xs nb-align-middle"></div>Shadow Chrome Painted</span></li>
    <li class=""><span><div class="nb-icon-small nb-inline-block nb-text-gray-200 nb-mr-2xs nb-align-middle"></div>http://www.mydifferentsite.com</span></li>
</ul>'''

soup = BeautifulSoup(html, 'lxml')
tag = soup.select_one('ul > li:nth-child(4)')
print(tag.text)

Output:
http://www.mydifferentsite.com
Find find_all().
>>> tag_ul = soup.find('ul', class_="nb-type-md")
>>> tag_li = tag_ul.find_all('li')
>>> tag_li[3].text
'http://www.mydifferentsite.com'

# Loop
>>> for item in tag_li:
...     print(item.text)
...     
Green
Designed in England
Shadow Chrome Painted
http://www.mydifferentsite.com
Look at Web-Scraping part-1.
Reply
#3
Hi snippsat,

Thanks a lot for your explanation and direction- appreciate you taking time out to help me understand more.

Since posting, I also kept trying different things and I eventually came across a way to extract the url text where on one webpage it was the 3rd item of the list and on another webpage, it was the forth item on the list (as per my example)

I'm not sure whether this is a good or bad way of doing things- but it works and may help someone :)

soup = BeautifulSoup(r.text, 'html.parser')
    lists = soup.find_all('ul', {'class': 'nb-type-md nb-list-undecorated undefined'})
    li = lists.find_all('li')
    number_elements = len(li)

    if number_elements == 3:
          for wa in lists.find_all('li')[2]:
                webaddress = wa.text
                
    elif number_elements == 4:
           for wa in lists.find_all('li')[3]:
                webaddress = wa.text
                
Thank you for the link to learn more of the basics too- I'll try and go through as much of that as I can on the weekend. Have a good week mate.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Why am I getting list elements < 0 ? Mark17 8 607 Aug-26-2021, 09:31 AM
Last Post: naughtyCat
  Looping through nested elements and updating the original list Alex_James 3 349 Aug-19-2021, 12:05 PM
Last Post: Alex_James
  Make Groups with the List Elements quest 2 477 Jul-11-2021, 09:58 AM
Last Post: perfringo
  I cannot delete and the elements from the list quest 4 920 May-11-2021, 12:01 PM
Last Post: perfringo
  List of lists - merge sublists with common elements medatib531 1 968 May-09-2021, 07:49 AM
Last Post: Gribouillis
  Find Common Elements in 2 list quest 4 864 Apr-14-2021, 03:57 PM
Last Post: quest
  Sorting Elements via parameters pointing to those elements. rpalmer 3 876 Feb-10-2021, 04:53 PM
Last Post: rpalmer
  Group List Elements according to the Input with the order of binary combination quest_ 19 1,930 Jan-28-2021, 03:36 AM
Last Post: bowlofred
  Extracting data from a website tgottsc1 2 730 Jan-09-2021, 08:14 PM
Last Post: tgottsc1
  Adding List Element if Second part of the List Elements are the Same quest_ 3 843 Nov-25-2020, 04:33 PM
Last Post: bowlofred

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020