Extracting Elements From A Website List

[knight2000]

Hi all,

I'm practicing some webscraping and have come up to an obstacle that has me stuck.

I'm trying to use code that will go through a few different pages of the same website and extract certain text from a list that is visible in each page. The challenge is, that the site list can have a different number of elements, so I'm not how to handle that if an element is not available.

Let me demo this to make it clearer...

Example html of one page in the website:

<ul class="nb-type-md nb-list-undecorated undefined">
	<li class=""><span><div class="nb-icon-small nb-inline-block nb-text-gray-200 nb-mr-2xs nb-align-middle"></div>Blue</span></li>
	<li class=""><span><div class="nb-icon-small nb-inline-block nb-text-gray-200 nb-mr-2xs nb-align-middle"></div>Designed in China</span></li>
	<li class=""><span><div class="nb-icon-small nb-inline-block nb-text-gray-200 nb-mr-2xs nb-align-middle"></div>http://www.mysupersite.com</span></li>
</ul> 

And here's an example of another page in the same website:

<ul class="nb-type-md nb-list-undecorated undefined">
	<li class=""><span><div class="nb-icon-small nb-inline-block nb-text-gray-200 nb-mr-2xs nb-align-middle"></div>Green</span></li>
	<li class=""><span><div class="nb-icon-small nb-inline-block nb-text-gray-200 nb-mr-2xs nb-align-middle"></div>Designed in England</span></li>
	<li class=""><span><div class="nb-icon-small nb-inline-block nb-text-gray-200 nb-mr-2xs nb-align-middle"></div>Shadow Chrome Painted</span></li>
	<li class=""><span><div class="nb-icon-small nb-inline-block nb-text-gray-200 nb-mr-2xs nb-align-middle"></div>http://www.mydifferentsite.com</span></li>
</ul>

As you can see, the first page has 3 items, whilst the second page has 4.

So if for example I'm trying to extract the url from these two pages (ie- http://www.mysupersite.com and http://www.mydifferentsite.com), how would I go about doing that?

My latest trial:

    for wa in lists.find_all('li'):
        if wa[3] is KeyError:
            wa[2]
        else:
            wa[3]

I get:

Error:
Traceback (most recent call last):
  File "C:/Users/testscrape.py", line 28, in <module>
    if wa[3] is KeyError:
  File "C:\Users\lib\site-packages\bs4\element.py", line 1406, in __getitem__
    return self.attrs[key]
KeyError: 3

I thought an IF statement would be what works- something like: IF wa[3] doesn't exist, then use wa[2], else wa[3]- but I don't know how to translate that into code

Could someone please enlighten me how to handle these sort's of optional indexes?

Thanks a lot.