Python Forum

Full Version: Web crawler extracting specific text from HTML
You're currently viewing a stripped down version of our content. View the full version with proper formatting.
Hi - I've just started to learn how to use python and am exploring the elements of a web crawler. I'm trying to extract the text that follows "Licences" from this page (in this instance, I would like the result to be 'COPD Licence' for example).

So far I have the basics:
import requests
from bs4 import BeautifulSoup

result = requests.get(
    "https://www.rightbreathe.com/medicines/eklira-322microgramsdose-genuair-astrazeneca-uk-ltd-60-dose/?s=")
src = result.content
soup = BeautifulSoup(src, 'html.parser')
but then I'm struggling to successfully define the specific element that I'm going after - can anyone help please?
Can use CSS selector here as many of class name are the same.
>>> soup.select('div.MedicineDeviceProduct-detail > div > div:nth-child(2) > div > span > ul > li')
[<li>COPD Licence</li>]
>>> soup.select_one('div.MedicineDeviceProduct-detail > div > div:nth-child(2) > div > span > ul > li')
<li>COPD Licence</li>
>>> soup.select_one('div.MedicineDeviceProduct-detail > div > div:nth-child(2) > div > span > ul > li').text
'COPD Licence'
You can find selector in browser and copy it.