Python Forum
Web crawler extracting specific text from HTML - Printable Version

+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/Forum-Python-Coding)
+--- Forum: Web Scraping & Web Development (https://python-forum.io/Forum-Web-Scraping-Web-Development)
+--- Thread: Web crawler extracting specific text from HTML (/Thread-Web-crawler-extracting-specific-text-from-HTML)



Web crawler extracting specific text from HTML - lewdow - Jan-03-2020

Hi - I've just started to learn how to use python and am exploring the elements of a web crawler. I'm trying to extract the text that follows "Licences" from this page (in this instance, I would like the result to be 'COPD Licence' for example).

So far I have the basics:
import requests
from bs4 import BeautifulSoup

result = requests.get(
    "https://www.rightbreathe.com/medicines/eklira-322microgramsdose-genuair-astrazeneca-uk-ltd-60-dose/?s=")
src = result.content
soup = BeautifulSoup(src, 'html.parser')
but then I'm struggling to successfully define the specific element that I'm going after - can anyone help please?


RE: Web crawler extracting specific text from HTML - snippsat - Jan-03-2020

Can use CSS selector here as many of class name are the same.
>>> soup.select('div.MedicineDeviceProduct-detail > div > div:nth-child(2) > div > span > ul > li')
[<li>COPD Licence</li>]
>>> soup.select_one('div.MedicineDeviceProduct-detail > div > div:nth-child(2) > div > span > ul > li')
<li>COPD Licence</li>
>>> soup.select_one('div.MedicineDeviceProduct-detail > div > div:nth-child(2) > div > span > ul > li').text
'COPD Licence'
You can find selector in browser and copy it.