Python Forum

On the job web page I am scraping, some of the jobs have a Job Requirements. I am trying to get the list of requirements and having difficulty.

Below is the html code I need to get the requirement info:

`
<div class="jobCardReqContainer">
<div class="jobCardReqHeader">Requirements</div>
<div class="jobCardReqList"><div class="jobCardReqItem">Windows: 2 years</div>
<div class="jobCardReqItem">Python Software Development: 3 years</div>
<div class="jobCardReqItem">Cloud Technologies: 3 years</div>
</div>
`

Below is my code that is not working:

`
job_req = jobs[rcount].find_all("div", "jobCardReqItem").get_text().strip()
print(job_req)
`

Quick test so it work,check turn of JavaScript in Browser then reload page and see if still can see Remote.
If not may need to use Selenium so can get value generated bye JavaScript.

>>> from bs4 import BeautifulSoup
>>> 
>>> html = '<span class="remote">Remote</span>'
>>> soup = BeautifulSoup(html, 'lxml')
>>> s_tag = soup.find('span', class_="remote")
>>> s_tag
<span class="remote">Remote</span>
>>> s_tag.text
'Remote'

>>> # Using CSS selector
>>> soup.select_one('.remote').text
'Remote'