Wanted to extract how many data sets are on 'https://catalog.data.gov/dataset#sec-organization_type'.
The HTML file was:
<body>
...
<div class="new-results">
<!-- Snippet snippets/search_result_text.html start -->
184,298 datasets found
<!-- Snippet snippets/search_result_text.html end -->
</div>
I used this python code:
The HTML file was:
<body>
...
<div class="new-results">
<!-- Snippet snippets/search_result_text.html start -->
184,298 datasets found
<!-- Snippet snippets/search_result_text.html end -->
</div>
I used this python code:
from lxml import html import requests response = requests.get('https://catalog.data.gov/dataset#sec-organization_type') doc = html.fromstring(response.text) link = doc.cssselect('div.new-results') for i in link: print(i.text)I don't know where the problem is