Hi,
I new to Python, and am trying to automate data download from a webpage.
The website (https://hirds.niwa.co.nz/search) needs a set of coordinates to generate the corresponding data.
It looks like my POST request is working (not certain of that though) but then I am stuck when I want to download the results of my request.
I can print part of the html but not all of it, and from the page source code it seems I have to look into hirds-app to get to the data but I don't know how to do that.
Any help appreciated! :)
I new to Python, and am trying to automate data download from a webpage.
The website (https://hirds.niwa.co.nz/search) needs a set of coordinates to generate the corresponding data.
It looks like my POST request is working (not certain of that though) but then I am stuck when I want to download the results of my request.
I can print part of the html but not all of it, and from the page source code it seems I have to look into hirds-app to get to the data but I don't know how to do that.
Any help appreciated! :)
import requests URL = 'https://hirds.niwa.co.nz/search' latitude = -37 longitude = 175 PARAMS = {'idf':'false','latitude':'-37', 'longitude':'175', 'site_name':'Custom Location'} r = requests.post(URL, data= PARAMS) if r.status_code == 200: print("Search successful") print(r.text) print r.headers.get('content-type') else: print("Search failed")from which I get the output:
Output:Search successful
<!doctype html><html lang="en"><head><meta charset="utf-8"><title>High Intensity Rainfall System</title><base href="/"><meta name="viewport" content="width=device-width,initial-scale=1"><link rel="icon" type="image/x-icon" href="favicon.ico"><link href="https://styles.niwa.co.nz/dist/css/bootstrap.min.css" type="text/css" rel="stylesheet"/><link href="https://styles.niwa.co.nz/dist/css/bootstrap-theme.min.css" type="text/css" rel="stylesheet"/><link href="styles.7a0326d5b8fae08856ca.bundle.css" rel="stylesheet"/></head><body class="gray-lighter"><script>(function (i, s, o, g, r, a, m) {
i['GoogleAnalyticsObject'] = r; i[r] = i[r] || function () {
(i[r].q = i[r].q || []).push(arguments)
}, i[r].l = 1 * new Date(); a = s.createElement(o),
m = s.getElementsByTagName(o)[0]; a.async = 1; a.src = g; m.parentNode.insertBefore(a, m)
})(window, document, 'script', 'https://www.google-analytics.com/analytics.js', 'ga');
ga('create', 'UA-84422203-1', 'auto');
ga('send', 'pageview');</script><hirds-app>Loading...</hirds-app><script type="text/javascript" src="inline.889ef8fa81a66d3df1cc.bundle.js"></script><script type="text/javascript" src="polyfills.f86dd2e1cfeb0afc2f82.bundle.js"></script><script type="text/javascript" src="scripts.db6201ca1e70bf304d00.bundle.js"></script><script type="text/javascript" src="vendor.f27f543e7c03baf34549.bundle.js"></script><script type="text/javascript" src="main.989f9044ad5db4f79794.bundle.js"></script></body></html>
text/html; charset=UTF-8