Python Forum
Download data from webpage after POST request - Printable Version

+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: General Coding Help (https://python-forum.io/forum-8.html)
+--- Thread: Download data from webpage after POST request (/thread-15806.html)

Download data from webpage after POST request - AlDe - Feb-02-2019

Hi,

I new to Python, and am trying to automate data download from a webpage.
The website (https://hirds.niwa.co.nz/search) needs a set of coordinates to generate the corresponding data.

It looks like my POST request is working (not certain of that though) but then I am stuck when I want to download the results of my request.

I can print part of the html but not all of it, and from the page source code it seems I have to look into hirds-app to get to the data but I don't know how to do that.
Any help appreciated! :)

import requests

URL = 'https://hirds.niwa.co.nz/search'
latitude = -37
longitude = 175
PARAMS = {'idf':'false','latitude':'-37', 'longitude':'175', 'site_name':'Custom Location'}

r = requests.post(URL, data= PARAMS)

if r.status_code == 200:
    print("Search successful")
    print(r.text)
    print r.headers.get('content-type')
else:
    print("Search failed")
from which I get the output:

Output:
Search successful
<!doctype html><html lang="en"><head><meta charset="utf-8"><title>High Intensity Rainfall System</title><base href="/"><meta name="viewport" content="width=device-width,initial-scale=1"><link rel="icon" type="image/x-icon" href="favicon.ico"><link href="https://styles.niwa.co.nz/dist/css/bootstrap.min.css" type="text/css" rel="stylesheet"/><link href="https://styles.niwa.co.nz/dist/css/bootstrap-theme.min.css" type="text/css" rel="stylesheet"/><link href="styles.7a0326d5b8fae08856ca.bundle.css" rel="stylesheet"/></head><body class="gray-lighter"><script>(function (i, s, o, g, r, a, m) {
        i['GoogleAnalyticsObject'] = r; i[r] = i[r] || function () {
            (i[r].q = i[r].q || []).push(arguments)
        }, i[r].l = 1 * new Date(); a = s.createElement(o),
            m = s.getElementsByTagName(o)[0]; a.async = 1; a.src = g; m.parentNode.insertBefore(a, m)
        })(window, document, 'script', 'https://www.google-analytics.com/analytics.js', 'ga');

        ga('create', 'UA-84422203-1', 'auto');
        ga('send', 'pageview');</script><hirds-app>Loading...</hirds-app><script type="text/javascript" src="inline.889ef8fa81a66d3df1cc.bundle.js"></script><script type="text/javascript" src="polyfills.f86dd2e1cfeb0afc2f82.bundle.js"></script><script type="text/javascript" src="scripts.db6201ca1e70bf304d00.bundle.js"></script><script type="text/javascript" src="vendor.f27f543e7c03baf34549.bundle.js"></script><script type="text/javascript" src="main.989f9044ad5db4f79794.bundle.js"></script></body></html>
text/html; charset=UTF-8