Bottom Page

Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
 Download data from webpage after POST request
#1
Hi,

I new to Python, and am trying to automate data download from a webpage.
The website (https://hirds.niwa.co.nz/search) needs a set of coordinates to generate the corresponding data.

It looks like my POST request is working (not certain of that though) but then I am stuck when I want to download the results of my request.

I can print part of the html but not all of it, and from the page source code it seems I have to look into hirds-app to get to the data but I don't know how to do that.
Any help appreciated! :)

import requests

URL = 'https://hirds.niwa.co.nz/search'
latitude = -37
longitude = 175
PARAMS = {'idf':'false','latitude':'-37', 'longitude':'175', 'site_name':'Custom Location'}

r = requests.post(URL, data= PARAMS)

if r.status_code == 200:
    print("Search successful")
    print(r.text)
    print r.headers.get('content-type')
else:
    print("Search failed")
from which I get the output:

Output:
Search successful <!doctype html><html lang="en"><head><meta charset="utf-8"><title>High Intensity Rainfall System</title><base href="/"><meta name="viewport" content="width=device-width,initial-scale=1"><link rel="icon" type="image/x-icon" href="favicon.ico"><link href="https://styles.niwa.co.nz/dist/css/bootstrap.min.css" type="text/css" rel="stylesheet"/><link href="https://styles.niwa.co.nz/dist/css/bootstrap-theme.min.css" type="text/css" rel="stylesheet"/><link href="styles.7a0326d5b8fae08856ca.bundle.css" rel="stylesheet"/></head><body class="gray-lighter"><script>(function (i, s, o, g, r, a, m) { i['GoogleAnalyticsObject'] = r; i[r] = i[r] || function () { (i[r].q = i[r].q || []).push(arguments) }, i[r].l = 1 * new Date(); a = s.createElement(o), m = s.getElementsByTagName(o)[0]; a.async = 1; a.src = g; m.parentNode.insertBefore(a, m) })(window, document, 'script', 'https://www.google-analytics.com/analytics.js', 'ga'); ga('create', 'UA-84422203-1', 'auto'); ga('send', 'pageview');</script><hirds-app>Loading...</hirds-app><script type="text/javascript" src="inline.889ef8fa81a66d3df1cc.bundle.js"></script><script type="text/javascript" src="polyfills.f86dd2e1cfeb0afc2f82.bundle.js"></script><script type="text/javascript" src="scripts.db6201ca1e70bf304d00.bundle.js"></script><script type="text/javascript" src="vendor.f27f543e7c03baf34549.bundle.js"></script><script type="text/javascript" src="main.989f9044ad5db4f79794.bundle.js"></script></body></html> text/html; charset=UTF-8
Quote

Top Page

Possibly Related Threads...
Thread Author Replies Views Last Post
  why does my python request cannot work with proxy and cannot get captured by fiddler szhangpy 0 135 Jun-27-2019, 10:48 PM
Last Post: szhangpy
  o365 special subject mail download issue anna 3 225 May-16-2019, 07:16 PM
Last Post: micseydel
  Issues parsing the response from a request garnold 3 273 May-14-2019, 12:39 PM
Last Post: snippsat
  urllib request error 404 Coco 2 410 May-11-2019, 02:47 PM
Last Post: Larz60+
  Better Coding request aankrose 5 310 May-09-2019, 01:54 PM
Last Post: perfringo
  [Idea Required] How can i track any change on a webpage? Prince_Bhatia 1 202 Apr-24-2019, 11:21 AM
Last Post: metulburr
  How to multiprocessing Api post request kinojom 1 257 Apr-03-2019, 01:11 AM
Last Post: Larz60+
  Error in request, cache key a21250450 2 319 Apr-02-2019, 11:20 AM
Last Post: a21250450
  How to get python to download YouTube videos in the background? Pythenx 11 750 Mar-25-2019, 04:57 AM
Last Post: samsonite
  Can't log in with requests post LilDog 2 922 Mar-11-2019, 08:16 AM
Last Post: yourv3nom

Forum Jump:


Users browsing this thread: 1 Guest(s)