Python Forum
Download data from webpage after POST request
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Download data from webpage after POST request
#1
Hi,

I new to Python, and am trying to automate data download from a webpage.
The website (https://hirds.niwa.co.nz/search) needs a set of coordinates to generate the corresponding data.

It looks like my POST request is working (not certain of that though) but then I am stuck when I want to download the results of my request.

I can print part of the html but not all of it, and from the page source code it seems I have to look into hirds-app to get to the data but I don't know how to do that.
Any help appreciated! :)

import requests

URL = 'https://hirds.niwa.co.nz/search'
latitude = -37
longitude = 175
PARAMS = {'idf':'false','latitude':'-37', 'longitude':'175', 'site_name':'Custom Location'}

r = requests.post(URL, data= PARAMS)

if r.status_code == 200:
    print("Search successful")
    print(r.text)
    print r.headers.get('content-type')
else:
    print("Search failed")
from which I get the output:

Output:
Search successful <!doctype html><html lang="en"><head><meta charset="utf-8"><title>High Intensity Rainfall System</title><base href="/"><meta name="viewport" content="width=device-width,initial-scale=1"><link rel="icon" type="image/x-icon" href="favicon.ico"><link href="https://styles.niwa.co.nz/dist/css/bootstrap.min.css" type="text/css" rel="stylesheet"/><link href="https://styles.niwa.co.nz/dist/css/bootstrap-theme.min.css" type="text/css" rel="stylesheet"/><link href="styles.7a0326d5b8fae08856ca.bundle.css" rel="stylesheet"/></head><body class="gray-lighter"><script>(function (i, s, o, g, r, a, m) { i['GoogleAnalyticsObject'] = r; i[r] = i[r] || function () { (i[r].q = i[r].q || []).push(arguments) }, i[r].l = 1 * new Date(); a = s.createElement(o), m = s.getElementsByTagName(o)[0]; a.async = 1; a.src = g; m.parentNode.insertBefore(a, m) })(window, document, 'script', 'https://www.google-analytics.com/analytics.js', 'ga'); ga('create', 'UA-84422203-1', 'auto'); ga('send', 'pageview');</script><hirds-app>Loading...</hirds-app><script type="text/javascript" src="inline.889ef8fa81a66d3df1cc.bundle.js"></script><script type="text/javascript" src="polyfills.f86dd2e1cfeb0afc2f82.bundle.js"></script><script type="text/javascript" src="scripts.db6201ca1e70bf304d00.bundle.js"></script><script type="text/javascript" src="vendor.f27f543e7c03baf34549.bundle.js"></script><script type="text/javascript" src="main.989f9044ad5db4f79794.bundle.js"></script></body></html> text/html; charset=UTF-8
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Failed to download Qualtrics data using API balaKrishnaV 4 3,812 Sep-16-2023, 03:12 PM
Last Post: Jfreeland
  Unable to request image from FORM Data usman 0 968 Aug-18-2022, 06:23 PM
Last Post: usman
  Yfinance - Intraday stock data with yf.download diogo_80 2 5,870 Apr-29-2022, 05:07 AM
Last Post: processingclouds
  how can I correct the Bad Request error on my curl request tomtom 8 4,966 Oct-03-2021, 06:32 AM
Last Post: tomtom
  download with internet download manager coral_raha 0 2,880 Jul-18-2021, 03:11 PM
Last Post: coral_raha
  Code should download and output trading data tayy 2 2,594 Mar-04-2021, 04:07 AM
Last Post: tayy
  Fetching data from multiple tables in a single request. swaroop 0 1,855 Jan-09-2021, 04:23 PM
Last Post: swaroop
  getting back both: the map and the data from a request via overpass-turbo.eu apollo 0 1,545 Sep-26-2020, 11:28 AM
Last Post: apollo
  post data to influxdb shahrilmajid 0 1,382 Aug-28-2020, 03:12 PM
Last Post: shahrilmajid
  ImportError: cannot import name 'Request' from 'request' abhishek81py 1 3,859 Jun-18-2020, 08:07 AM
Last Post: buran

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020