Upload takes too long.

[Makada]

Hi,

As a test, i stopped updating the sourcefile for some minutes and tried to upload it, it uploaded fine. So it wont watch for a timestamp in the data uploaded.

[Makada]

I have added the correct time ("localdt") in the form WOW like, example: dateutc=2011-02-02+10%3A32%3A55
But the time in that particular line doesnt update...
Here is a partial output from the script beneath.
The upper date time (bold) is good, the power one (bold) isnt and should contain the date and time between those %3A from the above instead of 14:58:09.

Received KNMI WOW 429 The custom error module does not recognize this error.
Thu, 02 Apr 2020 17:10:02
winddir= 299.7
windspeedmph= 0.021
windgustmph= 0.032
windgustdir= 290.9
humidity= 70.68
dewptf= 29.63
tempf= 38.31
rainin= 0
dailyrainin= 0
baromin= 30.12243
solarradiation= 0
&dateutc=2020-04-02+14%3A58%3A09

import requests
import time
import schedule    
import csv
import urllib

date = time.strftime("%Y-%m-%d",time.gmtime())
hours = time.strftime("%H",time.gmtime())
minutes = time.strftime("%M",time.gmtime())
seconds = time.strftime("%S",time.gmtime())
# Save it as a string in a format that WOW likes


WUurl = "https://wow.metoffice.gov.uk/automaticreading?"
siteid = "xxxxxxxx" 
siteAuthenticationKey = "xxxxxxx" 
WUcreds = "siteid=" + siteid + "&siteAuthenticationKey="+ siteAuthenticationKey
action_str = "&action=updateraw"
localdt = "&dateutc="+date+"+"+hours+"%3A"+minutes+"%3A"+seconds

def task():

        WeatherUnderground= open("C:\\Campbellsci\\LoggerNet\\CR1000_upload.dat", "r")
        csvReader = csv.reader(WeatherUnderground)
        for field in csvReader:
         a = (field[12])
         b = (field[17])
         c = (field[19])
         d = (field[16])
         e = (field[8])
         f = (field[7])
         g = (field[5])
         h = (field[27])
         i = (field[25])
         j = (field[10])
         k = (field[23])
         l = (field[16])
        winddir=a
        windspeedmph=b
        windgustmph=c
        windgustdir=d
        humidity=e  
        dewptf=f
        tempf=g
        rainin=h
        dailyrainin=i
        baromin=j
        solarradiation=k
        softwaretype=1

        r= requests.get(
        WUurl +
        WUcreds +
        localdt +
        "&winddir=" +
        str(winddir)+
        "&windspeedmph=" +
        str(windspeedmph)+
        "&windgustmph=" +
        str(windgustmph)+
        "&windgustdir=" +
        str(windgustdir)+
        "&humidity=" +
        str(humidity)+
        "&dewptf=" +
        str(dewptf)+
        "&tempf=" +
        str(tempf)+
        "&rainin=" +
        str(rainin)+
        "&dailyrainin=" +
        str(dailyrainin)+
        "&baromin=" +
        str(baromin)+
        "&solarradiation=" +
        str(solarradiation)+
        str(softwaretype)+    
        action_str)
        
        print("Received KNMI WOW " + str(r.status_code) + " " + str(r.text),flush=True)
        print (time.strftime("%a, %d %b %Y %H:%M:%S"), flush=True)
        print ("winddir= " + a, flush=True)
        print ("windspeedmph= " + b, flush=True)
        print ("windgustmph= " + c, flush=True)
        print ("windgustdir= " + d, flush=True)
        print ("humidity= " + e, flush=True)
        print ("dewptf= " + f, flush=True)
        print ("tempf= " + g, flush=True)
        print ("rainin= " + h, flush=True)
        print ("dailyrainin= " + i, flush=True)
        print ("baromin= " + j, flush=True)
        print ("solarradiation= " + k, flush=True)
        print (minutes, flush=True)

while True:
 try:
     task()
     time.sleep(300 - time.time() % 300)
     refresh()
 except:
    pass
    time.sleep(1)

else:
 time.sleep(5)

---

Hi,

With some changes i managed to get it working.
Now lets see if its uploading the way it supposed to do .

import requests
import csv
import hashlib
import time
import schedule
import urllib
from time import sleep
import sys
starttime=time.time()

def task():

    date = time.strftime("%Y-%m-%d")
    hours = time.strftime("%H")
    minutes = time.strftime("%M")
    seconds = time.strftime("%S")
    localdt = "&dateutc="+date+"+"+hours+"%3A"+minutes+"%3A"+seconds


    WUurl = "https://wow.metoffice.gov.uk/automaticreading?"
    siteid = "" 
    siteAuthenticationKey = "" 
    WUcreds = "siteid=" + siteid + "&siteAuthenticationKey="+ siteAuthenticationKey
    action_str = "&action=updateraw"


    WeatherUnderground= open("C:\\Campbellsci\\LoggerNet\\CR1000_upload.dat", "r")
    csvReader = csv.reader(WeatherUnderground)
    for field in csvReader:
        a = (field[12])
        b = (field[17])
        c = (field[19])
        d = (field[16])
        e = (field[8])
        f = (field[7])
        g = (field[5])
        h = (field[27])
        i = (field[25])
        j = (field[10])
        k = (field[23])
        l = (field[16])
        winddir=a
        windspeedmph=b
        windgustmph=c
        windgustdir=d
        humidity=e  
        dewptf=f
        tempf=g
        rainin=h
        dailyrainin=i
        baromin=j
        solarradiation=k
        softwaretype=1
        r= requests.get(
        WUurl +
        WUcreds +
        localdt +
        "&winddir=" +
        str(winddir)+
        "&windspeedmph=" +
        str(windspeedmph)+
        "&windgustmph=" +
        str(windgustmph)+
        "&windgustdir=" +
        str(windgustdir)+
        "&humidity=" +
        str(humidity)+
        "&dewptf=" +
        str(dewptf)+
        "&tempf=" +
        str(tempf)+
        "&rainin=" +
        str(rainin)+
        "&dailyrainin=" +
        str(dailyrainin)+
        "&baromin=" +
        str(baromin)+
        "&solarradiation=" +
        str(solarradiation)+
        str(softwaretype)+    
        action_str)
        
    print("Received KNMI WOW " + str(r.status_code) + " " + str(r.text),flush=True)
    print(time.strftime("%Y%m%d%H%M%S"), flush=True)
    print ("winddir= " + a, flush=True)
    print ("windspeedmph= " + b, flush=True)
    print ("windgustmph= " + c, flush=True)
    print ("windgustdir= " + d, flush=True)
    print ("humidity= " + e, flush=True)
    print ("dewptf= " + f, flush=True)
    print ("tempf= " + g, flush=True)
    print ("rainin= " + h, flush=True)
    print ("dailyrainin= " + i, flush=True)
    print ("baromin= " + j, flush=True)
    print ("solarradiation= " + k, flush=True)
    print (localdt, flush=True)

while True:
 try:
     task()
     time.sleep(300 - time.time() % 300)
     refresh()
 except:
    pass
    time.sleep(1)

else:
 time.sleep(300)
 refresh()
     

[BrendanD]

WOW is obviously blocking uploads per site made less than five minutes apart. They do not state this in their docs they only suggest in their faq that five minutes is the recommended upload frequency for aws. So why not decrease the upload frequency to 10 minutes, or hourly if you can calculate the mean over the past hour. It sounds like your aws is configured to dump output at specified frequency.

1. Does it dump a new csv each time?
   
2. Does the csv only have one line?
   
3. Does your AWS let you choose whether to dump time averaged or instantaneous values?
   
4. Why are you not parsing dateutc from the fields in the csv?
   
5. Are you using their TEST site protocol? You should not be submitting real data until you have confidence you are doing the right thing. Unless you know you can manually delete the bad data after the fact.
   

[Makada]

Hi,

I really want to have it every 5 minutes, ive seen other users (with weathersoftware doing this task automatically, but i want it to work in python:)) with data uploaded every 5 minutes without an error.

Answers to your questions:

1. No
2. Yes
3. Averaged
4. I dont know how to do that, and what differnce does it make?
5. Dont know the test protocol....

---

I just see the data isnt shown on their website, although i get the message in the script output saying "Received 200 ()
While the specific timestamp is updating fine now.

[BrendanD]

Post an example of your csv and I'll help you parse the datetime. If you do not care about the accuracy of datetime to within five minutes, why care so much about uploading every five minutes? _shrug_
Are you not using an existing package like pywws for fun?

[Makada]

Heres an example of the source:
101,2020,93,2148,8.65,47.57,5.673,42.21,81.5,1013.315,29.92307,17.77,254.8,19.18,25.91,2147,262.9,11.04,9.6,16.1,2147,13.99,2147,0,0,0,0,0

[BrendanD]

Um.... you are lucky I am good at guessing...

I can guess the year is 2020, I can guess 93 is the day of the year, I can guess 2148 is the time, I can guess 101 is the record number. Amiright? Is 2148 local time? Or UTC? If it is local time, then I would need to know your timezone...

[Makada]

Hi,

Yes youre right
My time is local time, which is in the Netherlands.
Is that utc + 2 ?

[BrendanD]

So 21:48 was your local time?

(Can you set your AWS to UTC? When dealing with instruments and data, UTC is king of the hill. No worries about conversion.)

[Makada]

Yes 21:48 local time.
Its currently set to local daylight time, as seen in the picture.
I dont know what happens with my logger, program, and time when i choose UTC from the list shown?