+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: General Coding Help (https://python-forum.io/forum-8.html)
+--- Thread: Upload takes too long. (/thread-25506.html)
Upload takes too long. - Makada - Apr-01-2020
Hi all,
I have an upload script which uploads data every 5 minutes at every 00:00 00:05 00:10 00:15 and so on during 24/7.
But sometimes the upload takes too long.
The interval must be exactly at every 5 minutes and zero seconds,
Or for example when an upload is at 00:05:01 the next upload has to be at 00:10:01 with the same amount of seconds after the minute.
As you can see in the example upload below, one upload is done at 16:50:11 and the next upload is done at 16:55:01.
So thats shorter then exactly 5 minutes and zero seconds and thats why it gets an error and wont upload.
Ive also tried the "schedule.every(5).minutes.do(task)" but that wont start at exactly every 00:00:00 00:05:00 00:10:00 and has the same non exact interval.
My python script is included.
Received KNMI WOW 200 {}
Wed, 01 Apr 2020 16:50:11
winddir= 329.2
windspeedmph= 9.74
windgustmph= 14.28
windgustdir= 329.9
humidity= 44.24
dewptf= 31.54
tempf= 52.69
rainin= 0
dailyrainin= 0
baromin= 30.16438
solarradiation= 473.0833
Received KNMI WOW 429 The custom error module does not recognize this error.
Wed, 01 Apr 2020 16:55:01
winddir= 329.9
windspeedmph= 6.027
windgustmph= 13.15
windgustdir= 329.9
humidity= 44.29
dewptf= 31.36
tempf= 52.46
rainin= 0
dailyrainin= 0
baromin= 30.16297
solarradiation= 524.5936
import requests
import time
import schedule
import csv
import urllib
WUurl = "https://wow.metoffice.gov.uk/automaticreading?"
siteid = "xxxxxxxx"
siteAuthenticationKey = xxxxxx
WUcreds = "siteid=" + siteid + "&siteAuthenticationKey="+ siteAuthenticationKey
date_str = "&dateutc=now"
action_str = "&action=updateraw"
def task():
WeatherUnderground= open("C:\\Campbellsci\\LoggerNet\\CR1000_upload.dat", "r")
csvReader = csv.reader(WeatherUnderground)
for field in csvReader:
a = (field[12])
b = (field[17])
c = (field[19])
d = (field[16])
e = (field[8])
f = (field[7])
g = (field[5])
h = (field[27])
i = (field[25])
j = (field[10])
k = (field[23])
l = (field[16])
winddir=a
windspeedmph=b
windgustmph=c
windgustdir=d
humidity=e
dewptf=f
tempf=g
rainin=h
dailyrainin=i
baromin=j
solarradiation=k
softwaretype=1
r= requests.get(
WUurl +
WUcreds +
date_str +
"&winddir=" +
str(winddir)+
"&windspeedmph=" +
str(windspeedmph)+
"&windgustmph=" +
str(windgustmph)+
"&windgustdir=" +
str(windgustdir)+
"&humidity=" +
str(humidity)+
"&dewptf=" +
str(dewptf)+
"&tempf=" +
str(tempf)+
"&rainin=" +
str(rainin)+
"&dailyrainin=" +
str(dailyrainin)+
"&baromin=" +
str(baromin)+
"&solarradiation=" +
str(solarradiation)+
str(softwaretype)+
action_str)
timeout=60
print("Received KNMI WOW " + str(r.status_code) + " " + str(r.text),flush=True)
print (time.strftime("%a, %d %b %Y %H:%M:%S"), flush=True)
print ("winddir= " + a, flush=True)
print ("windspeedmph= " + b, flush=True)
print ("windgustmph= " + c, flush=True)
print ("windgustdir= " + d, flush=True)
print ("humidity= " + e, flush=True)
print ("dewptf= " + f, flush=True)
print ("tempf= " + g, flush=True)
print ("rainin= " + h, flush=True)
print ("dailyrainin= " + i, flush=True)
print ("baromin= " + j, flush=True)
print ("solarradiation= " + k, flush=True)
while True:
try:
task()
time.sleep(300 - time.time() % 300)
refresh()
except:
pass
time.sleep(1)
else:
time.sleep(300)
RE: Upload takes too long. - deanhystad - Apr-01-2020
To rephrase to make sure I understand,
This data gets updated on the clock every 5 minutes, but not exactly 5 minutes. When you try to read the data and it isn't yet available your program is in error.
If the above summary is correct, you cannot solve this problem by adjusting the upload time. If the producer is uploading five minutes after the last update but is prone to delays that shift the upload, their upload time will creep over time and you will not know when to do your first upload.
I have to believe that the producer is uploading every 5 minutes on the clock, but there is some small variation as to when the data becomes available. If this is the case, you need to provide error handling in your code to resample when the data is not available at the specified time.
RE: Upload takes too long. - Makada - Apr-01-2020
Hi,
Thanks for your reply.
It is my own data which is uploaded .
Ive got another upload with same data to other website every minute which is running errorless every second past the minute exactly.
But with this upload, as soon as theres exaclty 5 minutes and zero seconds less time between upload, it gives an error.
Is it possible to have the "dateutc=now" in zero seconds (00) at every Upload?
RE: Upload takes too long. - BrendanD - Apr-01-2020
See the following regarding error 429. https://stackoverflow.com/questions/22786068/how-to-avoid-http-error-429-too-many-requests-python#23367215
RE: Upload takes too long. - Makada - Apr-01-2020
Hi,
Thanks for your input.
It is exactly what im experiencing.
Due to the less then exactly 5 minute upload, it is by seconds a "too many requests"
Maybe a solution would be to have the seconds in "dateutc=now" set to 00 at every upload, so it wont notice a difference...?
Or a retry upload within some seconds?
RE: Upload takes too long. - BrendanD - Apr-01-2020
Well, I can't tell you what &dateutc means without reading up on whatever API metoffice is using for this service. Is it supposed to be the date of your data? Seems to me the five minute upload limit is something on the metoffice end to prevent spamming and should have nothing to do with &dateutc. Why is there not a data timestamp in your log file?
RE: Upload takes too long. - Makada - Apr-01-2020
Hi,
Heres an example upload from Metoffice website:
http://wow.metoffice.gov.uk/automaticreading?siteid=123456&siteAuthenticationKey=654321&dateutc=2011-02-02+10%3A32%3A55&winddir=230&windspeedmph=12&windgustmph=12&windgustdir=25&humidity=90&dewptf=68.2&tempf=70&rainin=0&dailyrainin=5&baromin=29.1&soiltempf=25&soilmoisture=25&visibility=25&softwaretype=weathersoftware1.0Regarding the "dateutc=now", thats maybe why its behaving this way, the upload issue.
I dont know what timestamp my script is using.
I found an example (although its written for Arduino)for the "dateutc=now", code below.
But i dont know how to get this working in my script.
Heres the example:
date = time.strftime("%Y-%m-%d",time.gmtime())
hours = time.strftime("%H",time.gmtime())
minutes = time.strftime("%M",time.gmtime())
seconds = time.strftime("%S",time.gmtime())
# Save it as a string in a format that WOW likes
localdt = date+"+"+hours+"%3A"+minutes+"%3A"+seconds
RE: Upload takes too long. - BrendanD - Apr-01-2020
Okay so:
1) Does WOW only take data with timestamps exactly 5 minutes apart? (per siteid)
OR
2) Does WOW only allow one upload every 5 minutes? (per siteid)
Without the correct answer to this question I can't help further.
RE: Upload takes too long. - Makada - Apr-01-2020
Hi,
Its not the data timestamp, its the time(stamp) of uploading.
If i dont update my data, its still getting uploaded.
Yes, one upload every 5 minutes at 00:00:00 0:05:00 00:10:00 etc.
RE: Upload takes too long. - BrendanD - Apr-01-2020
I looked over at the WOW page. Your datestr above is supposed to be the measurement time, the one that should be in your csv file. Or I am misunderstanding. I don;t understand what you mean by:
Quote:If i dont update my data, its still getting uploaded.