API JSON Iteration...again

[cyberblitz]

Wanted to edit my previous post but was not able to, so I'm reposting it here


Hi,

New to Python. I am trying to create some code that extracts API JSON from a website and display the results on an LCD screen.
I am trying to display a clock and a message on the LCD when there are no results to show. When results come in, it reverts to display the results, then when no results are available, go back to the clock.

Also, I can't find any better solution than extracting the entire API JSON data, which can take a few seconds pausing the code. IS there a better way to hit the URL than this too? I find this creates a timeout error all the time. I had tried to place a sleep function of 5 secs but found this too creates an error eventually.

Any advice on any of the above on the below code would be appreciated.

NOTE: I've turned off the LCD module to allow for testing. This will all be executed on a Raspberry Pi

import requests
import pickle
from time import sleep
import time
import datetime
import json
#import I2C_LCD_Driver
 
# API URL
rescue_url = "https://api.somesite.com/rescues"
headers = {"Authorization": "Bearer token"}
 
 
def get_data(url):
    data_json = []
    data = requests.get(url, headers=headers).json()
    data = data['data']
    for dta in data:
        if dta['attributes']['stat'] == 'open':
            data_json.append(dta['attributes'])
    return data_json
 
 
# function to convert string datetime value when rescue commenced (usually entered by Mecha)
def dte_convert(dte):
    d1 = datetime.datetime.strptime(dte, "%Y-%m-%dT%H:%M:%S.%fZ") + datetime.timedelta(hours=10)
    d2 = datetime.datetime.now()  # set variable to current dateTime
    dur_s = (d2 - d1).total_seconds()  # similar to datediff and calculates to seconds
    dy = divmod(dur_s, 86400)
    hrs = divmod(dy[1], 3600)
    mins = divmod(hrs[1], 60)
    ret = '%d d %d h %d m' % (dy[0], hrs[0], mins[0])
    return ret
 
 
while True:
    json_data = get_data(rescue_url)
    if json_data:
        for each in json_data:
            platform = each['platform']
            client = each['client']
            cde = each['code']
            system = each['sys']
            res_datetime = dte_convert(each['createdAt'])
        #           mylcd.lcd_display_string("C: %s" %client ,1)
            #           mylcd.lcd_display_string("P: %s" %platform + "C: %s" %str(cde), 2)
            #           mylcd.lcd_display_string("S: %s" %sys, 1)
            #           mylcd.lcd_display_string("A: %s" %str(res_datetime), 2)
            print("C: %s" % client)
            print("P: %s" % platform + " CR: %s" % str(cde_red))
            print("S: %s" % system)
            print("A: %s" % str(res_datetime))
    else:
            print("Date: %s" % time.strftime("%d/%m/%Y"))
            print("Time: %s" % time.strftime("%H:%M"))
            print('No Active Rescues')

[cyberblitz]

Hi,

Has no one got any advice?? Please!

Or is this code perfect :)

[metulburr]

you could try this
https://web.archive.org/web/201808180544...g-timeouts

import requests

timeout = 60 # How long to wait until trying again
url = 'http://winstonlarson.com'

while True: # Keep trying until the webpage successfully downloads
  try:
    page = requests.get(url, timeout=60)
    break # If it downloads, get out and get on with life
  except requests.exceptions.RequestException as e: # If it doesn't download after the timeout period, an exceptions is thrown, and we try again
    print('Try again')
    pass

[cyberblitz]

The timeout, I believe, is not how most think it works. The timeout variable default setting is infinity, its the server you're requesting that will send the timeout message. Setting it to 60 means my code will wait 60 secs then stop, but the server side can still send timeout messages before then, though

I have made some adjustments to the code, thus:

The code still stops without any errors, though

import requests
from time import sleep
import time
import datetime
#import I2C_LCD_Driver
#mylcd = I2C_LCD_Driver.lcd()

# api URL
rescue_url = "https://someapi.com/rescues"
headers = {"Authorization": "Bearer Key"}

def get_data(url):
    try:
        data = requests.get(url, headers=headers).json()
        data = data['data']
        return [dta['attributes'] for dta in data if dta['attributes']['status'] == 'open']
    except requests.exceptions.RequestException:
        sleep(5)
        data = requests.get(url, headers=headers).json()
        data = data['data']
        return [dta['attributes'] for dta in data if dta['attributes']['status'] == 'open']


# function to convert string datetime value when rescue commenced (usually entered by Mecha)
def dte_convert(dte):
    d1 = datetime.datetime.strptime(dte, "%Y-%m-%dT%H:%M:%S.%fZ") + datetime.timedelta(hours=10)
    d2 = datetime.datetime.now()  # set variable to current dateTime
    dur_s = (d2 - d1).total_seconds()  # similar to datediff and calculates to seconds
    dy = divmod(dur_s, 86400)
    hrs = divmod(dy[1], 3600)
    mins = divmod(hrs[1], 60)
    ret = '%d d %d h %d m' % (dy[0], hrs[0], mins[0])
    return ret


while True:
    json_data = get_data(rescue_url)

    if json_data:
        for each in json_data:

            client = each['client']
            platform = each['platform']
            code_red = each['codeRed']
            system = each['system']
            active = dte_convert(each['createdAt'])

            print("C: %s" % client)
            print("P: %s" % platform + " CR: %s" % code_red)
            print("S: %s" % system)
            print("A: %s" % str(active))

#            mylcd.lcd_display_string("C: %s" %client ,1)
#            mylcd.lcd_display_string("P: %s" %platform + "CR: %s" %str(cde_red), 2)
#            mylcd.lcd_display_string("S: %s" %system, 1)
#            mylcd.lcd_display_string("A: %s" %str(res_datetime), 2)

    else:
            print("Date: %s" % time.strftime("%d/%m/%Y"))
            print("Time: %s" % time.strftime("%H:%M"))
            print('No Active Rescues')

    sleep(5)

#         #   mylcd.lcd_display_string("Date: %s" %time.strftime("%d/%m/%Y"), 1)
#         #   mylcd.lcd_display_string("Time: %s" %time.strftime("%H:%M:%S"), 2)
#           # mylcd.lcd_display_string('No Active Rescues', 3)

[cyberblitz]

I suppose my biggest question is how do I keep the code running to display the clock? with seconds and all, while at the same time, checking to see if there's any data to display.

I find that because of the linear nature of the code, I can't get the clock to run without it being inturrpted by the code checking for data.

Is there a way to do have the clock running smoothly while at the same time checking for data without inturruptions?

[metulburr]

I would use the threading module to create 2 threads, one to handle time while the other handles the data
https://pymotw.com/3/threading/index.html

[cyberblitz]

Thanks. This is my code now.

Any ideas on how to improve it would be much appreciated..

import requests
from time import sleep
import time
import datetime
import threading

#import I2C_LCD_Driver
#mylcd = I2C_LCD_Driver.lcd()

#  api URL
rescue_url = "https://api.somesite.com/rescues"
headers = {"Authorization": "Bearer Key"}
json_data = []


def lcd_enter(ln1, ln2, ln3, ln4):
#  mylcd.lcd_display_string(ln1 ,1)
#  mylcd.lcd_display_string(ln2, 2)
#  mylcd.lcd_display_string(ln3, 1)
#  mylcd.lcd_display_string(ln4, 2)

    print(ln1)
    print(ln2)
    print(ln3)
    print(ln4)


def get_data(url):
    try:
        data = requests.get(url, headers=headers).json()
        data = data['data']
        return [dta['attributes'] for dta in data if dta['attributes']['status'] == 'open']
    except requests.exceptions.RequestException:
        sleep(5)
        data = requests.get(url, headers=headers).json()
        data = data['data']
        return [dta['attributes'] for dta in data if dta['attributes']['status'] == 'open']


# function to convert string datetime value when rescue commenced (usually entered by Mecha)
def dte_convert(dte):
    d1 = datetime.datetime.strptime(dte, "%Y-%m-%dT%H:%M:%S.%fZ") + datetime.timedelta(hours=10)
    d2 = datetime.datetime.now()  # set variable to current dateTime
    dur_s = (d2 - d1).total_seconds()  # similar to datediff and calculates to seconds
    dy = divmod(dur_s, 86400)
    hrs = divmod(dy[1], 3600)
    mins = divmod(hrs[1], 60)
    ret = '%d d %d h %d m' % (dy[0], hrs[0], mins[0])
    return ret


def clock():
    while True:
        if not json_data:
            lcd_enter("Date: %s" % time.strftime("%d/%m/%Y"),
                      "Time: %s" % time.strftime("%H:%M:%S"),
                      'No Active Rescues',
                      '')
            sleep(1)
            

def main():
    while True:
        global json_data
        json_data = get_data(rescue_url)

        if json_data:
            for each in json_data:
                lcd_enter("C: %s" % each['client'],
                          "P: %s" % each['platform'] + " CR: %s" % each['codeRed'],
                          "S: %s" % each['system'],
                          "A: %s" % dte_convert(each['createdAt']))
        sleep(5)


t_main = threading.Thread(name='t_main', target=main)
t_clock = threading.Thread(name='t_clock', target=clock)
t_main.start()
t_clock.start()

[cyberblitz]

I am now finding, because of threading (potentially), that the lcd displays jibberish. It works fine at first, displaying the clock as expected, but as soon as data comes in it goes jibberish and stays that way untill I restart it (going back to the clock doesn't work). The clear_lcd method doesn't seem to help either.

I have read sending more than 1 block of messages to the lcd would do this but as far as I can tell, the code doesnt, at least when i use print when testing it works fine.

Anyone have any ideas?

[metulburr]

you could try the multiprocessing module as well
https://docs.python.org/3/library/multiprocessing.html

EDIT:
I forgot you are using a rasp pi. I dont use them so i am not familiar with their limitations on hardware or... to be honest anything about them.