[Tkinter] scheduling and timining issues using mqtt and tkinter

[kiko4590]

Please be patient with me as I am a newbie inpython and I really need your help. I have a custom GUI created in tkinter using Python3.7 on Raspbian (fork of Debian-Linux for Raspberri Pi). the code is meant to receive MQTT messages (two temp value and on humidity value) from an ESP32 device every 2 seconds, update the widgets in the GUI and send the values to mydevices (an IoT platform).

import subprocess  
import cayenne.client  
import socket  
from tkinter import *  
from tkinter import messagebox  
import tkinter  
import serial  
import struct  
import datetime  
import gaugelib  
import codecs  
import paho.mqtt.client as mqtt #import the client1  
import json  
import logging  
import mqtthandler  
import ast  
from tkinter import ttk  
import threading  
from threading import Thread  
import tkinter as tk  
def monitor():  
    global x  
    client.loop()  
    mqtt_client.loop_start()  
    room_temp1 = read_external_temp_hum()[2]  ## Value sent from ESP is forwarded to mydevices  
    room_temp2 = read_external_temp_hum()[0]    ## Value sent from ESP is forwarded to mydevices  
    room_humidity = read_external_temp_hum()[1] ## Value sent from ESP is forwarded to mydevices  
    conc= read_CO2()  
    mqtt_client.loop_start()   
    client.loop()  
    check_wifi_connection()  
    check_cable_connection()  
    p1.set_value(int(room_temp1))   
    p2.set_value(int(room_humidity))  
    p3.set_value(int(conc))  
    p4.set_value(int(room_temp2))  
    print(room_temp1)  
    print(room_temp2)  
    print(room_humidity)        
    x=x+1  
    if x>100:  
        x=0   
    root.after(10000, monitor)  
#def run_client():  
#    threading.Timer(1.0, run_client).start()  
#    client.loop()  
#def run_mqtt_client():  
#    threading.Timer(0.1, run_client).start()  
#    mqtt_client.loop_start()  
#run_client()  
#run_mqtt_client()  
root.geometry('%dx%d' % (root.winfo_screenwidth(), root.winfo_screenheight()))  
root.title('testGUI')  
root.configure(bg='black')  
createGaugeWidgets()  
ButtonWidgets()  
monitor()  
root.mainloop()

The program works well when adjusting the time into the after method and putting the client_loop and the mqtt_client.loop.start() into the monitor function, but cannot send command from the mydevices panel. The commands are receivedI can see them into my console, but mydevices tell that there is a problem sending the command without any other explanation and from my console I can see the message received with a response error. please see the log below. The button press into the mydevices keep turning with nothing happening.

Error:
PUB v1/1xxxxxxxxxxxxxxxxxxxxxxxd/things/1xxxxxxxxxxxxxxxxxxxxxxxd/data/1 temp,c=34.0
RCV v1/1xxxxxxxxxxxxxxxxxxxxxxxd/things/1xxxxxxxxxxxxxxxxxxxxxxxd/cmd/8 b'GwquHIyGGpL647y,1'
message received: {'msg_id': 'GwquHIyGGpL647y', 'client_id': '1xxxxxxxxxxxxxxxxxxxxxxxd', 'channel': 8, 'value': '1', 'topic': 'cmd'}
PUB v1/1xxxxxxxxxxxxxxxxxxxxxxxd/things/1xxxxxxxxxxxxxxxxxxxxxxxd/data/8 Digital,d=0
{'msg_id': 'GwquHIyGGpL647y', 'client_id': '1d285f40-2948-11e9-a08c-c5a286f8c00d', 'channel': 8, 'value': '1', 'topic': 'cmd'}
PUB v1/1xxxxxxxxxxxxxxxxxxxxxxxd/things/1xxxxxxxxxxxxxxxxxxxxxxxd/response error,GwquHIyGGpL647y={'msg_id': 'GwquHIyGGpL647y', 'client_id': '1xxxxxxxxxxxxxxxxxxxxxxxd', 'channel': 8, 'value': '1', 'topic': 'cmd'}
room_temp1 34.0
room_temp2 27.0
humidity 14.0
message received  {"MacAddress":"A4:CF:12:D9:15:FB","temperature":25.00,"humidity":30.00,"int_temperature":35.00,"RSSI":"-48"}

When threading the two MQTT loops another phenomena is happening and the mydevices console got disconnected and an exception error is raised into my console please see the log below.

Error:
PUB v1/1xxxxxxxxxxxxxxxxxxxxxxxd/things/1xxxxxxxxxxxxxxxxxxxxxxxd/sys/model Python
PUB v1/1xxxxxxxxxxxxxxxxxxxxxxxd/things/1xxxxxxxxxxxxxxxxxxxxxxxd/sys/version 1.1.0
Error: Unrecognised command 0
Exception in thread Thread-257:
Traceback (most recent call last):
  File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.5/threading.py", line 1180, in run
    self.function(*self.args, **self.kwargs)
  File "test_update.py", line 216, in run_client
    client.loop()
  File "/usr/local/lib/python3.5/dist-packages/cayenne_mqtt-1.1.0-py3.5.egg/cayenne/client.py", line 122, in loop
    self.client.loop()
  File "/usr/local/lib/python3.5/dist-packages/paho_mqtt-1.4.0-py3.5.egg/paho/mqtt/client.py", line 1072, in loop
    rc = self.loop_read(max_packets)
  File "/usr/local/lib/python3.5/dist-packages/paho_mqtt-1.4.0-py3.5.egg/paho/mqtt/client.py", line 1374, in loop_read
    rc = self._packet_read()
  File "/usr/local/lib/python3.5/dist-packages/paho_mqtt-1.4.0-py3.5.egg/paho/mqtt/client.py", line 2071, in _packet_read
    rc = self._packet_handle()
  File "/usr/local/lib/python3.5/dist-packages/paho_mqtt-1.4.0-py3.5.egg/paho/mqtt/client.py", line 2568, in _packet_handle
    return self._handle_suback()
  File "/usr/local/lib/python3.5/dist-packages/paho_mqtt-1.4.0-py3.5.egg/paho/mqtt/client.py", line 2701, in _handle_suback
    (mid, packet) = struct.unpack(pack_format, self._in_packet['packet'])
struct.error: bad char in struct format
Disconnected with result code 2

I please need your help to understand how the timing is managed in tkinter with the MQTT loops, afetr methods, threading, I am just confused and need to understand what is going on. My understanding is that what I am doing is fine, the monitor function is refreshed every 20s and when uncommenting the threads they are just running concurrently in background.

It was too long, my apologies.

Thank you in advance
Renaud

[deanhystad]

This looks interesting.

Tkinter has its own event- and GUI-update- loop (mainloop()). Mqtts start_loop() will block Tkinters mainloop() and vice versa.

General ways to solve this are

Polling

If you can use Mqtt without its loop but with manual polling, then establish an userloop under control of Tkinters mainloop with after() method. In said userloop poll messages from mqtt.

Multithreading or multiprocessing

If you must use Mqtts loop put all mqtt stuff in a thread or process and communicate with Tkinter over Queues (unidirectional) or Pipes (bidirectional). Tkinter can handle this communication in userloop explained above in minimal intervals of few 10 ms.

What i read briefly from IMHO you have both options (Other loop...() functions are available that give a threaded interface and a manual interface)
https://pypi.org/project/paho-mqtt/

[kiko4590]

This looks interesting.

Tkinter has its own event- and GUI-update- loop (mainloop()). Mqtts start_loop() will block Tkinters mainloop() and vice versa.

General ways to solve this are

Polling

If you can use Mqtt without its loop but with manual polling, then establish an userloop under control of Tkinters mainloop with after() method. In said userloop poll messages from mqtt.

Multithreading or multiprocessing

If you must use Mqtts loop put all mqtt stuff in a thread or process and communicate with Tkinter over Queues (unidirectional) or Pipes (bidirectional). Tkinter can handle this communication in userloop explained above in minimal intervals of few 10 ms.

What i read briefly from IMHO you have both options (Other loop...() functions are available that give a threaded interface and a manual interface)
https://pypi.org/project/paho-mqtt/

Thanks a lot for your answer, as you can see in my code I already tried multithreading (the part of my code which is commented ) and I got the errors above.

#def run_client():  
#    threading.Timer(1.0, run_client).start()  
#    client.loop()  
#def run_mqtt_client():  
#    threading.Timer(0.1, run_client).start()  
#    mqtt_client.loop_start()  
#run_client()  
#run_mqtt_client()  

The issue is that temperature, humidity... values are all sent correctly to my devices, but the commands which are sent from mydevices are received with a response error as you can see in the log I previously sent. The button from mydevices keep running (turning) without any change into the interface. When I change delays, things change and works correctly.

Error:
PUB v1/1xxxxxxxxxxxxxxxxxxxxxxxd/things/1xxxxxxxxxxxxxxxxxxxxxxxd/sys/model Python
PUB v1/1xxxxxxxxxxxxxxxxxxxxxxxd/things/1xxxxxxxxxxxxxxxxxxxxxxxd/sys/version 1.1.0
Error: Unrecognised command 0
Exception in thread Thread-257:
Traceback (most recent call last):
  File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.5/threading.py", line 1180, in run
    self.function(*self.args, **self.kwargs)
  File "test_update.py", line 216, in run_client
    client.loop()
  File "/usr/local/lib/python3.5/dist-packages/cayenne_mqtt-1.1.0-py3.5.egg/cayenne/client.py", line 122, in loop
    self.client.loop()
  File "/usr/local/lib/python3.5/dist-packages/paho_mqtt-1.4.0-py3.5.egg/paho/mqtt/client.py", line 1072, in loop
    rc = self.loop_read(max_packets)
  File "/usr/local/lib/python3.5/dist-packages/paho_mqtt-1.4.0-py3.5.egg/paho/mqtt/client.py", line 1374, in loop_read
    rc = self._packet_read()
  File "/usr/local/lib/python3.5/dist-packages/paho_mqtt-1.4.0-py3.5.egg/paho/mqtt/client.py", line 2071, in _packet_read
    rc = self._packet_handle()
  File "/usr/local/lib/python3.5/dist-packages/paho_mqtt-1.4.0-py3.5.egg/paho/mqtt/client.py", line 2568, in _packet_handle
    return self._handle_suback()
  File "/usr/local/lib/python3.5/dist-packages/paho_mqtt-1.4.0-py3.5.egg/paho/mqtt/client.py", line 2701, in _handle_suback
    (mid, packet) = struct.unpack(pack_format, self._in_packet['packet'])
struct.error: bad char in struct format
Disconnected with result code 2