Program running on RPi 3b+ Very Strange Behavior - Out of Bound Index

[MadMacks]

So, new to Python and still trying to learn the scope of variables. But, I have a small program that uses Bluetooth (BLE) to connect to a LiFePO4 battery management system controller, read some data, then publish a MQTT message.

Here's the very odd thing. When I was testing last night, and learning variable scope, I kept getting an index out of range error while unloading a list. So I put in a bunch of print statements to try and see what the heck was happening, as I think I was using the list properly (maybe not in the most efficient way, but should work).

When I did that and reloaded to the RPI the script ran fine with no range error! This confused me to know end. I then commented out the print statements and then I got the range error again. WTF? Here is the code if you have the time to help...

I've labeled the test print statements with # TEST.. If they are used the code runs. If they are commented out the code throws an error at the line marked # ERROR LINE.

I'm specifically looking for feedback in the error and why commenting out print statements would cause a problem. If you have style or "it's better to do it this way" comments, feel free. If you just want to tell me I'm a crappy Python programmer, save your breath, I already know that!

#!/usr/bin/env python3

# This code should be run as SUDO as it does low level BLE scanning.
#
# Run it as follows:
#
#   sudo python jbd-bms-mqtt-name.py -n "Cabin 580ah 100a" -i 0 -t solar
#
# This will scan for a bluetooth device called "Cabin 580ah 100a" and extract its
# bluetooth MAC address, then use the address to attach to the BMS and read a set
# if variables, one time, and then publish a MQTT message with those variable to
# the Topic "solar".
#
# If you use a value other than 0 for the -i parm the program will loop forever
# and read the BMS values every X seconds based on the value you passed.

from bluepy.btle import Peripheral, DefaultDelegate, BTLEException
from bluepy.btle import Scanner, DefaultDelegate
import sys
import struct
import argparse
import json
import time
import binascii
import atexit
import paho.mqtt.client as paho


def disconnect():
    mqtt.disconnect()
    print("broker disconnected")


def cellinfo1(data):  # process pack info
    print("In cellinfo1...") #TEST
    infodata = data
    i = 4  # Unpack into variables, skipping header bytes 0-3
    volts, amps, remain, capacity, cycles, mdate, balance1, balance2 = struct.unpack_from('>HhHHHHHH', infodata, i)
    volts = volts / 100
    amps = amps / 100
    capacity = capacity / 100
    remain = remain / 100
    global ginfo
    ginfo.append(volts)
    ginfo.append(amps)
    ginfo.append(capacity)
    ginfo.append(remain)
    print(volts, amps, capacity, remain) # TEST
    print(ginfo) # TEST
    print("Leaving cellinfo1") # TEST


def cellvolts1(data):  # process cell voltages
    print("In cellvolts1...") # TEST
    celldata = data  # Unpack into variables, skipping header bytes 0-3
    i = 4
    cell1, cell2, cell3, cell4 = struct.unpack_from('>HHHH', celldata, i)
    global ginfo
    ginfo.append(cell1)
    ginfo.append(cell2)
    ginfo.append(cell3)
    ginfo.append(cell4)
    print(cell1, cell2, cell3, cell4) #TEST
    print(ginfo) #TEST
    print("Leaving cellvolts1") #TEST

class MyDelegate(DefaultDelegate):  # notification responses
    def __init__(self):
        DefaultDelegate.__init__(self)

    def handleNotification(self, cHandle, data):
        hex_data = binascii.hexlify(data)  # Given raw bytes, get an ASCII string representing the hex values
        text_string = hex_data.decode('utf-8')  # check incoming data for routing to decoding routines
        if text_string.find('dd04') != -1:  # x04
            print("handleNotification-dd04") # TEST
            cellvolts1(data)
        elif text_string.find('dd03') != -1:  # x03
            print("handleNotification-dd03") #TEST
            cellinfo1(data)


# Mainline Code
# Process Command Line Arguments
parser = argparse.ArgumentParser(description='Fetches and outputs JBD bms data')
parser.add_argument("-n", "--name", help="BLE Device Name", required=True)
parser.add_argument("-i", "--interval", type=int, help="Data fetch interval in seconds, 0=Once and Done", required=True)
parser.add_argument("-t", "--topic", help="MQTT Topic name", required=True)
args = parser.parse_args()
z = args.interval
topic = args.topic
bleName = args.name
ginfo = []
broker = "192.168.1.98"
port = 1883

print("Searching for = '%s'" % (bleName))

scanner = Scanner()
devices = scanner.scan(5)

for dev in devices:
    for (adtype, desc, value) in dev.getScanData():
        if value == bleName:
            print("=========== Found It ==============")
            print("Name = %s" % (value))
            print("Device = %s" % (dev.addr))
            print("RSSI = %d dB" % (dev.rssi))
            print("===================================")
            bleAddr = dev.addr


try:                    # If the variable was created we found the BLE device
    bleAddr
except NameError:
    sys.exit("Did not find the BLE device...Halting!")


try:
    print('attempting to connect')
    bms = Peripheral(bleAddr, addrType="public")
except BTLEException as ex:
    time.sleep(5)
    print('2nd try connect')
    bms = Peripheral(bleAddr, addrType="public")
except NameError:
    sys.exit("Can not connect to the BLE device...Halting!")
else:
    print('connected ', bleAddr)

atexit.register(disconnect)
mqtt = paho.Client("control3")  # create and connect mqtt client
mqtt.connect(broker, port)
bms.setDelegate(MyDelegate())  # setup bt delegate for notifications

# write empty data to 0x15 for notification request   --  address x03 handle for info & x04 handle for cell voltage
# using waitForNotifications(5) as less than 5 seconds has caused some missed notifications
while True:
    print("calling bms-x03") # TEST
    result = bms.writeCharacteristic(0x15, b'\xdd\xa5\x03\x00\xff\xfd\x77', False)  # write x03 w/o  cell info
    print(result) # TEST
    bms.waitForNotifications(5)
    print("calling bms-x04") # TEST
    result = bms.writeCharacteristic(0x15, b'\xdd\xa5\x04\x00\xff\xfc\x77', False)  # write x04 w/o  cell voltages
    print(result) #TEST
    bms.waitForNotifications(5)
    print(ginfo) # TEST
    gvolts = ginfo[0]
    gamps = ginfo[1]
    gcapacity = ginfo[2]
    gremain = ginfo[3]
    gcellvolt1 = ginfo[4]  # ERROR LINE
    gcellvolt2 = ginfo[5]
    gcellvolt3 = ginfo[6]
    gcellvolt4 = ginfo[7]
    message0 = {
        "topic": topic,
        "volts": gvolts,
        "amps": gamps,
        "capacity": gcapacity,
        "remain": gremain,
        "cell1": gcellvolt1,
        "cell2": gcellvolt2,
        "cell3": gcellvolt3,
        "cell4": gcellvolt4
    }
    ret = mqtt.publish(topic, payload=json.dumps(message0), qos=0, retain=False)

    if z != 0:  # Loop unless interval is 0 otherwise one and done
        time.sleep(z)
    else:
        break

[MadMacks]

Here are the console message I'm getting depending of if the print statements are live of commented out...

Output:
With comments….

Searching for = 'Cabin 560ah 100a'
=========== Found It ==============
Name = Cabin 560ah 100a
Device = a4:c1:38:f4:bd:25
RSSI = -55 dB
===================================
attempting to connect
connected  a4:c1:38:f4:bd:25
calling bms-x03
{'rsp': ['wr']}
handleNotification-dd03
In cellinfo1...
13.55 60.33 560.0 503.67
[13.55, 60.33, 560.0, 503.67]
Leaving cellinfo1
calling bms-x04
{'rsp': ['wr']}
handleNotification-dd04
In cellvolts1...
3387 3387 3386 3391
[13.55, 60.33, 560.0, 503.67, 3387, 3387, 3386, 3391]
Leaving cellvolts1
[13.55, 60.33, 560.0, 503.67, 3387, 3387, 3386, 3391]
broker disconnected


Without Comments…
Searching for = 'Cabin 560ah 100a'
=========== Found It ==============
Name = Cabin 560ah 100a
Device = a4:c1:38:f4:bd:25
RSSI = -55 dB
===================================
attempting to connect
connected  a4:c1:38:f4:bd:25

Traceback (most recent call last):
  File "/home/pi/Scripts/bms-main/testcode.py", line 153, in <module>
    gcellvolt1 = ginfo[4]
IndexError: list index out of range
broker disconnected

[MadMacks]

How in the heck can commenting out print lines cause this error?

[MadMacks]

So I've gone thru the print statements line by line and have determined that the issue is being triggered in this bit of the code...

def cellinfo1(data):  # process pack info
    print("In cellinfo1...")                         # this is the trigger comment/print
    infodata = data
    i = 4  # Unpack into variables, skipping header bytes 0-3
    volts, amps, remain, capacity, cycles, mdate, balance1, balance2 = struct.unpack_from('>HhHHHHHH', infodata, i)
    volts = volts / 100
    amps = amps / 100
    capacity = capacity / 100
    remain = remain / 100
    global ginfo
    ginfo.append(volts)
    ginfo.append(amps)
    ginfo.append(capacity)
    ginfo.append(remain)
    print(volts, amps, capacity, remain)     # this is the trigger comment/print
    print(ginfo)                                       # this is the trigger comment/print
    # print("Leaving cellinfo1")

If this function is used as displayed above the code runs fine.

If I comment out any one of the "print(volts, amps, capacity, remain)" line or the "print(ginfo)" line or the print("In cellinfo1...") lines I get the out of bound error further down in the code...

So it seems like there's something in the print statements that resolves something with the global ginfo list. Im completely stumped so PLEASE HELP IF YOU CAN!!!

[MadMacks]

Dang it...

So I can run the code and it works some times...and the same code run again throws the index error...

I believe the issue is being caused by either the global defined ginfo list, or some stupid issue with python running on my RPI 3B+ (1 gb version).

Maybe I'll just use Perl or some other language (come on, tell me that's stupid and help solve this!)...

[deanhystad]

Maybe your writes/notify fails. If you only get one data back, the info array is only 4 long, and info[4] is out of range.

Why aren't you saving your data in your delegate? Something like this:

class MyDelegate(DefaultDelegate):
    def __init__(self):
        super().__init__()
        self.dd03_updates = 0
        self.dd04_updates = 0
        # If you need data as a dictionary, may as well save it as a dictionary.
        self.data = {
            "volts": None,
            "amps": None,
            "capacity": None,
            "remain": None,
            "cell1": None,
            "cell2": None,
            "cell3": None,
            "cell4": None,
        }
 
    def handleNotification(self, cHandle, data):
        hex_data = binascii.hexlify(data)
        if b'dd04' in hex_data:
            self.dd04_messages += 1
            self.data.update(zip(
                ("volts", "amps", "capacity", "remain"),
                struct.unpack_from('>HHHH', data, 4)
            ))
        elif b'dd03' in hex_data:
            self.dd03_messages += 1
            self.data.update(zip(
                ("cell1", "cell2", "cell3", "cell4"),
                struct.unpack_from('>HHHH', data, 4)
            ))
 
    def __str__(self):
        return "\n\t".join((
            f'Messages  dd03:{self.dd03_messages}, dd04{self.dd04_messages}',
            *[f'{key:10} = {value}' for key, value in self.data.items()]
        ))
 
delegate = MyDelegate()
bms.setDelegate(delegate)  # setup bt delegate for notifications
 
. . .

while True:
    bms.writeCharacteristic(0x15, b'\xdd\xa5\x03\x00\xff\xfd\x77', False)
    bms.waitForNotifications(5)
    bms.writeCharacteristic(0x15, b'\xdd\xa5\x04\x00\xff\xfc\x77', False)
    bms.waitForNotifications(5)
    print(delegate)


    ret = mqtt.publish(topic, payload=json.dumps(delegate.data), qos=0, retain=False)

[MadMacks]

Well...

First, thank you for the reply as it's appreciated...

Second, the reason is because I'm a complete newbie Python coder and haven't yet reached a level of understanding to even understand the above code...but I'll certainly give it a try, and try and learn from it...I'm diligently reading my O'Reilly Intro to Python book right now (sadly I bought it years ago and it's an older addition). Python is quite different from the other languages I've used so there's a bit of a learning curve for me...

But Thirdly, I'm trying to understand, given the code I posted, why the addition of Print statements or the lack thereof can cause random failures when the code is executed. I'm assuming it's because of how variable scope declarations are handled in Python, but I'm not sure.

I'm trying to build a system that will monitor 4 of my off grid buildings, each with its own solar and battery system. Right now I have to walk to within Bluetooth range of each building (spread over 40 acres) and connect to the BMS' with a phone app, to get readings. All these building are interconnected on my LAN (some wired, some microwave linked). I found a potential solution for bluetooth connectivity using RPis on GitHub (username tgalarneau, project BMS) and have tweaked his code base for my purposes. The idea is to use an existing RPi in each building to attach and read the BMS data every 5 minutes and publish that data to an MQTT broker in the main building. I'll then grab that JSON based MQTT data and push it into my Maria/MySQL DB and then present a unified view of all the systems using Grafana. I'm more familiar with all of that process and less with Python, but heck, I like to learn...and I'm snowed in now anyway...

The borrowed code was writing multiple messages to MQTT and I wanted to unify that into a single message containing the data I want to track, so I pulled the MQTT post process out of the original individual functions (and deleted a couple I didn't need at all to streamline things) and thought I could use a global list to hold the values retrieve by each function that I want to use, and then have a singled MQTT post process. So, I've kind of hacked tgalarneau's code without really understanding its finer points.

Your code dealing with "delegates" is new to me so I guess I need to dig into that a lot more to understand why the "tgalarneau" code was set up the way it was.

So, do you see anything in the code I have that would be causing a seemingly random "works/fails" situation and the "list index out of range" issue?

Again, thanks for the reply...

[MadMacks]

I swapped in the code you provided, as best as i could understand it.

But it throws an error here...

    def __str__(self):
        return "\n\t".join((
            f'Messages  dd03:{self.dd03_messages}, dd04{self.dd04_messages}',
            *[f'{key:10} = {value}' for key, value in self.data.items()]
            f'volts    = {self.volts}',
        ))

Error:
  File "/home/pi/Scripts/bms-main/testcode2.py", line 107
    f'volts    = {self.volts}',
    ^
SyntaxError: invalid syntax

Know idea how that value parser command works (looking though) so I have know idea why it's invalid syntax...

Yoriz write Mar-06-2023, 06:15 AM:
Please post all code, output and errors (in their entirety) between their respective tags. Refer to BBCode help topic on how to post. Use the "Preview Post" button to make sure the code is presented as you expect before hitting the "Post Reply/Thread" button.

[deanhystad]

When printing changes how things run, I first think of some sort of race condition. But you are waiting 5 seconds for a response, so that isn't likely. I think it more likely that printing didn't make any difference at all and it was just chance, and maybe tainted observation, that lead you to believe that printing was "fixing" the problem. That happens to me all the time. Your mind creates patterns regardless of the data. Sometimes this is insight and sometimes it is "That cloud looks like a bunny." You should really check the value retuned by the waitForNotifications(). If it is False, nothing was received.

I cannot test the code, so I am not surprised there are some bugs. This particular bug is a cut/paste error. This does not belong.

f'volts = {self.volts}',

I updated my original post.

I will try to describe the code.

You sublcassed bluepy.btle.DefaultDelegate to make your MyDelegate class. Then you provided a custom handleNotification() method to use when a notification is received. In addition to custom methods, you can also provide custom attributes. Instead of making a global variable and having the delegate copy data to the global variable, it makes more sense to have the delegate keep the data. That is what I was doing by defining a bunch of attributes in the __init__() method. When MyDelegate's handleNotification() method is called, it stores the notification data in the delegate's instance attributes.

I saw that elsewhere in your code you were publishing a dictionary. I thought it made sense to save the notification data as a dictionary. That is why I did this in the __init__() method.

        self.data = {
            "volts": None,
            "amps": None,
            "capacity": None,
            "remain": None,
            "cell1": None,
            "cell2": None,
            "cell3": None,
            "cell4": None,
        }

The data starts out as None, but as notifications are received, the dictionary is updated.

This is just a shorthand way of updating a dictionary.

            self.data.update(zip(
                ("volts", "amps", "capacity", "remain"),
                struct.unpack_from('>HHHH', data, 4)
))

It is the same as doing this:

volts, amps, capacity, remain= struct.unpack_from('>HHHH', data, 4)
self.data["volts"] = volts
self.data["amps"] = amps
self.data["capacity"] = capacity
self/data["remain"] = remain

And this is called unpacking:

volts, amps, capacity, remain= struct.unpack_from('>HHHH', data, 4)

The first value returned by struct.unpack_from() is assigned to volts, the second to amps, the third to capacity, and fourth to remain. Packing and unpacking are very useful in Python. Well worth mastering.

Let me know if you have more questions.

[MadMacks]

That is very very helpful for me, I understand a lot more thanks to the last post. Thank you.

One quick question… in the last part of your post you have…

volts, amps, capacity, remain= struct.unpack_from('>HHHH', data, 4)
self.data["volts"] = volts
self.data["amps"] = amps
self.data["capacity"] = capacity
self/data["remain"] = remain

Is the last element supposed to be

self.data["remain"] = remain

Without the slash, or is there some meaning for the slash? I assume it’s a typo…

It’s been interesting trying to take someone else’s code, in a language I haven’t used before, and try to refactor it to meet my specific needs. The original code (built by tgalarneau) was built in a particular way for his purposes, and I have no idea why he made the programming style choices he did (but I’m grateful he wrote it anyway). So, hacking it up to meet my needs and trying to make it use a different python style at the same time is confusing for a python beginner. Your thoughtful explanations are really helping.

This “Python thing” looks like it just might be a good language to learn!