Issue in changing data format (2 bytes) into a 16 bit data.

[GiggsB]

UPDATED THE QUESTION

Hello,

I am writing this python code for Raspberry Pi (rpi) where RPi is connected to a microcontroller (mcu). The mcu is sending data to rpi through SPI communication protocol. The mcu acts as the slave device and rpi as the master device.

Every time mcu wants to send any data, it sends out a high signal (or an interrupt) to rpi and waits for the rpi's response. Rpi then sends 0x2 after which rpi starts data collection. On rpi, the first time when data comes, it opens a csv file in 'w' mode and stores the data into csv file. And next time whenever data comes, I want rpi to store data in the same file and start from the position after the last data was written.

Data format: I am sending data in the form of bytes from the mcu. On Rpi, I want to combine every two 8-bit data (two bytes) to form into a 16-bit data and ignore the next two data bytes and store the combined 16-bit data in the file in every new line. For ex., if there is data as "data[0], data[1],data[2], data[3],data[4], data[5],data[6], data[7],data[8], data[9]...", then I want to use "data[0], data[1]" to form the first 16-bit data, then ignore "data[2], data[3]" and so on.

I have written some code, but I am unable to update it according to my requirement. I have 2 options:
a. Either every 4 data bytes, combine it to form in to 32-bit data and right shift the data by 16 bits and then store into file. OR
b. Get only every 2 data bytes, combine them and store it into file and ignore the next 2 data bytes and so on.

Can someone suggest which one would be better and can help me with the same? Thanks.

import datetime 
import os 
import struct 
import time 
import pigpio 
import spidev 
import zlib

INTERRUPT_GPIO=25

bus = 0
device = 0
spi = spidev.SpiDev()
spi.open(bus, device)
spi.max_speed_hz = 4000000
spi.mode = 0

pi = pigpio.pi()
if not pi.connected:
    exit()

pi.set_mode(INTERRUPT_GPIO, pigpio.INPUT)
 
def output_file_path():
    return os.path.join(os.path.dirname(__file__),
               datetime.datetime.now().strftime("%dT%H.%M.%S") + ".csv")
 
def spi_process(gpio,level,tick):
    data = bytes([0]*1024)
    spi.xfer2([0x02])
    for x in range(1):
        f.truncate()
        recv = spi.xfer2(data)
        values = struct.unpack("<" +"I"*256, bytes(recv))
        f.write("\n")
        f.write("\n".join([str(x) for x in values]))

input("Press Enter to start the process ")
spi.xfer2([0x01])
f = open(output_file_path(), 'w')

cb1=pi.callback(INTERRUPT_GPIO, pigpio.RISING_EDGE, spi_process)

while True:
    time.sleep(1)

[rob101]

And next time whenever data comes, I want rpi to store data in the same file and start from the position after the last data was written.

From what I can see, I think that the issue you have is that you're oping the file with access mode 'w' -- Write: Create a new file or overwrite the contents of an existing file.

What you need is access mode 'a' -- Append: Write data to the end of an existing file. This mode will also create a file if the specified file does not exist.

[GiggsB]

[/quote]
What you need is access mode 'a' -- Append: Write data to the end of an existing file. This mode will also create a file if the specified file does not exist.
[/quote]

Hi Rob, when I change the 'w' to 'a', nothing is getting stored into the file. When I change it back to 'w', I can see the data again. Also, I have updated my question a bit, if you can take a look again and help me with the same.

[rob101]

... if you can take a look again and help me with the same.

No worries.

I can't see a 'f.close()' in your code and I can't see that you're using the 'with open' option, but you'll need one or the other; not both.

[GiggsB]

Rob,

Thanks for the suggestion. However, I was trying to play with the 'w' itself and I used 'f.truncate()' method while using the 'w' mode and I am getting the operation as expected.

For my second issue, I was planning to right shift the data by 16 bits...but I get below error (below is also the modified code):

import datetime 
import os 
import struct 
import time 
import pigpio 
import spidev 
import zlib

INTERRUPT_GPIO=25

bus = 0
device = 0
spi = spidev.SpiDev()
spi.open(bus, device)
spi.max_speed_hz = 4000000
spi.mode = 0

pi = pigpio.pi()
if not pi.connected:
    exit()

pi.set_mode(INTERRUPT_GPIO, pigpio.INPUT)
 
def output_file_path():
    return os.path.join(os.path.dirname(__file__),
               datetime.datetime.now().strftime("%dT%H.%M.%S") + ".csv")
 
def spi_process(gpio,level,tick):
    data = bytes([0]*2048)
    spi.xfer2([0x02])
    for x in range(1):
        recv = spi.xfer2(data)
        values = struct.unpack(">" +"I"*512, bytes(recv))
        values = values>>16
        f.write("\n")
        f.write("\n".join([str(x) for x in values]))
        #f.seek()
        f.truncate()
input("Press Enter to start the process ")
spi.xfer2([0x01])
f = open(output_file_path(), 'w')

cb1=pi.callback(INTERRUPT_GPIO, pigpio.RISING_EDGE, spi_process)

while True:
    time.sleep(1)

And error that I get is:

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3/dist-packages/pigpio.py", line 1213, in run
    cb.func(cb.gpio, newLevel, tick)
  File "2Knowles_Final_spi_crc_zlib_getting_data_from_mcu_again.py", line 34, in spi_process
    values = values>>16
TypeError: unsupported operand type(s) for >>: 'tuple' and 'int'

Can you suggest what can I do?

Thanks.

[rob101]

I'm pleased for you, that you've sorted the file issue.

I'm no expert with Python; I know enough to do the basics, and them some, but when it comes to bit operations, I'm getting out for my depth, so I'll leave that to one of the many OPs here that have way more knowledge than I.

All the best with it; it looks to be a very interesting project

[GiggsB]

All the best with it; it looks to be a very interesting project

Thanks, Rob for the help!

Can anyone help me how to resolve the issue? Thanks.

[deanhystad]

Your use of truncate() confuses me. In your initial post you say:

it opens a csv file in 'w' mode and stores the data into csv file. And next time whenever data comes, I want rpi to store data in the same file and start from the position after the last data was written.

This happens automatically when you write to a file. There is no need to use truncate() to move the file position, it is already at the end of the file.
And this is what truncate() does.

truncate
Description
Truncates the file’s size.

Syntax
file. truncate([size])

size
Optional. If the optional size argument is present, the file is truncated to (at most) that size. The size defaults to the current position.
Remarks
The current file position is not changed.

Note that if a specified size exceeds the file’s current size, the result is platform-dependent: possibilities include that the file may remain unchanged, increase to the specified size as if zero-filled, or increase to the specified size with undefined new content. Availability: Windows, many Unix variants.

With no argument, truncate() does nothing. There is no need to call truncate() as all it does is confuse the readier into thinking you are resetting the file. It is not change the file position, and there is no need to change the file position since it after the last write.

[GiggsB]

it opens a csv file in 'w' mode and stores the data into csv file. And next time whenever data comes, I want rpi to store data in the same file and start from the position after the last data was written.

This happens automatically when you write to a file. There is no need to use truncate() to move the file position, it is already at the end of the file.

Hello deanhystad. Thank you for the insight. I have corrected it. Do you have knowledge about the second issue that I am having? Thanks.

[deanhystad]

What does struct.unpack() return? Have you read the documentation for the struct library? The answer is there.