How can I add an end line character after every value that I receive?

[GiggsB]

Hi,
I have the below python code. I am using the UART communication protocol to get data from a microcontroller. Currently, I am sending an end line character with each data from the microcontroller itself (For ex: printf("%d\n",data). But instead of sending from the microcontroller, I want to do this in the python script.

import time
import serial
import datetime
import os
  
ser = serial.Serial(port='/dev/serial0', baudrate=900000, parity=serial.PARITY_NONE,
        stopbits=serial.STOPBITS_ONE,
        bytesize=serial.EIGHTBITS, rtscts=1,)
  
def output_file_path():
    return os.path.join(os.path.dirname(__file__),
               datetime.datetime.now().strftime("%dT%H.%M.%S") + ".csv")
  
while (True):
    if ser.inWaiting():
        with open(output_file_path(), 'w') as text_file:
            while True:
                while not ser.inWaiting():
                    time.sleep(0.01)
                data_str = ser.read(ser.inWaiting()).decode('ascii')
                text_file.write(data_str)
                #text_file.write("\n") #this line I wrote to enter an end line character
                if "D" in data_str:
                    break
    time.sleep(0.01)

The issue is when I remove the end line character from my microcontroller's code and uncomment the above highlighted line, I receive data like this below picture. I am not sure why that is happening.
   

Expected:
   

Also, I believe, in my example above, "data" is actually a 32-bit integer but UART on microcontroller's side sends 8 bits at a time. Since I am just using an API microcontroller side for UART, I am not 100% sure about it. Can someone help?

Thank you.

[Larz60+]

you should only initialize ports once and close at the end of your session.
see: https://pyserial.readthedocs.io/en/lates...intro.html

[deanhystad]

An advantage to sending \n from the microcontroller is you can send multiple bytes terminated by \n and use readline() in your Python program instead of reading a byte at a time. But it looks like your end of transmission character is "D". If so, you can use readUntil(b'D').

That would simplify your loop:

while (True):
    if ser.inWaiting():
        with open(output_file_path(), 'w') as text_file:
            text_file.write(f"{ser.readUntil(b'D').decode('ascii')}\n")
    time.sleep(0.01)

You should set a timeout for the serial port to prevent it from hanging.

Were you receiving data when the microcontroller was appending \n to the data? It is implied, but not directly stated that the posted code worked before but stopped working when the newline was no longer sent.

I don't understand this:

"data" is actually a 32-bit integer but UART on microcontroller's side sends 8 bits at a time

Are you sending ascii or binary data? Your program uses .decode('ascii') which implies this is character data, not integers. If the microcontroller is sending integers you need to "decode" them differently.

[Jeff_t]

Would I be right to say that your data are 32 bit integer values transmitted as ASCII characters 0 to 9 and terminated with newline ('\n') and that at the end of the logging session you send a non numeric value ("D") to signal that logging is ended and break from the While loop. That is what I am seeing especially with printf("%d\n",data).

Are you logging just one sensor value or more.

You should be using readline() as deanhystad suggested.

[deanhystad]

Not readline. Only use readline if your data is terminated by a newline (\n). If you have a different termination character use readuntil().

If the data in the previous post is real, I have no idea what is being passed. 77-85 is upper case J-U. If it is 4 bytes for an integer, these are really big integers.

[Jeff_t]

I believe the data is terminated with newline due to the example he gave (For ex: printf("%d\n",data). The code he posted is receiving and decoding ASCII codes ? and printing them to file. Originally he left the newline untouched which formatted his csv file into columns, when he removed that format the data got placed on the spreadsheet in continuous rows, the fact that the column data got separated makes me think that those values each equate to the value of one 32 bit integer printf("%d\n",data). The purposed of the 'D' character is to break out of the while loop when he has finished his logging session not for separating samples. I also believe there may be a certain lack of understanding about the difference of reading bytes or bytes encoded as ASCII characters. Until he responds we wont know for certain, it would be nice to see what is being sent and compare to what is received, the 17 or so values that we see range from 77 to 85 it would be nice to know if that was expected.

[GiggsB]

Hi,
Thank you all for your replies. Since I was closer to my deadline of a milestone of the project, for now ...I used the code where MCU was sending an endline character.

Were you receiving data when the microcontroller was appending \n to the data?

Yes, I used the same code posted in the original question and that was working fine.

On looking deeper into the API being used on MCU, I got to know that data is being converted into 8 bits integer and transmitted. And thank you for the code, I will try using that.

@Jeff_t, No, the data is being transmitted as 8 bit integer. I am passing a 32 bit integer value in that "printf" statement, however, that 32 bit integer is being broken up into 8 bits and transmitted as 8 bits. Also, as I mentioned above, the code was working fine and those values ranging from in 70s were also expected.
in the 2nd picture, where, I receive data in continuous rows...I believe I was also expecting values like 4095,4094...you can see those are printed, so basically...I was receiving correct values but without any space/endline character to distinguish the values.

[GiggsB]

you should only initialize ports once and close at the end of your session.
see: https://pyserial.readthedocs.io/en/lates...intro.html

Hi,

Thanks for the suggestion. But I am a bit confused, I am only initializing the ports once I believe.
By "closing at the end of your session" do you mean after the loop breaks (or after the file is filled with data and closed)?
Or do you mean in general, like at the end of the code, when I am done with everything?
Basically, in my case, I am running the code until the battery pack (power bank connected as power supply to RPi) gets dead, so I believe all the ports get reset when RPi is powered up gain.
Please correct me if I am wrong somewhere. Thanks.

[Larz60+]

Thanks for the suggestion. But I am a bit confused, I am only initializing the ports once I believe.

I see that now, my mistake.

Not sure it's required, but probably should use serial.close() after you break from loop, or just prior to break.

[Jeff_t]

I have an example you can try thats based upon your original code and the fact you said the mcu appended a newline. One of the main differences is that this example uses a list of lists to store the incoming serial data, when you have all your data the csv module then writes the list of lists to a csv file.

from datetime import datetime
import time
import sys
import serial
import csv

ser=serial.Serial("COM7",115200)

list_cnt=0
input_string=""
Main_List=[]

def save_results():

	global Main_List

	with open('text_file.csv', "w",newline="") as file:
			csvwriter = csv.writer(file)
			csvwriter.writerows(Main_List)


def add_tolist(some_string):

	global list_cnt
	global Main_List

	today = datetime.now().strftime("%m/%d/%Y %H:%M:%S")
	final_string=today + ',' + some_string
	Main_List.append([])
	Main_List[list_cnt]=final_string.split(",")
	list_cnt+=1
	

while True:
	if ser.isOpen():

		input_string=ser.readline().strip().decode("utf-8")

	if input_string=='D':
		save_results()
		break

	if len(input_string)>0:
		add_tolist(input_string)

its a little raw but might give you something to work with