Python Forum
PC why serial code = converting to ascii ?
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
PC why serial code = converting to ascii ?
#1
hello,

I have written a piece of code in virtual studio code on the PC that stores the data received via the USB in an, I think an array, but I am not sure that it is an array.
when i take a look in the serial monitor from VS code The code also arrives as I sent it.
but when i look in the terminal, then the code appears to have changed to ascii.
Does anyone know how I can prevent this change to ascii?

The code (comments deleted):
import serial

k = 0

# Configure the serial connection
port = "com9" 
baudrate = 115200
serial_connection = serial.Serial(port, baudrate)

# Open a file on your computer to write the received data
destination_file = open("/Users/Beheerder/Dropbox/documenten arjo/proj raspberry pico/pico sending data to PC/store.txt", "wb")

while True:
    
    data = serial_connection.read(1000)

    print(data)
    destination_file.write(data)

    destination_file.flush() # the data is writing in to the destination_file after flush()
in serial monitor: (this is wat i need)
85
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
255
255
255

in terminal: (this is wat i not need)
b'85\r\n0\r\n0\r\n0\r\n0\r\n0\r\n0\r\n0\r\n0\r\n0\r\n0\r\n0\r\n0\r\n0\r\n0\r\n0\r\n0\r\n0\r\n0\r\n0\r\n0\r\n0\r\n0\r\n255\r\n255\r\n255\r
I'm not French, but from the netherlands
Reply
#2
Please post text, not screenshots.

Serial.read returns bytes. bytes is an array-like thing in python for small integers (range 0 to 255). When printing bytes, python uses visible ascii characters when possible. When the "byte" doesn't map to an ascii character, it is printed as an escape sequence.

You are wrong thinking the problem is that data is converted to ascii when written to the terminal. The problem is the opposite. To hide the \r (carriage return) and \n (newline) you need to convert the bytes to a string. It appears that data is an encoded string. To convert it to a str, you need to decode it.
data = b"""This is a
string encoded
as bytes."""
print(data)
as_str = data.decode("utf8")
print(as_str)
Output:
b'This is a\nstring encoded\nas bytes.' This is a string encoded as bytes.
Reply
#3
(Oct-07-2024, 02:28 PM)trix Wrote: but when i look in the terminal, then the code appears to have changed to ascii.
Does anyone know how I can prevent this change to ascii?

If you print() a bytes or a bytearray object, then any elements that are in the ascii range are rendered in ascii. Note that this is just the way print() behaves. The integers are still in the object.

If you don't want to see the ascii, then you can interate over the bytes/bytearray object and get the integers and print them instead.

Example:
>>> numeric_sequence = (0, 255, 65, 70, 75)
>>> b = bytes(numeric_sequence)
>>> b
b'\x00\xffAFK'
>>> for i in b:
...   print(i)
...
0
255
65
70
75
Reply
#4
thanks for the answers. I'm going to try that
I'm not French, but from the netherlands
Reply
#5
That actually works, thanks for that.
but now I see that I have a new problem. when I print that, each individual number is printed one below the other.
8
5
0
0
0
0
2
5
5
2
5
5
instead of:
85
0
0
0
0
255
255
later on, i want to do some calculations with these numbers.
Tnx.
I'm not French, but from the netherlands
Reply
#6
decode() is the correct solution for your issue. The decoded str will print the \r and \n as carriage return and linefeed instead of escape sequences.
Reply
#7
thanks I will take a look.
I'm not French, but from the netherlands
Reply
#8
"85", "0" and "255" are not numbers, they are strings. You'll need to use int(string) to convert them to numbers.
data = b"85\r\n0\r\n255\r\n244"
print("data", data)
as_str = data.decode("utf8")
print("as_str", as_str)
lines = as_str.splitlines()
print("lines", lines)
numbers = list(map(int, lines))
print("numbers", numbers)
Output:
data b'85\r\n0\r\n255\r\n244' as_str 85 0 255 244 lines ['85', '0', '255', '244'] numbers [85, 0, 255, 244]
If all you want are the integers you can skip a few steps because bytes also has a splitlines() method.
data = b"85\r\n0\r\n255\r\n244"
print("data", data)
numbers = [int(line) for line in data.splitlines()]
print("numbers", numbers)
Output:
data b'85\r\n0\r\n255\r\n244' numbers [85, 0, 255, 244]
If you have control over what is producing the numbers, you should send them as binary data (ints) instead of a string. Then you wouldn't have to do any data conversion at all.
Reply
#9
yes, i write that code on a rapberry pico with also python.
i'am using same USB cable, for programming the pico and sending de data to the PC (pull the USB out and put it back in to change direction)

I would indeed prefer to send the bytes correctly and not have to convert the received data on the receiving side....simpler is better Wink
the sending code from the raspberry pico:
import machine
from machine import I2C, Pin
import time

button=Pin(22, Pin.IN, Pin.PULL_DOWN)

i = 0
y = 1

r7 = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 255, 255, 255, 255, 255, 255, 255, 255, 255, 31, 31, 31, 31,
    31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31,
    31, 31, 31, 31, 31, 31, 31, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31,
    31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 255, 255, 255, 255, 255, 255, 255, 255, 255, 0, 0, 0, 0, 0,
    0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
    0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
    0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

while True:
    
#    print(len(r7)) # shows how many variables are in the array
    
    if button()==0 and y == 1:
        
        y = 0
    
        data = (85)
        print(data)
        
        while i < 230: # 229 + a 0 makes 230
            
            data = r7[i]    
            print(data)
            i += 1
            time.sleep_ms (1)
    
            if i == 230:	# 229 passed the while argument then i += 1 makes 230
                data = (170)
                print(data)
        i = 0
        y = 1
I'm not French, but from the netherlands
Reply
#10
This is bad code:
i = 0
y = 1
while True:
    data = (85)
    print(data)
    if button()==0 and y == 1:
        y = 0
        while i < 230: # 229 + a 0 makes 230
            data = r7[i]    
            print(data)
            i += 1
            time.sleep_ms (1)
            
            if i == 230:    # 229 passed the while argument then i += 1 makes 230
                data = (170)
                print(data)
        i = 0
        y = 1
Is y supposed to be a button debouncer? It doesn't work. You should wait for the button to be released before checking again if it is pressed.

A bigger problem is "while i < 230:". Looping a fixed number of times forces r7 to always be the same size. If you ever change the length of r7 you need to remember that the count must be updated in the while. You also have to count, which people are notoriously bad at. Let python count how many numbers are in r7 and loop through the numbers. Like this:
while True:
    if button() == 0:
        print(85)
        for number in r7:
            print(number)
        print(170)
        while button() == 0:  # wait for button release
            pass
Why do you sleep between printing the numbers in r7? There is no reason for this. The PC program is not going to read any bytes until 1000 bytes have been written.

You do not set a timeout, so serial_connection.read(1000) waits until it has read 1000 bytes. Your pico program only sends (prints) around 820 bytes (calculated using r7 as example data). The PC program must wait for multiple sends before it runs. The length of the printed data will vary depending on the values in r7, so using a fixed length read is also a problem. You print at least 3 bytes for each number (1 digit and \r\n) and at most 5 bytes (3 digits and \r\n), and you print 232 numbers (230 in r7 plus 85 at the start and 170 at the end). This means the pico may print anywhere between 699 and 1159 bytes. The pico needs to tell the PC program when to stop reading bytes.

You could use a special marker to indicate the end of the transmission. This lets you use Serial.read_until(marker) to read until it sees the marker. Pick a marker that will not otherwise appear in the data you are printing (like "|"). I would also stop printing the numbers one at a time and instead print everything using one print statement, making a string that contains the prefix (85), the suffix(170), all the numbers (r7) and the end of message marker ("|"). Something like this:
from machine import Pin
 
button = Pin(22, Pin.IN, Pin.PULL_DOWN)
end_of_message = "|"
r7 = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 255, 255, 255, 255, 255, 255, 255, 255, 255, 31, 31, 31, 31,
    31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31,
    31, 31, 31, 31, 31, 31, 31, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31,
    31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 255, 255, 255, 255, 255, 255, 255, 255, 255, 0, 0, 0, 0, 0,
    0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
    0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
    0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
 
while True:
    if button() == 0:
        msg = ",".join(map(str, [85] + r7 + [170]))
        print(msg, end=end_of_message)
        while button() == 0:  # wait for button to be released.
            pass
On the PC side the program would look like this:
import serial


end_of_message = b"|"
serial_connection = serial.Serial("com9", 115200)
while True:
    data = serial_connection.read_until(end_of_message)  # Read until we get end of message marker
    print(data)
    numbers = [int(x) for x in data.split(b",")]  # convert bytes to array of int.
    print(numbers)
I don't know if the serial_connection.read() will return the end of message marker. If so, it must be removed before converting the bytes to an array of int. The default marker for Serial.read_until() is \n. As long as we don't include any newlines in the message, we can use that as the end of message marker and use rstrip() to remove it from the data.

pico program
from machine import Pin
 
button = Pin(22, Pin.IN, Pin.PULL_DOWN)
r7 = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 255, 255, 255, 255, 255, 255, 255, 255, 255, 31, 31, 31, 31,
    31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31,
    31, 31, 31, 31, 31, 31, 31, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31,
    31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 255, 255, 255, 255, 255, 255, 255, 255, 255, 0, 0, 0, 0, 0,
    0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
    0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
    0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
 
while True:
    if button() == 0:
        msg = ",".join(map(str, [85] + r7 + [170]))
        print(msg)
        while button() == 0:  # wait for button to be released.
            pass
Program on PC
import serial


serial_connection = serial.Serial("com9", 115200)
while True:
    data = serial_connection.read_until()  # Read until we get end of message marker, \n
    print(data)
    numbers = [int(x) for x in data.rstrip().split(b",")]  # convert bytes to array of int.
    print(numbers)
I'm unclear about how to write the data to a file, so I left it off. But it would probably be one of these.
    # do you really want a binary file?  This writes the numbers as binary data.
    with open(destination_file, "ab") as file:
        destination(file.write(numbers))

    # or write the csv string since it is a .txt file?
    with open(destination_file, "a") as file:
        destination(file.write(data.decode("utf8")))
Instead of leaving the file open, open the file, append data, close file. If you want the file to be empty at the start of the program. This writes the values to a CSV format file.
import serial

destination_file = "/Users/Beheerder/Dropbox/documenten arjo/proj raspberry pico/pico sending data to PC/store.txt"
with open(destination_file, "w") as file:  # Create empty file or empty existing file.
    pass

serial_connection = serial.Serial("com9", 115200)
while True:
    data = serial_connection.read_until()  # Read until we get end of message marker, \n
    with open(destination_file, "a") as file:
        file.write(data.decode("utf8"))
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  pyserial/serial "has no attribute 'Serial' " gowb0w 11 22,830 Sep-27-2024, 12:18 PM
Last Post: NigelHalse
  Problem Converting Tradingview Indicator to Python code kralxs 1 2,453 Apr-27-2024, 06:10 PM
Last Post: kralxs
  bytearray object - why converting to ascii? Chepilo 2 4,892 Nov-21-2022, 07:25 PM
Last Post: Chepilo
  Converting SQL Code To Python Code Query eddywinch82 13 39,819 Feb-15-2020, 06:42 PM
Last Post: buran
  Reading mixed ASCII/binary serial input N8UR 1 4,582 Jun-11-2019, 02:49 PM
Last Post: N8UR
  Converting R code to python mcva 2 13,833 Mar-09-2019, 04:01 PM
Last Post: mcva
  Converting handwriting into code that can be recreated kill3rcat456 2 3,347 Mar-04-2018, 02:27 AM
Last Post: kill3rcat456
  Resources for converting Python 2.x code to 3.x Luke_Drillbrain 4 6,191 May-03-2017, 06:34 PM
Last Post: snippsat
  Python code with serial port and global undefined marciokoko 13 17,237 Jan-17-2017, 06:14 PM
Last Post: micseydel
  Need help converting some code from python 2.7 to 3.5 Blue Dog 3 5,839 Oct-18-2016, 12:30 AM
Last Post: sparkz_alot

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020