PC why serial code = converting to ascii ?

[trix]

hello,

I have written a piece of code in virtual studio code on the PC that stores the data received via the USB in an, I think an array, but I am not sure that it is an array.
when i take a look in the serial monitor from VS code The code also arrives as I sent it.
but when i look in the terminal, then the code appears to have changed to ascii.
Does anyone know how I can prevent this change to ascii?

The code (comments deleted):

import serial

k = 0

# Configure the serial connection
port = "com9" 
baudrate = 115200
serial_connection = serial.Serial(port, baudrate)

# Open a file on your computer to write the received data
destination_file = open("/Users/Beheerder/Dropbox/documenten arjo/proj raspberry pico/pico sending data to PC/store.txt", "wb")

while True:
    
    data = serial_connection.read(1000)

    print(data)
    destination_file.write(data)

    destination_file.flush() # the data is writing in to the destination_file after flush()

in serial monitor: (this is wat i need)
85
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
255
255
255

in terminal: (this is wat i not need)
b'85\r\n0\r\n0\r\n0\r\n0\r\n0\r\n0\r\n0\r\n0\r\n0\r\n0\r\n0\r\n0\r\n0\r\n0\r\n0\r\n0\r\n0\r\n0\r\n0\r\n0\r\n0\r\n0\r\n255\r\n255\r\n255\r

[deanhystad]

Please post text, not screenshots.

Serial.read returns bytes. bytes is an array-like thing in python for small integers (range 0 to 255). When printing bytes, python uses visible ascii characters when possible. When the "byte" doesn't map to an ascii character, it is printed as an escape sequence.

You are wrong thinking the problem is that data is converted to ascii when written to the terminal. The problem is the opposite. To hide the \r (carriage return) and \n (newline) you need to convert the bytes to a string. It appears that data is an encoded string. To convert it to a str, you need to decode it.

data = b"""This is a
string encoded
as bytes."""
print(data)
as_str = data.decode("utf8")
print(as_str)

Output:
b'This is a\nstring encoded\nas bytes.'
This is a     
string encoded
as bytes.

[bowlofred]

but when i look in the terminal, then the code appears to have changed to ascii.
Does anyone know how I can prevent this change to ascii?

If you print() a bytes or a bytearray object, then any elements that are in the ascii range are rendered in ascii. Note that this is just the way print() behaves. The integers are still in the object.

If you don't want to see the ascii, then you can interate over the bytes/bytearray object and get the integers and print them instead.

Example:

>>> numeric_sequence = (0, 255, 65, 70, 75)
>>> b = bytes(numeric_sequence)
>>> b
b'\x00\xffAFK'
>>> for i in b:
...   print(i)
...
0
255
65
70
75

[trix]

thanks for the answers. I'm going to try that

[trix]

That actually works, thanks for that.
but now I see that I have a new problem. when I print that, each individual number is printed one below the other.
8
5
0
0
0
0
2
5
5
2
5
5
instead of:
85
0
0
0
0
255
255
later on, i want to do some calculations with these numbers.
Tnx.

[deanhystad]

decode() is the correct solution for your issue. The decoded str will print the \r and \n as carriage return and linefeed instead of escape sequences.

[trix]

thanks I will take a look.

[deanhystad]

"85", "0" and "255" are not numbers, they are strings. You'll need to use int(string) to convert them to numbers.

data = b"85\r\n0\r\n255\r\n244"
print("data", data)
as_str = data.decode("utf8")
print("as_str", as_str)
lines = as_str.splitlines()
print("lines", lines)
numbers = list(map(int, lines))
print("numbers", numbers)

Output:
data b'85\r\n0\r\n255\r\n244'
as_str 85
0
255
244
lines ['85', '0', '255', '244']
numbers [85, 0, 255, 244]

If all you want are the integers you can skip a few steps because bytes also has a splitlines() method.

data = b"85\r\n0\r\n255\r\n244"
print("data", data)
numbers = [int(line) for line in data.splitlines()]
print("numbers", numbers)

Output:
data b'85\r\n0\r\n255\r\n244'
numbers [85, 0, 255, 244]

If you have control over what is producing the numbers, you should send them as binary data (ints) instead of a string. Then you wouldn't have to do any data conversion at all.

[trix]

yes, i write that code on a rapberry pico with also python.
i'am using same USB cable, for programming the pico and sending de data to the PC (pull the USB out and put it back in to change direction)

I would indeed prefer to send the bytes correctly and not have to convert the received data on the receiving side....simpler is better
the sending code from the raspberry pico:

import machine
from machine import I2C, Pin
import time

button=Pin(22, Pin.IN, Pin.PULL_DOWN)

i = 0
y = 1

r7 = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 255, 255, 255, 255, 255, 255, 255, 255, 255, 31, 31, 31, 31,
    31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31,
    31, 31, 31, 31, 31, 31, 31, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31,
    31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 255, 255, 255, 255, 255, 255, 255, 255, 255, 0, 0, 0, 0, 0,
    0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
    0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
    0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

while True:
    
#    print(len(r7)) # shows how many variables are in the array
    
    if button()==0 and y == 1:
        
        y = 0
    
        data = (85)
        print(data)
        
        while i < 230: # 229 + a 0 makes 230
            
            data = r7[i]    
            print(data)
            i += 1
            time.sleep_ms (1)
    
            if i == 230:	# 229 passed the while argument then i += 1 makes 230
                data = (170)
                print(data)
        i = 0
        y = 1

[deanhystad]

This is bad code:

i = 0
y = 1
while True:
    data = (85)
    print(data)
    if button()==0 and y == 1:
        y = 0
        while i < 230: # 229 + a 0 makes 230
            data = r7[i]    
            print(data)
            i += 1
            time.sleep_ms (1)
            
            if i == 230:    # 229 passed the while argument then i += 1 makes 230
                data = (170)
                print(data)
        i = 0
        y = 1

Is y supposed to be a button debouncer? It doesn't work. You should wait for the button to be released before checking again if it is pressed.

A bigger problem is "while i < 230:". Looping a fixed number of times forces r7 to always be the same size. If you ever change the length of r7 you need to remember that the count must be updated in the while. You also have to count, which people are notoriously bad at. Let python count how many numbers are in r7 and loop through the numbers. Like this:

while True:
    if button() == 0:
        print(85)
        for number in r7:
            print(number)
        print(170)
        while button() == 0:  # wait for button release
            pass

Why do you sleep between printing the numbers in r7? There is no reason for this. The PC program is not going to read any bytes until 1000 bytes have been written.

You do not set a timeout, so serial_connection.read(1000) waits until it has read 1000 bytes. Your pico program only sends (prints) around 820 bytes (calculated using r7 as example data). The PC program must wait for multiple sends before it runs. The length of the printed data will vary depending on the values in r7, so using a fixed length read is also a problem. You print at least 3 bytes for each number (1 digit and \r\n) and at most 5 bytes (3 digits and \r\n), and you print 232 numbers (230 in r7 plus 85 at the start and 170 at the end). This means the pico may print anywhere between 699 and 1159 bytes. The pico needs to tell the PC program when to stop reading bytes.

You could use a special marker to indicate the end of the transmission. This lets you use Serial.read_until(marker) to read until it sees the marker. Pick a marker that will not otherwise appear in the data you are printing (like "|"). I would also stop printing the numbers one at a time and instead print everything using one print statement, making a string that contains the prefix (85), the suffix(170), all the numbers (r7) and the end of message marker ("|"). Something like this:

from machine import Pin
 
button = Pin(22, Pin.IN, Pin.PULL_DOWN)
end_of_message = "|"
r7 = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 255, 255, 255, 255, 255, 255, 255, 255, 255, 31, 31, 31, 31,
    31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31,
    31, 31, 31, 31, 31, 31, 31, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31,
    31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 255, 255, 255, 255, 255, 255, 255, 255, 255, 0, 0, 0, 0, 0,
    0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
    0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
    0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
 
while True:
    if button() == 0:
        msg = ",".join(map(str, [85] + r7 + [170]))
        print(msg, end=end_of_message)
        while button() == 0:  # wait for button to be released.
            pass

On the PC side the program would look like this:

import serial


end_of_message = b"|"
serial_connection = serial.Serial("com9", 115200)
while True:
    data = serial_connection.read_until(end_of_message)  # Read until we get end of message marker
    print(data)
    numbers = [int(x) for x in data.split(b",")]  # convert bytes to array of int.
    print(numbers)

I don't know if the serial_connection.read() will return the end of message marker. If so, it must be removed before converting the bytes to an array of int. The default marker for Serial.read_until() is \n. As long as we don't include any newlines in the message, we can use that as the end of message marker and use rstrip() to remove it from the data.

pico program

from machine import Pin
 
button = Pin(22, Pin.IN, Pin.PULL_DOWN)
r7 = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 255, 255, 255, 255, 255, 255, 255, 255, 255, 31, 31, 31, 31,
    31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31,
    31, 31, 31, 31, 31, 31, 31, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31,
    31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 255, 255, 255, 255, 255, 255, 255, 255, 255, 0, 0, 0, 0, 0,
    0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
    0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
    0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
 
while True:
    if button() == 0:
        msg = ",".join(map(str, [85] + r7 + [170]))
        print(msg)
        while button() == 0:  # wait for button to be released.
            pass

Program on PC

import serial


serial_connection = serial.Serial("com9", 115200)
while True:
    data = serial_connection.read_until()  # Read until we get end of message marker, \n
    print(data)
    numbers = [int(x) for x in data.rstrip().split(b",")]  # convert bytes to array of int.
    print(numbers)

I'm unclear about how to write the data to a file, so I left it off. But it would probably be one of these.

    # do you really want a binary file?  This writes the numbers as binary data.
    with open(destination_file, "ab") as file:
        destination(file.write(numbers))

    # or write the csv string since it is a .txt file?
    with open(destination_file, "a") as file:
        destination(file.write(data.decode("utf8")))

Instead of leaving the file open, open the file, append data, close file. If you want the file to be empty at the start of the program. This writes the values to a CSV format file.

import serial

destination_file = "/Users/Beheerder/Dropbox/documenten arjo/proj raspberry pico/pico sending data to PC/store.txt"
with open(destination_file, "w") as file:  # Create empty file or empty existing file.
    pass

serial_connection = serial.Serial("com9", 115200)
while True:
    data = serial_connection.read_until()  # Read until we get end of message marker, \n
    with open(destination_file, "a") as file:
        file.write(data.decode("utf8"))