Problem with delimiters - Printable Version

Problem with delimiters - Printable Version

+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: General Coding Help (https://python-forum.io/forum-8.html)
+--- Thread: Problem with delimiters (/thread-24044.html)

Problem with delimiters - johnprada - Jan-28-2020

Hello everyone
I am quite stuck. i am trying to get a message from a TCP communication protocol, this message is converted in a list of strings and then into a list of float, The thing is that im getting an error because of the delimiters.
<the message is dynamic and it has this form:
message = 0.2081006,0.0000000,0.0000000,0.2081006,0.0000000,0.0000000,0.2081006,0.0000000,0.0000000*
i want to get everything behind the '*'
here is my code

while True:

    message = conn.recv(BUFFER_SIZE)
    if not message:
        break
                
    t1 = time.time()
    message = message.decode("utf-8")
    print("Raw message: {}".format(message))


    data = list(message.split(","))
    print("Data Size:{} ".format(len(data)))
    

    
    MyNewData = []
    for item in data:
        MyNewData.append(float(item))
    print("The new list: {}".format(MyNewData))

        
    print("Size:{} ".format(len(data)))
    if len(MyNewData) < 9:
        continue


    DataX = MyNewData[0:3]
    DataY = MyNewData[3:6]
    DataZ = MyNewData[6:9]

This the ouput and error

Output:
Raw message: 0.2081006,0.0000000,0.0000000,0.2081006,0.0000000,0.0000000,0.2081006,0.0000000,0.0000000*

expected output should be like this
message: 0.2081006,0.0000000,0.0000000,0.2081006,0.0000000,0.0000000,0.2081006,0.0000000,0.0000000

Error:  File "lstm_server_3.py", line 158, in <module>
    MyNewData.append(float(item))
ValueError: could not convert string to float: '0.0000000*'

The i changed the code into this

while True:

    message = conn.recv(BUFFER_SIZE)
    if not message:
        break
                
    t1 = time.time()
    message = message.decode("utf-8")
    print("Raw message: {}".format(message))


    data = list(message.split(","))
    print("Data Size:{} ".format(len(data)))
    

        


  
    MyLastString = data[-1]

    
    result2 = MyLastString.find('*')
    if result2 > 0:
        MyLastString = MyLastString[0:result2]

    data[len(data)-1] = MyLastString
    
    MyNewData = []
    for item in data:
        MyNewData.append(float(item))
    print("The new list: {}".format(MyNewData))

        
    print("Size:{} ".format(len(data)))
    if len(MyNewData) < 9:
        continue


    DataX = MyNewData[0:3]
    DataY = MyNewData[3:6]
    DataZ = MyNewData[6:9]

Now the error is in the first position of the list. Also it seems that some part of the last element is in the first element

Output:
message: 35*0.2593872,0.0000000,-5.0712390,0.2593872,0.0000000,-0.4532299,0.2593872,0.0000000,-0.10

so my expected output should be like this
message: 0.2593872,0.0000000,-5.0712390,0.2593872,0.0000000,-0.4532299,0.2593872,0.0000000,-0.1035

Error:  File "lstm_server_3.py", line 158, in <module>
    MyNewData.append(float(item))
ValueError: could not convert string to float: '35*0.2593872'

RE: Problem with delimiters - buran - Jan-28-2020

I think you need to know what the asterisk within the message means and also make sure you receive the whole message
I think at the moment you try to parse just a part of the message (i.e. just buffer_suze chunk)

message = ''
try:
    while True:
        data = conn.recv(BUFFER_SIZE)
        if data:
            message += data.decode('utf-8')
        else:
            break
    print("Raw message: {}".format(message))
finally:
    conn.close()

RE: Problem with delimiters - johnprada - Jan-29-2020

the asterisk means 'end of the message'
i don't get any the output with the code that @buran suggested me
Think

RE: Problem with delimiters - buran - Jan-29-2020

If what you stated:

(Jan-28-2020, 04:39 PM)johnprada Wrote: Also it seems that some part of the last element is in the first element

it's not likely that asterisk means end of message. Maybe some kind of separator... And what you think it to be part of the last element is actually part of the previous chunk

RE: Problem with delimiters - buran - Jan-29-2020

(Jan-29-2020, 04:15 AM)buran Wrote: If what you stated:
(Jan-28-2020, 04:39 PM)johnprada Wrote: Also it seems that some part of the last element is in the first element

it's not likely that asterisk means end of message. Maybe some kind of separator... And what you think it to be part of the last element is actually part of the previous chunk
Obviously my code is untested, so it's possible that there is problem

RE: Problem with delimiters - DeaD_EyE - Jan-29-2020

asyncio.StreamReader has some convenience functions like readuntil, readline, readexcatly

This code could handle many simultaneous connections.
I use type hints in the first function, to show
what is the expected data and what is returned by this function.

The function handle_connection has also type hints.
I used this inside an IDE (PyCharm), which gives you auto complete for this type.
I hope this is not too much. You don't have to understand everything.
asyncio is a little bit complex and has some quirks.

import asyncio
from typing import List


def convert_values(data: bytes) -> List[float]:
    """
    Function to convert floats delimited by a comma
    into a list with floats.

    The eof marker * at the end will be removed.
    """
    try:
        # 0. The data type is bytes
        # 1. rstrip the *
        # 2. split by , -> list with bytes
        # 3. MAP: call float for each element in the list
        # 4. consume the map by a list, map is lazy evaluated
        values = list(map(float, data.rstrip(b'*').split(b',')))
        # or written in more lines and using a list comprehension
        # data = data.rstrip(b'*').split(b',')
        # data is now a list with bytes
        # the function float can handle both: bytes and str
        # whitespaces are removed automatically by this function
        # values = [float(val) for val in data]
    except ValueError:
        # In case of an error, return a empty list
        return []
    # no error -> return values
    return values


async def handle_connection(reader: asyncio.StreamReader, writer: asyncio.StreamWriter):
    addr, port = writer.get_extra_info('peername')
    print(f'Client {addr} connected.')
    while True:
        try:
            # read until *
            # a timeout should be used
            future = reader.readuntil(b'*')  # bytes, not str
            data = await asyncio.wait_for(future, timeout=10)
        except asyncio.TimeoutError:
            print(f'Got timeout from {addr}')
            writer.close()
            await writer.wait_closed()
            # use break or return to stop the
            # handle_connection
            # forgetting to leave this loop, will raise the
            # exception: asyncio.IncompleteReadError
            break
        else:
            # no error, print the values
            values = convert_values(data)
            print(values)
            # or do something else with it.


async def main():
    """
    Copy and Paste from:
    https://docs.python.org/3/library/asyncio-stream.html#tcp-echo-server-using-streams
    """
    server = await asyncio.start_server(handle_connection, 'localhost', 1234)
    async with server:
        await server.serve_forever()

# convenience function
# https://docs.python.org/3/library/asyncio-task.html#asyncio.run
asyncio.run(main())

You don't have to use asyncio, but the function convert_values explains how your original problem could be solved. The length of example data seems to be variable. If the data is sent over and over again and * is the start/end marker, then you need to change the timeout. Maybe it's too short or too long. Depends how fast the data is sent over the wire.