Python Forum
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Problem with delimiters
#1
Hello everyone
I am quite stuck. i am trying to get a message from a TCP communication protocol, this message is converted in a list of strings and then into a list of float, The thing is that im getting an error because of the delimiters.
<the message is dynamic and it has this form:
message = 0.2081006,0.0000000,0.0000000,0.2081006,0.0000000,0.0000000,0.2081006,0.0000000,0.0000000*
i want to get everything behind the '*'
here is my code
while True:

    message = conn.recv(BUFFER_SIZE)
    if not message:
        break
                
    t1 = time.time()
    message = message.decode("utf-8")
    print("Raw message: {}".format(message))


    data = list(message.split(","))
    print("Data Size:{} ".format(len(data)))
    

    
    MyNewData = []
    for item in data:
        MyNewData.append(float(item))
    print("The new list: {}".format(MyNewData))

        
    print("Size:{} ".format(len(data)))
    if len(MyNewData) < 9:
        continue


    DataX = MyNewData[0:3]
    DataY = MyNewData[3:6]
    DataZ = MyNewData[6:9]
This the ouput and error
Output:
Raw message: 0.2081006,0.0000000,0.0000000,0.2081006,0.0000000,0.0000000,0.2081006,0.0000000,0.0000000*
expected output should be like this
message: 0.2081006,0.0000000,0.0000000,0.2081006,0.0000000,0.0000000,0.2081006,0.0000000,0.0000000
Error:
File "lstm_server_3.py", line 158, in <module> MyNewData.append(float(item)) ValueError: could not convert string to float: '0.0000000*'
The i changed the code into this
while True:

    message = conn.recv(BUFFER_SIZE)
    if not message:
        break
                
    t1 = time.time()
    message = message.decode("utf-8")
    print("Raw message: {}".format(message))


    data = list(message.split(","))
    print("Data Size:{} ".format(len(data)))
    

        


  
    MyLastString = data[-1]

    
    result2 = MyLastString.find('*')
    if result2 > 0:
        MyLastString = MyLastString[0:result2]

    data[len(data)-1] = MyLastString
    
    MyNewData = []
    for item in data:
        MyNewData.append(float(item))
    print("The new list: {}".format(MyNewData))

        
    print("Size:{} ".format(len(data)))
    if len(MyNewData) < 9:
        continue


    DataX = MyNewData[0:3]
    DataY = MyNewData[3:6]
    DataZ = MyNewData[6:9]
Now the error is in the first position of the list. Also it seems that some part of the last element is in the first element
Output:
message: 35*0.2593872,0.0000000,-5.0712390,0.2593872,0.0000000,-0.4532299,0.2593872,0.0000000,-0.10
so my expected output should be like this
message: 0.2593872,0.0000000,-5.0712390,0.2593872,0.0000000,-0.4532299,0.2593872,0.0000000,-0.1035

Error:
File "lstm_server_3.py", line 158, in <module> MyNewData.append(float(item)) ValueError: could not convert string to float: '35*0.2593872'
Reply
#2
I think you need to know what the asterisk within the message means and also make sure you receive the whole message
I think at the moment you try to parse just a part of the message (i.e. just buffer_suze chunk)
message = ''
try:
    while True:
        data = conn.recv(BUFFER_SIZE)
        if data:
            message += data.decode('utf-8')
        else:
            break
    print("Raw message: {}".format(message))
finally:
    conn.close()
                 
If you can't explain it to a six year old, you don't understand it yourself, Albert Einstein
How to Ask Questions The Smart Way: link and another link
Create MCV example
Debug small programs

Reply
#3
the asterisk means 'end of the message'
i don't get any the output with the code that @buran suggested me
Think
Reply
#4
If what you stated:
(Jan-28-2020, 04:39 PM)johnprada Wrote: Also it seems that some part of the last element is in the first element
it's not likely that asterisk means end of message. Maybe some kind of separator... And what you think it to be part of the last element is actually part of the previous chunk
If you can't explain it to a six year old, you don't understand it yourself, Albert Einstein
How to Ask Questions The Smart Way: link and another link
Create MCV example
Debug small programs

Reply
#5
(Jan-29-2020, 04:15 AM)buran Wrote: If what you stated:
(Jan-28-2020, 04:39 PM)johnprada Wrote: Also it seems that some part of the last element is in the first element
it's not likely that asterisk means end of message. Maybe some kind of separator... And what you think it to be part of the last element is actually part of the previous chunk
Obviously my code is untested, so it's possible that there is problem
If you can't explain it to a six year old, you don't understand it yourself, Albert Einstein
How to Ask Questions The Smart Way: link and another link
Create MCV example
Debug small programs

Reply
#6
asyncio.StreamReader has some convenience functions like readuntil, readline, readexcatly

This code could handle many simultaneous connections.
I use type hints in the first function, to show
what is the expected data and what is returned by this function.

The function handle_connection has also type hints.
I used this inside an IDE (PyCharm), which gives you auto complete for this type.
I hope this is not too much. You don't have to understand everything.
asyncio is a little bit complex and has some quirks.


import asyncio
from typing import List


def convert_values(data: bytes) -> List[float]:
    """
    Function to convert floats delimited by a comma
    into a list with floats.

    The eof marker * at the end will be removed.
    """
    try:
        # 0. The data type is bytes
        # 1. rstrip the *
        # 2. split by , -> list with bytes
        # 3. MAP: call float for each element in the list
        # 4. consume the map by a list, map is lazy evaluated
        values = list(map(float, data.rstrip(b'*').split(b',')))
        # or written in more lines and using a list comprehension
        # data = data.rstrip(b'*').split(b',')
        # data is now a list with bytes
        # the function float can handle both: bytes and str
        # whitespaces are removed automatically by this function
        # values = [float(val) for val in data]
    except ValueError:
        # In case of an error, return a empty list
        return []
    # no error -> return values
    return values


async def handle_connection(reader: asyncio.StreamReader, writer: asyncio.StreamWriter):
    addr, port = writer.get_extra_info('peername')
    print(f'Client {addr} connected.')
    while True:
        try:
            # read until *
            # a timeout should be used
            future = reader.readuntil(b'*')  # bytes, not str
            data = await asyncio.wait_for(future, timeout=10)
        except asyncio.TimeoutError:
            print(f'Got timeout from {addr}')
            writer.close()
            await writer.wait_closed()
            # use break or return to stop the
            # handle_connection
            # forgetting to leave this loop, will raise the
            # exception: asyncio.IncompleteReadError
            break
        else:
            # no error, print the values
            values = convert_values(data)
            print(values)
            # or do something else with it.


async def main():
    """
    Copy and Paste from:
    https://docs.python.org/3/library/asyncio-stream.html#tcp-echo-server-using-streams
    """
    server = await asyncio.start_server(handle_connection, 'localhost', 1234)
    async with server:
        await server.serve_forever()

# convenience function
# https://docs.python.org/3/library/asyncio-task.html#asyncio.run
asyncio.run(main())
You don't have to use asyncio, but the function convert_values explains how your original problem could be solved. The length of example data seems to be variable. If the data is sent over and over again and * is the start/end marker, then you need to change the timeout. Maybe it's too short or too long. Depends how fast the data is sent over the wire.
My code examples are always for Python >=3.6.0
Almost dead, but too lazy to die: https://sourceserver.info
All humans together. We don't need politicians!
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Delimiters - How to skip some html tags from being translate Melcu54 0 294 May-26-2021, 06:21 AM
Last Post: Melcu54
  Parse String between 2 Delimiters and add as single list items lastyle 5 610 Apr-11-2021, 11:03 PM
Last Post: lastyle
  Split string between two different delimiters, with exceptions DreamingInsanity 2 656 Aug-24-2020, 08:23 AM
Last Post: DreamingInsanity
  Problem with delimiters johnprada 1 633 Jan-28-2020, 04:27 PM
Last Post: buran
  splitting a string with 2 different delimiters Skaperen 4 1,003 Dec-30-2019, 04:49 AM
Last Post: BamBi25
  Split a long string into other strings with no delimiters/characters krewlaz 4 975 Nov-15-2019, 02:48 PM
Last Post: ichabod801
  re.split multiple delimiters problem gw1500se 2 1,906 Jun-24-2019, 02:43 PM
Last Post: gw1500se
  Finding nested delimiters wfsteadman 4 1,729 Jan-22-2018, 07:23 AM
Last Post: DeaD_EyE

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020