Dec-05-2018, 12:51 AM
First of all, I am writing as I am thinking, so take this as something to think about rather than verbatim.
One thing that I notice is your baud rate is 9600.
If the data is being sent at 100 Hz, then the baud rate is fast enough to capture everything.
You need to read up on several things.
I think that you need to have the read routine running all the time in a thread, filling a FIFO (First in, First Out) buffer continuously as it reads.
Then you need another thread that will take data from the FIFO in chunks and transfer to either a process that will analyze it in whatever fashion you need, or to write the data to files of a certain size (starting a new one whenever sizes bytes are read. The data appears to have a well structured format, and I'm guessing each item is terminated with a newline. If this is the case, files should contain only whole records.
Now you can label the files with timestamps so that you know what order to process them in.
Finally, you can have multiple threads running that process the data. This will allow you to stay ahead of the flow.
Something to think about.
One thing that I notice is your baud rate is 9600.
If the data is being sent at 100 Hz, then the baud rate is fast enough to capture everything.
You need to read up on several things.
I think that you need to have the read routine running all the time in a thread, filling a FIFO (First in, First Out) buffer continuously as it reads.
Then you need another thread that will take data from the FIFO in chunks and transfer to either a process that will analyze it in whatever fashion you need, or to write the data to files of a certain size (starting a new one whenever sizes bytes are read. The data appears to have a well structured format, and I'm guessing each item is terminated with a newline. If this is the case, files should contain only whole records.
Now you can label the files with timestamps so that you know what order to process them in.
Finally, you can have multiple threads running that process the data. This will allow you to stay ahead of the flow.
Something to think about.