Python Forum
Parse Binary Data File and convert Epoch Time
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Parse Binary Data File and convert Epoch Time
#1
Hello All!

I am trying to extract and convert data from a binary data file. Each data entry follows this order:
  • sample time in seconds since unix epoch, 4 bytes
  • 2 byte integer
  • 2 byte integer
  • 2 byte integer
  • 2 byte integer

All values are integers

Below is the code I'm using based off a different binary data extraction project. It exports the correct number of data entries but the values do not make sense. I know I'm not parsing the binary file correctly but I;m not sure how to do the date conversion. Thanks!

import math, struct, time

field_names = ('time1', 'vol_conc', 'part_size', \
          'opt_trans', 'laser_ref')

def decode(data, offset=0):
    '''Decipher a lisst datagram from binary'''

    # Offset now tells it how far to start
    with open('li0002a', 'rb') as data:
        values = struct.unpack('i4h', data.read(12))

    # Create a dictionary for all the values
    lisst_values = dict(zip (field_names, values))

    return lisst_values

def lisst_print(lisst_values):
    'Print out all the values of a lisst dictionary'
    print 'results:'
    for key in lisst_values:
        print '    ', key, lisst_values[key]

datagram_size = 12 # 2*6 bytes per datagram

def num_datagrams(data):
    #How many packets are in data'

    # Make sure we have an even number of datagrams
    #assert (len(data) % datagram_size == 0)

    return len(data) / datagram_size

def get_offset(datagram_number):
    'Calculate the starting offset of a datagram'
    return datagram_number * datagram_size

def main():
    lisst_file = open('li0002a')
    lisst_data = lisst_file.read()

    print 'Number of datagrams:', num_datagrams(lisst_data)

    print 'Datagram Number, Time, Volume Concentration, Particle Size,\
 Optical Transmission, Laser Reference '

    for datagram_index in range( num_datagrams(lisst_data) ):
        offset = get_offset(datagram_index)
        datagram = decode(lisst_data,offset)

        print datagram_index, datagram['time1'], datagram['vol_conc'], datagram['part_size'], \
             datagram['opt_trans'], datagram['laser_ref']

if __name__=='__main__':
    main()
Reply
#2
pure binary as in a block of memory is arranged in machine addressing order, big or little endian, so your 4 byte value.
so you may have to do some transformations, Here's a good doc for that: https://wiki.python.org/moin/BitManipulation
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Convert legacy print file to XLSX file davidm 0 96 Oct-10-2021, 01:08 PM
Last Post: davidm
  how to parse data fakka 2 197 Sep-22-2021, 10:50 PM
Last Post: bowlofred
  How to convert binary data into text? ZYSIA 3 659 Jul-16-2021, 04:18 PM
Last Post: deanhystad
  Check last time file was accessed Pavel_47 4 729 Jun-01-2021, 05:47 PM
Last Post: Yoriz
Smile Set 'Time' format cell when writing data to excel and not 'custom' limors 3 1,350 Mar-29-2021, 09:36 PM
Last Post: Larz60+
  find the header location in a .bin file without reading the whole file at a time SANJIB 0 637 Mar-05-2021, 04:08 PM
Last Post: SANJIB
  Yahoo_fin, Pandas: how to convert data table structure in csv file detlefschmitt 14 2,605 Feb-15-2021, 12:58 PM
Last Post: detlefschmitt
  Read/Write binary file deanhystad 3 1,006 Feb-01-2021, 10:29 AM
Last Post: Larz60+
  Naming the file as time and date. BettyTurnips 3 819 Jan-15-2021, 07:52 AM
Last Post: BettyTurnips
  Stumped by my own code (ratio & epoch-time calculation). MvGulik 2 726 Dec-30-2020, 12:04 AM
Last Post: MvGulik

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020