Python Forum
Parse Binary Data File and convert Epoch Time
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Parse Binary Data File and convert Epoch Time
#1
Hello All!

I am trying to extract and convert data from a binary data file. Each data entry follows this order:
  • sample time in seconds since unix epoch, 4 bytes
  • 2 byte integer
  • 2 byte integer
  • 2 byte integer
  • 2 byte integer

All values are integers

Below is the code I'm using based off a different binary data extraction project. It exports the correct number of data entries but the values do not make sense. I know I'm not parsing the binary file correctly but I;m not sure how to do the date conversion. Thanks!

import math, struct, time

field_names = ('time1', 'vol_conc', 'part_size', \
          'opt_trans', 'laser_ref')

def decode(data, offset=0):
    '''Decipher a lisst datagram from binary'''

    # Offset now tells it how far to start
    with open('li0002a', 'rb') as data:
        values = struct.unpack('i4h', data.read(12))

    # Create a dictionary for all the values
    lisst_values = dict(zip (field_names, values))

    return lisst_values

def lisst_print(lisst_values):
    'Print out all the values of a lisst dictionary'
    print 'results:'
    for key in lisst_values:
        print '    ', key, lisst_values[key]

datagram_size = 12 # 2*6 bytes per datagram

def num_datagrams(data):
    #How many packets are in data'

    # Make sure we have an even number of datagrams
    #assert (len(data) % datagram_size == 0)

    return len(data) / datagram_size

def get_offset(datagram_number):
    'Calculate the starting offset of a datagram'
    return datagram_number * datagram_size

def main():
    lisst_file = open('li0002a')
    lisst_data = lisst_file.read()

    print 'Number of datagrams:', num_datagrams(lisst_data)

    print 'Datagram Number, Time, Volume Concentration, Particle Size,\
 Optical Transmission, Laser Reference '

    for datagram_index in range( num_datagrams(lisst_data) ):
        offset = get_offset(datagram_index)
        datagram = decode(lisst_data,offset)

        print datagram_index, datagram['time1'], datagram['vol_conc'], datagram['part_size'], \
             datagram['opt_trans'], datagram['laser_ref']

if __name__=='__main__':
    main()
Reply
#2
pure binary as in a block of memory is arranged in machine addressing order, big or little endian, so your 4 byte value.
so you may have to do some transformations, Here's a good doc for that: https://wiki.python.org/moin/BitManipulation
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  How to convert binary data into text? ZYSIA 3 267 Jul-16-2021, 04:18 PM
Last Post: deanhystad
  Check last time file was accessed Pavel_47 4 454 Jun-01-2021, 05:47 PM
Last Post: Yoriz
Smile Set 'Time' format cell when writing data to excel and not 'custom' limors 3 747 Mar-29-2021, 09:36 PM
Last Post: Larz60+
  find the header location in a .bin file without reading the whole file at a time SANJIB 0 459 Mar-05-2021, 04:08 PM
Last Post: SANJIB
  Yahoo_fin, Pandas: how to convert data table structure in csv file detlefschmitt 14 1,808 Feb-15-2021, 12:58 PM
Last Post: detlefschmitt
  Read/Write binary file deanhystad 3 715 Feb-01-2021, 10:29 AM
Last Post: Larz60+
  Naming the file as time and date. BettyTurnips 3 583 Jan-15-2021, 07:52 AM
Last Post: BettyTurnips
  Stumped by my own code (ratio & epoch-time calculation). MvGulik 2 548 Dec-30-2020, 12:04 AM
Last Post: MvGulik
  [split] How to convert the CSV text file into a txt file Pinto94 5 671 Dec-23-2020, 08:04 AM
Last Post: ndc85430
  xml file creation from an XML file template and data from an excel file naji_python 1 507 Dec-21-2020, 03:24 PM
Last Post: Gribouillis

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020