Custom file class

[deanhystad]

I don't know if this is the best solution, but it does appear to work fairly well for the header. This is the base class for binary files with headers.

"""Binfile.  A binary file with a header."""

def valid_key(key, valid_keys=None):
    """Validate key against a list of valid keys.  Return True if key in list"""
    if valid_keys is None:
        return True
    for k in valid_keys:
        if key == k:
            return True
    return False

class BinFile():
    """Class to read and write binary file format"""
    def __init__(self, filename=None, mode=None):
        self.filename = filename
        self.mode = mode
        self.file = None
        self.header = {}

    def __enter__(self):
        """Context manager enter.  Do open(filename, mode)"""
        self.open(self.filename, self.mode)
        return self

    def __exit__(self, *args):
        """Context manager exit.  Closes file if open"""
        self.close()

    def open(self, filename, mode):
        """Open file for reading ("r") or writing ("w")"""
        if mode in ('r', 'rb'):
            self.file = open(filename, 'rb')
            self.read_header()
        elif mode in ('w', 'wb'):
            self.file = open(filename, 'wb')
            self.write_header()  # Can call write_header again if changes are requried
        else:
            self.file = None
            raise ValueError(f'Invalid mode {mode} for open')

    def close(self):
        """Close file"""
        self.file.close()

    def read_header(self):
        """Read header from a file.  Looks for first "key" that contains a
        non-ascii character.  Override in subclass
        """
        self.file.seek(0)
        self.header = {}
        while True:
            key, value = self.read_header_line()
            if key is None:
                self.file.seek(len(self.header)*128)
                break
            self.header[key] = value

    def write_header(self):
        """Write header to a file"""
        self.file.seek(0)
        for key, value in self.header.items():
            self.write_header_line(key, value)

    def read_header_line(self, valid_keys=None):
        """Read a line from the header.  Return key and value.
        Use optional valid_keys to validate header key.
        """
        try:
            line = self.file.read(128).decode('ascii')
        except UnicodeDecodeError:
            return None, None

        if len(line) < 128:
            return None, None

        key = line[:32].strip()
        value = line[32:].strip()
        if valid_keys is not None:
            if not valid_key(key, valid_keys):
                raise ValueError(f'Unexpected key {key}')
        return key, value

    def write_header_line(self, key, value=None):
        """Write a header line to a file"""
        if value is None:
            value = self.header[key]
        self.file.write(bytes(f'{key:\0<32}{value:\0<96}', 'utf-8'))

Now I am starting to look at the time history part of the file. Let's say I have 2 channels , call them x and y, each with 5 data points. The data in the file will look like this:

x0, y0, x1, y1, x2, y2, x3, y3, x4, y4

When I use the data I need it to be arranged by channel.

x0, x1, x2, x3, x4, y0, y1, y2, y3, y4

A real file will probably be 8 channels instead of 2, and 10,000 data points per channel instead of 5. There is also a hard time limit on reading from the file and rotating the data (tens of milliseconds), so I need this to be fast.

I am looking at numpy.rot90, are there other tools that do this kind of thing? Other choices are to call an external C function to do the rotation.

[nilamo]

I am looking at numpy.rot90, are there other tools that do this kind of thing? Other choices are to call an external C function to do the rotation.

If that works, that's probably the best way forward. Numpy is optimized to work with large datasets, so (hopefully) as long as it works, you'll likely get the best performance possible while using python.

Pandas can also do something similar: https://pandas.pydata.org/pandas-docs/st...spose.html

Since you have the data in the header, you can pre-allocate the memory with numpy, which should help it with being fast (no resizing or reallocating). Something like:

>>> import numpy as np
>>> channels = 2 # from header data
>>> samples_per_channel = 5 # again, from header data
>>> data = np.empty((channels, samples_per_channel), dtype=np.uint32)
>>> data
array([[1886216563, 1601398124, 1601332592, 1851877475,  543974766],
       [ 221585469,         10,          0,    7798904,          0]],
      dtype=uint32)