How to sort a HDF5 file - Printable Version +- Python Forum (https://python-forum.io) +-- Forum: Python Coding (https://python-forum.io/forum-7.html) +--- Forum: Data Science (https://python-forum.io/forum-44.html) +--- Thread: How to sort a HDF5 file (/thread-28536.html) |
How to sort a HDF5 file - Robotguy - Jul-23-2020 Hey everyone, I am storing a large data file (10 GBs, N rows and 4 columns) in an HDF5 file using h5py package. Primarily because I do not want to use my RAM. I would like to sort the items in the file based on second column. Any suggestions on how to do that? Thanks! RE: How to sort a HDF5 file - DeaD_EyE - Jul-23-2020 Chunks | Sort | Merge Split the data of the hdf5 file into chunks. Then sort this chunks and write the sorted chunks to disk. Then open all chunk files and merge them. Write the output into a different file. You need heapq.merge which return a generator.Here an example how it could work. import os import heapq import random from contextlib import ExitStack from pathlib import Path def producer(size): """ Return some random integers between 0 and < 1024 """ return [random.randint(0, 1024) for _ in range(size)] # you could use chunked from more_itertools # https://more-itertools.readthedocs.io/en/stable/api.html#more_itertools.chunked def chunker(iterable, chunksize): """ Split a iterable into smaller chunks """ return zip(*[iter(iterable)] * chunksize) def sorter(iterable, filename): """ Sort the chunks and save them into a files The files are defined by filename """ for n, chunk in enumerate(iterable): chunk = "\n".join(map(str, sorted(chunk))) with open(f"{filename}_{n}", "w") as fd: fd.write(chunk) def merger(filename, output): """ Find all files related to filename_* Sort the files by last number Then open the files Merge the chunks and write it to output """ key = lambda x: int(x.name.replace(f"{filename}_", "")) files = sorted(Path(filename).parent.glob(f"{filename}_*"), key=key) with ExitStack() as stack, open(output, "w") as fd_out: files = [stack.enter_context(file.open()) for file in files] map_to_int = [map(int, fd) for fd in files] for number in map(str, heapq.merge(*map_to_int)): fd_out.write(number) fd_out.write("\n") sorter(chunker(producer(100), 10), "sorting") merger("sorting", "result.txt")By the way, do not try to save the output data into the source hdf5 file. Make a new hdf5 file. |