Python Forum
Convert file of hex strings to binary file
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Convert file of hex strings to binary file
#1
Hello,
I have a file that has several hex string values, separated by the newline character. E.g. the file looks like:

dd5bda81
ae0ac495
b97a7664
...
I can easily parse this file and find e.g. the 10th string based on the line number.

However I want to convert the file to binary to save disk space.
I was thinking of using
binascii.unhexlify()
however I'm not sure what's the best way to handle their ordering, i.e. just concatenate the byte arrays? Note that the original file can be huge, maybe several gigabytes in size, and I'm not sure how efficient it would be to parse the billion-th value.
Reply
#2
If the source file has line endings like your example, you can process line by line.
Here as short example:

with open("source.hex") as fd_in, open("destination.bin", "wb") as fd_out:
    for line in fd_in:
        chunk = binascii.unhexlify(line.rstrip())
        fd_out.write(chunk)
  • Open source-file in read text mode, open output file in binary write mode. The example shows how to do it in one line.
  • iterate over lines. for line in fd_in
  • Strip from the right side whitespace: line.rstrip()
  • Convert hex-string into bytes (binary data) with binascii.unhexlify
  • Write the processed data to fd_out
My code examples are always for Python >=3.6.0
Almost dead, but too lazy to die: https://sourceserver.info
All humans together. We don't need politicians!
Reply
#3
Sometimes it makes more sense to write a C program. Am I going to get kicked off the forum now?
Reply
#4
Hm, why?

Try your luck.

If you have done it right, then create a Python Module in C.
My code examples are always for Python >=3.6.0
Almost dead, but too lazy to die: https://sourceserver.info
All humans together. We don't need politicians!
Reply
#5
My test:

[deadeye@nexus ~]$ dd if=/dev/urandom of=random.bin bs=1M count=64
64+0 Datens├Ątze ein
64+0 Datens├Ątze aus
67108864 Bytes (67 MB, 64 MiB) kopiert, 0,815292 s, 82,3 MB/s
[deadeye@nexus ~]$ python file2hex.py 
[deadeye@nexus ~]$ md5sum random.bin random2.bin 
929b3a89653f956721743a93955e2ec2  random.bin
929b3a89653f956721743a93955e2ec2  random2.bin
Code:
from binascii import hexlify, unhexlify


def file2hex(input_file, output_file):
    with open(input_file, "rb") as fd_in, open(output_file, "wb") as fd_out:
        while chunk := fd_in.read(20):
            fd_out.write(hexlify(chunk))
            fd_out.write(b"\n")


def hex2file(input_file, output_file):
    with open(input_file, "rb") as fd_in, open(output_file, "wb") as fd_out:
        for line in fd_in:
            fd_out.write(unhexlify(line.rstrip()))



file2hex("random.bin", "random.hex")
hex2file("random.hex", "random2.bin")
My code examples are always for Python >=3.6.0
Almost dead, but too lazy to die: https://sourceserver.info
All humans together. We don't need politicians!
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Is there anyway to make a .py file into a .exe file? Oshadha 2 134 Jan-13-2021, 04:58 AM
Last Post: Oshadha
  [split] How to convert the CSV text file into a txt file Pinto94 5 166 Dec-23-2020, 08:04 AM
Last Post: ndc85430
  xml file creation from an XML file template and data from an excel file naji_python 1 125 Dec-21-2020, 03:24 PM
Last Post: Gribouillis
  how to automate sending bulk emails from an excel file or a calc-file apollo 0 168 Dec-10-2020, 08:43 PM
Last Post: apollo
  how to change the range of read CSV file every time python file runs greenpine 6 239 Dec-08-2020, 10:11 PM
Last Post: greenpine
  Creating csv file from another file pisike 0 132 Nov-24-2020, 02:02 PM
Last Post: pisike
  Saving text file with a click: valueerror i/o operation on closed file vizier87 5 433 Nov-16-2020, 07:56 AM
Last Post: Gribouillis
  CPC File Format (Cartesian Perceptual Compression) - Can Python Convert / Handle Them PSKrieger 2 175 Nov-11-2020, 02:57 PM
Last Post: PSKrieger
  Binary File Error SalsaBeanDip 1 230 Oct-12-2020, 09:13 PM
Last Post: bowlofred
  saving data from text file to CSV file in python having delimiter as space K11 1 304 Sep-11-2020, 06:28 AM
Last Post: bowlofred

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020