Python Forum
Read microcontroller eeprom and feed into python to find checksums.
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Read microcontroller eeprom and feed into python to find checksums.
#1
Good afternoon,

Looking for a little help if possible.
I've done some coding in python but not a massive amount.
In this case I've never done anything with checksums - especially CRC16 - CCITT.

I have a project car and I'm looking to make some mods such as heated seats. power fold mirrors etc. This part is easy enough, read the microcontroller EEPROM, change the byte options from 0x00 to 0x01 however I need to find and recalculate the CRC16-CCITT.(there are many within the binary file)

The CRC16-CCITT's could be anywhere in the file but are straight after the variable length data.

Basically I'm looking to have python script that will read the file byte by byte increasing the data one byte at a time, calculate the checksum and compare it with the next two bytes in the file. When the calculated checksum matches the next two bytes I want to ouput the start address of data, end address of data, checksum location and the actual checksum. Of course it needs to do this repeatedly until the end of file is reached.

When I know these parameters I will be able to mod the data and just recalculate the checksum. change it and reflash the eeprom.

In fairness I know how to read and write files etc in python but I need some pointers on CRC16-CCITT 0xFFFF checksums and how to implement them.
Will the processing overheads be too long and too slow?

To give a quick idea I have copied and pasted some of the data below:

80 07 24 06 FF 00 00 00 FF FF FF FF FF FF FF FF AC DC FF 03 88 DC FF 03 60 DC FF 03 20 56 06 01
7F 00 64 07 7D 82 64 07 81 82 64 07 99 82 7F 00 80 57 04 00 20 36 34 B0 46 36 FF 03 A6 51 7F 00
40 56 00 04 2A 56 94 B0 40 5E 00 04 2B 5E 9C B0 C5 1D 40 56 00 04 2A 56 9C B0 40 5E 00 04 2B 5E
4A B1 B5 15 40 56 00 04 2A 56 4C B1 40 5E 00 04 2B 5E F6 B2 A5 0D 40 56 00 04 2A 56 F8 B2 40 5E
00 04 2B 5E 24 B6 95 05 66 57 01 00 AA 59 67 5F 01 00 7F 00 00 00 00 00 80 07 06 00 7F 00 5C 1A
63 0F 01 00 A0 07 21 00 63 37 3D 00 63 3F 39 00 63 47 35 00 63 4F 31 00 63 57 2D 00 63 5F 29 00
63 67 25 00 63 6F 21 00 63 77 1D 00 63 7F 19 00 63 87 15 00 63 8F 11 00 63 97 0D 00 63 9F 09 00
F0 0F 40 00 63 0F 05 00 F1 0F 40 00 63 0F 01 00 BF FF AC FF 23 0F 01 00 E1 8F 20 00 23 0F 05 00
E1 87 20 00 23 9F 09 00 23 97 0D 00 23 8F 11 00 23 87 15 00 23 7F 19 00 23 77 1D 00 23 6F 21 00
23 67 25 00 23 5F 29 00 23 57 2D 00 23 4F 31 00 23 47 35 00 23 3F 39 00 23 37 3D 00 60 06 20 00
23 0F 01 00 44 1A E0 07 40 01 56 94 86 00 20 5E 32 00 20 56 20 00 40 37 .....data continues>>>

As you can see in the data above, the checksum (in bold) could be anywhere due to the data length(unknown). After the checksum the next data block starts followed by the next checksum.
My main task is to find these checksums within the data and log their locations.
Hope you guys can point me in the right direction.

Thanks in advance.
Mike
Reply
#2
This package looks like it might have what you need to calculate chacksum: https://pypi.org/project/crcengine/
Reply
#3
Thank you for the reply.

The package has helped a lot, I can now find the checksums and log the start and end of the data blocks.
Can anyone give me some pointers of speeding up this script. I am reading a 2048kb file and calculating all data blocks with their checksums. However it is so so slow. Running the script on the file for 24hrs and the script only read through 23% of the file, I don't really want to be running the computer for 4 days on each file if I can cut down the time.
My coding is as below:

#!/usr/bin/python

import os
import sys
import crcengine

def progress(count, total, suffix=''):
    bar_len = 60
    filled_len = int(round(bar_len * count / float(total)))

    percents = round(100.0 * count / float(total), 1)
    bar = '=' * filled_len + '-' * (bar_len - filled_len)

    sys.stdout.write('[%s] %s%s ...%s\r' % (bar, percents, '%', suffix))
    sys.stdout.flush()


def main(argv):
	inputfile = sys.argv[1]
	inputfilesize = os.path.getsize(sys.argv[1])
	outputfile = sys.argv[1]+'.txt'
	woutputfiledata = open(outputfile, 'a+')

	print ('Input file is:' + inputfile)
	print ('Finding checksums...')

	f1 = open(inputfile, "rb")   	# open argv[1] file for binary reading
									#
	block = 1						# used to keep track of found blocks - increments
	previousPos = 0					# starts at zero - increments by amount of bytes found in each block
	currentPos = 0					# current byte position within the file
	bytelist = []					# bytes currently read
	message = b''					# hex string passed to the crcengine function
	chkmessage = b''				# calculated checksum of bytelist and message used to see if it equal to the last two bytes
	CRCcheck = ['00','00']			# last two bytes of bytelist - compared to the checksum of bytelist[-2]
	CRCcheckMessage = 'CRC=0x'			# CRCcheckMessage to compare to chkmessage
	crc_algorithm = crcengine.new('crc16-ccitt-false')	# set up crc algorithm
	successcounter = 0
	
	while 1:
		byte_s = f1.read(1)		# read 1 byte from the file
		if not byte_s:			# if no bytes read quit
			f1.close()
			break
			
		bytelist.append(byte_s)  # add read byte to bytelist
		
		while len(bytelist) < 3: 	# check theres at least 3 bytes
			byte_s = f1.read(1)		# last two are used as the checksum comparitor
			currentPos +=1			# increase currentPos to keep track of bytes read and position within file
			bytelist.append(byte_s) # then add the byte to bytelist
			
		CRCcheck[0] = bytelist[-2]	# add last but one byte read to the first position of the checksum list
		CRCcheck[1] = bytelist[-1]	# add last bite read to the second position of the checksum list
		message = message + bytelist[-3]	# add all bytes bar the last two to the byte string
		result = crc_algorithm(message)		# send the byte string and store the returned checksum int into result
		chkmessage ='CRC=0x{:04x}'.format(result)	# convert checksum message into a 2 byte hex string
		
		CRCcheckMessage = CRCcheckMessage + CRCcheck[0].hex()
		CRCcheckMessage = CRCcheckMessage + CRCcheck[1].hex()
		
		if CRCcheckMessage == chkmessage:
			woutputfiledata.write('BLK: '+ str(block)+'\n')
			woutputfiledata.write('SOB: '+ '0x{:08x}'.format(previousPos)+'\n')
			woutputfiledata.write('EOB: '+ '0x{:08x}'.format(currentPos-2)+'\n')
			woutputfiledata.write('CKS: '+ chkmessage+'\n')
			woutputfiledata.write('\n')
			block += 1
			successcounter += 1
			previousPos = currentPos + 1
			bytelist.clear()
			
		CRCcheckMessage = 'CRC=0x'
		currentPos +=1
		progress(currentPos, inputfilesize)

if __name__ == "__main__":
	main(sys.argv[1:])
I will only ever have to read a file once, when I have all the checksums I will be able to just mod a particular block and recalculate that blocks checksum.

The script ouputs to a txt file like below:

BLK: 1
SOB: 0x00000000
EOB: 0x000084fb
CKS: CRC=0x1a49

BLK: 2
SOB: 0x000084fe
EOB: 0x00039b34
CKS: CRC=0x4444

BLK: 3
SOB: 0x00039b37
EOB: 0x0006d4cb
CKS: CRC=0xffff

BLK: 4
SOB: 0x0006d4ce
EOB: 0x000754cc
CKS: CRC=0xffff

This is the last file I tried and it seems to be working from manually checking the file locations. I just need to be able to run through the file quicker.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  How to parse a live feed in Python? Daring_T 2 150 Jan-20-2022, 04:17 AM
Last Post: Daring_T
  Feed List items with Integer euras 9 1,509 May-19-2021, 07:45 PM
Last Post: snippsat
  splitting UAV/sat images to smaller pieces in order to feed a CNN hobbyist 0 648 Dec-08-2020, 11:48 AM
Last Post: hobbyist
  Rotation Effect on live Webcam Feed Leziiy 0 738 Sep-12-2020, 04:25 PM
Last Post: Leziiy
  Replace every 20th space with line feed T_Lafferty 3 1,073 Jul-19-2020, 10:37 PM
Last Post: T_Lafferty
  Help to find a string and read the next lines crlamaral 4 1,353 Mar-19-2020, 09:24 AM
Last Post: Larz60+
  How to remove form feed from file gurbhej_singh 5 1,852 May-26-2019, 05:51 AM
Last Post: gurbhej_singh
  programming external 93C66 EEPROM with Arduino UNO Mantix 0 1,877 Sep-22-2018, 06:52 AM
Last Post: Mantix
  Read and find elements which contains text craver35 2 1,362 Aug-12-2018, 04:57 PM
Last Post: craver35
  Cant find S3 Read or Write access from Python code tpc 0 1,444 Apr-19-2018, 04:00 AM
Last Post: tpc

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020