Python Forum
Memory Error While generating cheksum no - Printable Version

+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: General Coding Help (https://python-forum.io/forum-8.html)
+--- Thread: Memory Error While generating cheksum no (/thread-38208.html)



Memory Error While generating cheksum no - mg24 - Sep-16-2022

Hi Team,

I am trying to create checksum no , getting error memory error.
for syntax====> lines = f.read()

file size of CSV is 60gb , how to avoid memory error while creating checksum.


code I am using.
import hashlib
import os

def Create_chk_sum(fpath,tbl):
	fnameCSV = f'{tbl}.csv'
	fn = os.path.join(fpath,fnameCSV)
	fn_dn = fn
	m = hashlib.md5()
	with open(fn,'rb') as f:
		lines = f.read()
		m.update(lines)
		chk_sum = m.hexdigest()
		fn = os.path.join(fpath,f'ChkSum_No_{tbl}.csv')
		with open(fn,'w') as data:
			data.write(chk_sum)

if __name__ == "__main__":   
    folderpath = "C:\\Users\\mg\\OneDrive\\Desktop\\C\\test_data"
    tbl = "SQL_Table1"
    Create_chk_sum(folderpath, tbl)



RE: Memory Error While generating cheksum no - ibreeden - Sep-16-2022

f.read() will try to read the entire file. You tell us the file is 60GB. If your computer does not have 60GB of memory you will get a memory error.
Solution: read the file in chunks of let's say 10240 bytes (f.read(10240)) and repeat until you read a chunk of size 0.


RE: Memory Error While generating cheksum no - mg24 - Sep-25-2022

Hi ibreeden,

superb ! Thanks for your help. it worked.



Thanks
mg