Sep-16-2022, 12:21 AM
Hi Team,
I am trying to create checksum no , getting error memory error.
for syntax====> lines = f.read()
file size of CSV is 60gb , how to avoid memory error while creating checksum.
I am trying to create checksum no , getting error memory error.
for syntax====> lines = f.read()
file size of CSV is 60gb , how to avoid memory error while creating checksum.
code I am using. import hashlib import os def Create_chk_sum(fpath,tbl): fnameCSV = f'{tbl}.csv' fn = os.path.join(fpath,fnameCSV) fn_dn = fn m = hashlib.md5() with open(fn,'rb') as f: lines = f.read() m.update(lines) chk_sum = m.hexdigest() fn = os.path.join(fpath,f'ChkSum_No_{tbl}.csv') with open(fn,'w') as data: data.write(chk_sum) if __name__ == "__main__": folderpath = "C:\\Users\\mg\\OneDrive\\Desktop\\C\\test_data" tbl = "SQL_Table1" Create_chk_sum(folderpath, tbl)