![]() |
Memory Error While generating cheksum no - Printable Version +- Python Forum (https://python-forum.io) +-- Forum: Python Coding (https://python-forum.io/forum-7.html) +--- Forum: General Coding Help (https://python-forum.io/forum-8.html) +--- Thread: Memory Error While generating cheksum no (/thread-38208.html) |
Memory Error While generating cheksum no - mg24 - Sep-16-2022 Hi Team, I am trying to create checksum no , getting error memory error. for syntax====> lines = f.read() file size of CSV is 60gb , how to avoid memory error while creating checksum. code I am using. import hashlib import os def Create_chk_sum(fpath,tbl): fnameCSV = f'{tbl}.csv' fn = os.path.join(fpath,fnameCSV) fn_dn = fn m = hashlib.md5() with open(fn,'rb') as f: lines = f.read() m.update(lines) chk_sum = m.hexdigest() fn = os.path.join(fpath,f'ChkSum_No_{tbl}.csv') with open(fn,'w') as data: data.write(chk_sum) if __name__ == "__main__": folderpath = "C:\\Users\\mg\\OneDrive\\Desktop\\C\\test_data" tbl = "SQL_Table1" Create_chk_sum(folderpath, tbl) RE: Memory Error While generating cheksum no - ibreeden - Sep-16-2022 f.read() will try to read the entire file. You tell us the file is 60GB. If your computer does not have 60GB of memory you will get a memory error.Solution: read the file in chunks of let's say 10240 bytes ( f.read(10240) ) and repeat until you read a chunk of size 0.
RE: Memory Error While generating cheksum no - mg24 - Sep-25-2022 Hi ibreeden, superb ! Thanks for your help. it worked. Thanks mg |