Python Forum

Full Version: Segmentation fault with large files
You're currently viewing a stripped down version of our content. View the full version with proper formatting.
Hi,

Here I'm trying to remove double quotation character from all text files. I'm getting an error "Segmentation fault" while I reading more than 8gb files in folder '/data/DWH/29SEP/'. Any comments?

import os

def Main():
    path = '/data/DWH/29SEP/'
    files = []
    for r, d, f in os.walk(path):
                for file in f:
                        if '.TXT' in file:
                                files.append(os.path.join(r, file))
    for f in files:
        print(f)
        with open(f + 'e', 'w') as outfile:
            with open(f, 'r') as infile:
                temp = infile.read().replace("\"", "")
                outfile.write(temp)

if __name__ == "__main__":
    Main()
infile.read() loads the whole content of infile at once in memory. Read the file by chunks instead.
(Oct-01-2019, 07:17 AM)Gribouillis Wrote: [ -> ]infile.read() loads the whole content of infile at once in memory. Read the file by chunks instead.

Hi Gribouillis,

Have you sample code?

Regards
Kusal
Kusal1 Wrote:Hi Gribouillis,

Have you sample code?
Yes