Segmentation fault with large files

Segmentation fault with large files - Printable Version

+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: General Coding Help (https://python-forum.io/forum-8.html)
+--- Thread: Segmentation fault with large files (/thread-21461.html)

Segmentation fault with large files - kusal1 - Oct-01-2019

Hi,

Here I'm trying to remove double quotation character from all text files. I'm getting an error "Segmentation fault" while I reading more than 8gb files in folder '/data/DWH/29SEP/'. Any comments?

import os

def Main():
    path = '/data/DWH/29SEP/'
    files = []
    for r, d, f in os.walk(path):
                for file in f:
                        if '.TXT' in file:
                                files.append(os.path.join(r, file))
    for f in files:
        print(f)
        with open(f + 'e', 'w') as outfile:
            with open(f, 'r') as infile:
                temp = infile.read().replace("\"", "")
                outfile.write(temp)

if __name__ == "__main__":
    Main()

RE: Segmentation fault with large files - Gribouillis - Oct-01-2019

infile.read() loads the whole content of infile at once in memory. Read the file by chunks instead.

RE: Segmentation fault with large files - kusal1 - Oct-01-2019

(Oct-01-2019, 07:17 AM)Gribouillis Wrote: infile.read() loads the whole content of infile at once in memory. Read the file by chunks instead.

Hi Gribouillis,

Have you sample code?

Regards
Kusal

RE: Segmentation fault with large files - Gribouillis - Oct-01-2019

Kusal1 Wrote:Hi Gribouillis,

Have you sample code?

Yes