[solved] how to speed-up huge data in an ascii file ?

[paul18fr]

I've not paid any attention on that topic so far, but now I'm wondering how I can write (in a fast way) a huge amount of data in an ascii file (that's the current specification!)

The following snippet mimics of what I'm trying to do

import os, time
import numpy as np
import io
BufferSize = io.DEFAULT_BUFFER_SIZE
print (f"Default buffer size={BufferSize}")

path=str(os.getcwd())
FileName="myFile.txt"

n=1_000_000# 10_000_000
M=np.random.random((n, 3))

# without buffering
t1_0=time.time()
with open (path+'/'+FileName[:-4]+'_0.txt', 'w') as f:
    for i in range(n):
        f.write(f" X={M[i, 0]}, Y={M[i, 1]}, Z={M[i, 2]}\n")
t1_1=time.time()
print(f"With loops, duration={t1_1-t1_0}")

# with buffering
t2_0=time.time()
TestValue = 2**10
with open (path+'/'+FileName[:-4]+'_1.txt', 'w', buffering = TestValue) as f:
    for i in range(n):
        f.write(f" X={M[i, 0]}, Y={M[i, 1]}, Z={M[i, 2]}\n")
t2_1=time.time()
print(f"With buffering, duration={t2_1-t2_0}")

One can identify 2 mains issues at least:

1. the use of a loop
   
2. write function is called for each llop, which is time consuming
   

I'm currently trying to understand how to use buffering, but in practise, the value remains unclear for now; any general advice on how to write huge data in an ascii file?

Thanks

P.