Python Forum
[solved] how to speed-up huge data in an ascii file ?
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
[solved] how to speed-up huge data in an ascii file ?
#5
Thanks Gribouillis,

Your advice helped me a lot; based on your example, I modified it a bit in order to concatenate different data with different sizes.

import numpy as np
import os

path = str(os.getcwd())

# Creating a numpy array
n = 10 # 1_000_000# 10_000_000
M1 = np.random.random((n, 3))

m = 100
M2 = np.random.random((m, 10))

# Opening a file
with open(path + '/file.txt','w+') as f :
    
    #appending M1
    np.savetxt(f, M1, fmt='X=%.18e, Y=%.18e, Z=%.18e', header = 'blabla')
    # add of an intermediate line
    f.write("####### a comment is added\n")
    np.savetxt(f, M2)
Thanks

Paul
Reply


Messages In This Thread
RE: [solved] how to speed-up huge data in an ascii file ? - by paul18fr - May-16-2023, 08:36 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
  Finding the median of a column in a huge CSV file markagregory 5 1,914 Jan-24-2023, 04:22 PM
Last Post: DeaD_EyE
Smile How to further boost the data read write speed using pandas tjk9501 1 1,332 Nov-14-2022, 01:46 PM
Last Post: jefsummers
  visualizing huge correation matrix erdemath 3 2,141 Oct-13-2021, 09:44 AM
Last Post: erdemath
  [solved] Save a matplotlib figure into hdf5 file paul18fr 1 2,615 Jun-08-2021, 05:58 PM
Last Post: paul18fr
  huge and weird values after applying some calculations karlito 2 2,235 Dec-13-2019, 08:32 AM
Last Post: karlito
  [SOLVED on SO] Downsizing non-representative data in DataFrame volcano63 1 2,248 Sep-28-2018, 12:56 PM
Last Post: volcano63
  Loading HUGE data from Python into SQL SERVER Sandeep 2 21,235 Jan-13-2018, 07:52 AM
Last Post: Sandeep

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020