Python Forum
[solved] how to speed-up huge data in an ascii file ?
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
[solved] how to speed-up huge data in an ascii file ?
#4
The problem here is that you are not only saving the data to a file, you are creating the data. It is the data creation that takes time, not the writing.

You could perhaps play with the fmt keyword argument in numpy.savetxt, something like 'X=%.18e, Y=%.18e, Z=%.18e'.
import numpy as np
import sys
n=10 # 1_000_000# 10_000_000
M=np.random.random((n, 3))

np.savetxt(sys.stdout, M, fmt='X=%.18e, Y=%.18e, Z=%.18e')
Output:
X=1.778314452437265158e-01, Y=7.362842666045655848e-01, Z=8.358127207042234108e-01 X=5.591744788035918345e-01, Y=7.845951465943425962e-01, Z=6.039963855998189413e-01 X=8.327560563335355548e-01, Y=6.042091153798287984e-01, Z=1.590375469584719426e-01 X=9.855324666099820607e-01, Y=6.029884572061958714e-01, Z=3.114472999689985588e-01 X=7.433919307334269089e-01, Y=2.941350276294346644e-01, Z=6.780499010590056441e-01 X=7.791133512845780373e-01, Y=2.911042379946882086e-01, Z=8.546676365400691644e-01 X=2.481689914304145983e-01, Y=4.970687878118742464e-01, Z=1.684596602818245747e-01 X=5.134374653560572765e-01, Y=1.239447760698755285e-01, Z=3.211991817077095579e-01 X=6.812413908422187969e-02, Y=9.637812239995832142e-01, Z=8.384101532932353162e-01 X=7.187521467518211971e-01, Y=9.752591487821623550e-01, Z=1.938176050010664841e-01
Reply


Messages In This Thread
RE: how to speed-up huge data in an ascii file ? - by Gribouillis - May-16-2023, 11:58 AM

Possibly Related Threads…
Thread Author Replies Views Last Post
  Finding the median of a column in a huge CSV file markagregory 5 1,920 Jan-24-2023, 04:22 PM
Last Post: DeaD_EyE
Smile How to further boost the data read write speed using pandas tjk9501 1 1,332 Nov-14-2022, 01:46 PM
Last Post: jefsummers
  visualizing huge correation matrix erdemath 3 2,142 Oct-13-2021, 09:44 AM
Last Post: erdemath
  [solved] Save a matplotlib figure into hdf5 file paul18fr 1 2,621 Jun-08-2021, 05:58 PM
Last Post: paul18fr
  huge and weird values after applying some calculations karlito 2 2,237 Dec-13-2019, 08:32 AM
Last Post: karlito
  [SOLVED on SO] Downsizing non-representative data in DataFrame volcano63 1 2,249 Sep-28-2018, 12:56 PM
Last Post: volcano63
  Loading HUGE data from Python into SQL SERVER Sandeep 2 21,237 Jan-13-2018, 07:52 AM
Last Post: Sandeep

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020