Python Forum
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Iterating Large Files
#10
(Jul-22-2020, 06:09 PM)Gribouillis Wrote: s = np.array([9.94223926e-01, 7.55188959e-01, 2.87075284e-04, 6.60265593e-01,
       3.12176498e-02, 3.01580980e-01, 9.79960201e-01, 2.37826251e-01,
       1.74042656e-01, 1.39546100e-02, 2.14055048e-01, 8.73880775e-01,
       5.12656017e-01])

Correct me if I am wrong: this should give an issue as my array belongs to a large text file (10 GB, Nx4)? First question is how do I load my file and define in s? np.loadtxt is failing giving memory error, can we use h5py (or memmap) to load all the data to hard disk at once and sort based on column 2?
Reply


Messages In This Thread
Iterating Large Files - by Robotguy - Jun-25-2020, 10:46 PM
RE: Iterating Large Files - by Gribouillis - Jun-26-2020, 10:00 AM
RE: Iterating Large Files - by Robotguy - Jul-15-2020, 08:54 PM
RE: Iterating Large Files - by Gribouillis - Jul-15-2020, 11:01 PM
RE: Iterating Large Files - by Robotguy - Jul-17-2020, 04:23 PM
RE: Iterating Large Files - by Gribouillis - Jul-16-2020, 07:11 AM
RE: Iterating Large Files - by Gribouillis - Jul-17-2020, 07:41 PM
RE: Iterating Large Files - by Robotguy - Jul-22-2020, 03:23 PM
RE: Iterating Large Files - by Gribouillis - Jul-22-2020, 06:09 PM
RE: Iterating Large Files - by Robotguy - Jul-22-2020, 08:46 PM
RE: Iterating Large Files - by Gribouillis - Jul-22-2020, 09:13 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
  Iterate 2 large text files across lines and replace lines in second file medatib531 13 6,109 Aug-10-2020, 11:01 PM
Last Post: medatib531
  Handling Large XML Files (>10GB) in Python onlydibs 1 4,265 Dec-22-2019, 05:46 AM
Last Post: Clunk_Head
  Segmentation fault with large files kusal1 3 2,833 Oct-01-2019, 07:32 AM
Last Post: Gribouillis
  Compare two large CSV files for a match Python_Newbie9 3 5,864 Apr-22-2019, 08:49 PM
Last Post: ichabod801
  Comparing values in large txt files StevenVF 2 2,794 Feb-28-2019, 09:07 AM
Last Post: StevenVF
  Download multiple large json files at once halcynthis 0 2,826 Feb-14-2019, 08:41 AM
Last Post: halcynthis
  iterating over files clarablanes 17 7,422 Aug-30-2018, 02:18 PM
Last Post: clarablanes

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020