Python Forum
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
two iterations in parallel
#1
i have a massively huge set of hundreds of millions of numbers in a file. memory is not large enough to hold all these numbers in one list. so to run some statistics like statistics.mean() or statistics.median() i have made an iterator, which statistics.mean() and statistics.median() will accept, that reads these numbers in, one by one. the iterator is a bit complicated because it has to find the appropriate number on each line.

what i want to do is get both statistics, mean and median in just one pass over the file of numbers, in the future there could be more statistics. there will be several new sets of data.

basically, i have one iterator and two (or more) needs to iterate it that operate in parallel. is there a way to do this in Python?

i have written many generators before but in every case there is only one target for yield. now i think i need two. i have never done coroutines and wonder if that can solve this problem.
Tradition is peer pressure from dead people

What do you call someone who speaks three languages? Trilingual. Two languages? Bilingual. One language? American.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Redundant Iterations leoahum 3 2,202 Mar-13-2019, 07:26 PM
Last Post: ichabod801

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020