Python Forum
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Parallel Python
#1
Hi, I'm processing huge data around 13million records, I have my sequential algorithm running for small number of records, my plan is to run the same algorithm for different chunks of data, can any suggest me how could i accomplish the same? How could I I do multiprocessing or multi threading in python? Any supportive material available on line?
Reply
#2
How is the data structured?
You say have a sequential algorithm running, so is the file a collection of records?

Is each record fixed length or variable.
Is there a key associated with each record
If so is at at the same location within each record?

if the answer to the last three questions if yes, you can build a hash table in memory (just the hash and key)
and then use this to create random access to any record

This type of hash table will get you access to the data you want most often on the first read, if not on the first read,
within just a couple.
Reply
#3
Thank you for your kind response
Yes, all 3 conditions are true....... I will check with it...
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  parallel loop python caro 4 1,738 Jun-16-2022, 08:46 PM
Last Post: woooee
  Matlab to Python -- Parallel Computing zistambo 1 1,934 Jun-10-2020, 04:59 PM
Last Post: pyzyx3qwerty
  Use dynamic variable from parallel running python script Sicksym 0 1,823 May-15-2020, 02:52 PM
Last Post: Sicksym
  Parallel Processing in Python with Robot crcali 6 5,100 Apr-06-2018, 03:48 AM
Last Post: Larz60+
  parallel(offset) lines using python johnfriend 1 4,234 May-05-2017, 06:10 AM
Last Post: buran

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020