Python Forum
Multi-processing - problem with running multiple *.py files at the same time
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Multi-processing - problem with running multiple *.py files at the same time
#3
I check the task manager and it's only one core of the CPU that is working; plus the time is 5 times more than running a single file (I have 5 files in the loop as you see). The point is, how do I run the for loop in parallel?

(Sep-11-2018, 04:16 AM)Antonio Wrote: Hi there,

I have multiple *.py files that I need to run concurrently. Each file looks like this:

def my_function(file):

    # my code
Then I have written this to run my files simultaneously:

import multiprocessing

from file1 import my_function

if __name__ == '__main__':
    __spec__ = "ModuleSpec(name='builtins', loader=<class '_frozen_importlib.BuiltinImporter'>)" # This line is because I am using Spyder as my IDE

    files = ["path/to/file1.py","path/to/file2.py","path/to/file3.py","path/to/file4.py","path/to/file5.py"]

    for item in files:
        p = multiprocessing.Process(target=my_function, args=(item,))
        p.start()
        p.join()
        print("Done!")
The issue is that, however, the files are still being run one after another like in a sequence but not in parallel! Do you have any ideas how I may run my files concurrently?
Reply


Messages In This Thread
RE: Multi-processing - problem with running multiple *.py files at the same time - by Antonio - Sep-11-2018, 08:32 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
  Trying to generating multiple json files using python script dzgn989 4 309 May-10-2024, 03:09 PM
Last Post: deanhystad
  Filer and sort files by modification time in a directory tester_V 5 458 May-02-2024, 05:39 PM
Last Post: tester_V
  problem in running a code akbarza 7 792 Feb-14-2024, 02:57 PM
Last Post: snippsat
  python convert multiple files to multiple lists MCL169 6 1,714 Nov-25-2023, 05:31 AM
Last Post: Iqratech
Question Need Help with Vehicle Routing Problem with Time Windows (VRPTW) in Python kasper321421312 1 690 Nov-10-2023, 08:19 PM
Last Post: snippsat
  Downloading time zone aware files, getting wrong files(by date))s tester_V 9 1,221 Jul-23-2023, 08:32 AM
Last Post: deanhystad
  splitting file into multiple files by searching for string AlphaInc 2 1,012 Jul-01-2023, 10:35 PM
Last Post: Pedroski55
  Processing Files that are not in use randywberry 3 765 Jun-06-2023, 06:00 PM
Last Post: rajeshgk
  Merging multiple csv files with same X,Y,Z in each Auz_Pete 3 1,295 Feb-21-2023, 04:21 AM
Last Post: Auz_Pete
  unittest generates multiple files for each of my test case, how do I change to 1 file zsousa 0 1,015 Feb-15-2023, 05:34 PM
Last Post: zsousa

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020