Python Forum
Extract args=, value that was passed to Multiprocessing.Proc object during runtime?
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Extract args=, value that was passed to Multiprocessing.Proc object during runtime?
#1
The attaced image greatly goes into detail of I'm trying to do. The specific queue passed in:
p = multiprocessing.Process(target=worker, name=str(i), args=(qList[i],))
I would like to extract, and perform a .put() action, from a different worker instance that is running in parallel.
Also if there's a different way to get the current process id in the worker function, to be able to associate it with the current 'individual' command passed in the queue, this would be appreciated. I would prefer not to use os.getpid().

This is the code, with some pseudocode:
def worker(individualCMDfromASubQue):
    cmdToExecute = individualCMDfromASubQue.get(block=False)
    workerPID = qPIDs.get()
    
    ~if a certain type of command, assign workerPID:
        setPID(individualCMDfromASubQue, workerPID)
        # The problem with this approach is that relies on the multiprocessing
        # to execute in a specific order.
        
    ~if certain condition is met, send present individualCMDfromASubQue 
        ~to a different subqueue using PID as reference number
        for idx, i in enumerate(jobs):
            if i.pid == getRecordedPID_ofArecentCMD_stillExecuting(var1):
                if not subQueue[idx].full():        # order that list is appended
                    subQueue[idx].put(cmdToExecute) # again relies on order; what if subqueue is later deleted?                        


subQueue = [multiprocessing.Queue(maxsize=2) for i in range(2)]
qPIDs = queue.Queue()

jobs = []
if __name__ == '__main__':    
    subQueue[1] = [<command 1>, <command 2>, <command 3>]
    subQueue[2] = [<command 4>, <command 5>, <command 6>]

    for i in range(len(subQueue)):
        p = multiprocessing.Process(target=worker, args=(subQueue[i],))
        jobs.append(p)
            
    for p in jobs:
        p.start()
        qPIDs.put(p.pid)   # Generated after starting


Edit: I was able to use the memory address of the queue as a reference point, to decide which queue to put said command in. multiprocessing.current_process().pid was helpful, but would require me to write more code and such. I was able to add more items to each worker queue and have them continue on to the next while True: iteration if their qSize() is not 0.

IMO relying on the memory address isn't the ideal solution to getting the args= queue, passed as an argument.
Reply
#2
The pseudocode is difficult to understand: what do sepPID() and getRecordedPID_ofArecentCMD_stillExecuting(var1) do? also what is var1?

Also it seems that the workers receive queues of commands but they only read a single command in these queues, and they optionally put this command in another queue. Could you explain the intention from a higher level?

Perhaps you could describe which problem led you to this solution design?
« We can solve any problem by introducing an extra level of indirection »
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  change dataclass to frozen at runtime jpanico 2 1,178 Oct-06-2024, 09:38 AM
Last Post: snippsat
  [SOLVED] [listbox] Feed it with dict passed to class? Winfried 3 1,334 May-13-2024, 05:57 AM
Last Post: Larz60+
  class and runtime akbarza 4 1,654 Mar-16-2024, 01:32 PM
Last Post: deanhystad
  How to receive two passed cmdline parameters and access them inside a Python script? pstein 2 1,340 Feb-17-2024, 12:29 PM
Last Post: deanhystad
Question How to compare two parameters in a function that has *args? Milan 4 2,849 Mar-26-2023, 07:43 PM
Last Post: Milan
  Generate lists of devices and partitions from /proc/partitions? DachshundDigital 1 1,482 Feb-28-2023, 10:55 PM
Last Post: deanhystad
  painfully slow runtime when handling data dadazhu 3 2,100 Jan-20-2023, 07:11 PM
Last Post: snippsat
  Big O runtime nested for loop and append yarinsh 4 2,955 Dec-31-2022, 11:50 PM
Last Post: stevendaprano
  python multiprocessing help -- to extract 10 sql table into csv mg24 3 2,709 Nov-20-2022, 11:50 PM
Last Post: mg24
  Reducing runtime memory usage in Cpython interpreter david_the_graower 2 3,019 Oct-18-2021, 09:56 PM
Last Post: david_the_graower

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020