Python Forum
Sharing memory across threads / scripts
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Sharing memory across threads / scripts
#1
Hi,

I am trying to gather data from independent python scripts (data acquisition devices) into the main program. I came across multiprocessing, but I still cannot piece it together. Here is the general architecture:

Main program:
* execute device1.py
* execute device2.py

Device1.py
* continuously collects data from a GPIB instrument and places it to shared memory/variable device1.voltage

Device1.py
* continuously collects data from a a DAQ card and places it to shared memory/variable device2.temperature

Main program:
* at predefined interval, it retrieves device1.voltage & device2.temperature and stores them in a log file.
* when main is being shut down, it first termites both device1.py and device2.py

This is a part of a flexible data acquisition suite where different devices can be executed depending on needs. They all produce some kind of data which is collected by the main for recording. At this moment this is my plan, but if there is better way I'd like to hear it.

Thank you.

Best,
Radek
Reply
#2
A quick update.

I think I figured it out. I have always assumed that the child process (an instrument script) has to run. I was trying to run as a subpocessor and I was failing to share data with its parent. It finally occurred to me that my device can run as a "function" inside a new script. I can than use multiprocess and shared arguments to pass data.

main_multi.py (parent):
from multiprocessing import Process, Value, Pipe
import time
import os

# If there is a code outside of main() and other functions
# it will be executed here.
import device1

def main():
	reading.append(Value('d', 0.0))
	parent_com, child_com = Pipe()
		
	proc = Process(target=device1.instr, args=(reading, child_com))
	proc.start()
		
	for i in range(20):		
		print( 'Instrument1 = {}'.format(reading.value) )
		time.sleep(2)
	
	print ( 'Main is done ... Sending termination signal to device 1 via Pipe()' )
	parent_com.send(False)
	time.sleep(1)
	print ( 'Exit code for device 1: {}'.format(proc.exitcode) )
	
if __name__ == '__main__':
	main()	
device1.py (child)
from multiprocessing import Value
import time
import os
from random import gauss

def instr( result, conn ):
	result.value = 10.0
	running = True	
	while running:
        # check if anything coming via Pipe()
        # when message is "False", loop stops
		if conn.poll():
			running = conn.recv()
			print ( "Message is ", running)
		# generate fake instrument reading
		result.value += gauss(0, 1)
		time.sleep(.1)	
	return 0

def main():
	pass

if __name__ == "__main__":
	main()
I have also managed to extend it to more devices. One can loop over bunch of "devices" and excuse all scrips via for loop. Keep in mind that you will need same number of processes and pipes to connect to all your devices.

Keep in mind that Pipe() is bi-directional. Child can report back to parent when its running, etc.

On a side note. I was a bit confused by the main construct ...
if __name__ == "__main__":
	main()
Here it works well. When child is called by its parent, only "instr" function is called; main() is not executed. However, main() can be used to run it as a stand alone device if needed. <sorry, teaching moment for me :-) >

I hope it helps ...

If there is a better solution I would like to head about it ... I am just starting to code in python (about a month on and off) so I am sure these are more like hacks rather than elegant scripts.

Now I just need a graphical interface for all devices and I am in business!

Peace out!
Radek


P.S.
import os

is there for checking PIDs for various processes ... just saying ...
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Best way to secure API key when sharing quarinteen 2 292 Jan-19-2024, 04:46 PM
Last Post: deanhystad
  How to install modules for 2.7 (or sharing from 3.8)) Persisto 2 2,375 Dec-31-2021, 02:33 PM
Last Post: Persisto
  multiprocessing and sharing object cyrduf 0 2,012 Feb-02-2021, 08:16 PM
Last Post: cyrduf
  sharing variables between two processes Kiyoshi767 1 1,848 Nov-07-2020, 04:00 AM
Last Post: ndc85430
  Sharing my code emirasal 2 1,999 Oct-04-2020, 02:21 PM
Last Post: emirasal
  Sharing X Axis in Sub plots JoeDainton123 1 2,142 Aug-22-2020, 04:11 AM
Last Post: deanhystad
  sharing memory(read-write) between multiple processes mike000 1 2,157 Oct-31-2019, 08:45 PM
Last Post: woooee
  Instances sharing attributes midarq 4 2,446 Sep-20-2019, 11:13 AM
Last Post: midarq
  Sharing variables across modules j.crater 4 3,408 Jul-30-2018, 09:09 PM
Last Post: j.crater
  Output not printing when execute scripts in threads anna 1 2,706 Mar-21-2018, 05:08 PM
Last Post: woooee

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020