Python Forum
How I can use multiprocessing with upickled module variable?
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
How I can use multiprocessing with upickled module variable?
#1
Python 3.7.1

I use multiprocessing.Pool for paralleling my code, and it work good. But when I add contextvars in working module (for my coroutines, which I created 2 for each process):
import contextvars
line_var = contextvars.ContextVar('my_var')

def my_func():
    line = 1
    line_by_ticker_var.set(line)
    line2 = line_var.get()
I got this error:
Error:
multiprocessing.pool.MaybeEncodingError: Error sending result: '<multiprocessing.pool.ExceptionWithTraceback object at 0x7fe233cccf90>'. Reason: 'TypeError("can't pickle ContextVar objects")'
How can I use multiprocessing and asynchrony (with the necessary module contextvars) at the same time?
Reply
#2
https://docs.python.org/3.7/library/mult...guidelines
https://docs.python.org/3.7/library/mult...-processes

Objects shared between different processes need to be pickleable. ContextVars are apparently not pickleable. So I think the answer to your question is simply: "you can't".
Reply
#3
Mixing multiprocessing and asyncio is not very easy.
(contextvars was made for asyncio)

Use for process communication Queues, Manager (with namespace, list, dict).
Inside the process you should use contextvars for asyncio.
Almost dead, but too lazy to die: https://sourceserver.info
All humans together. We don't need politicians!
Reply
#4
(Oct-30-2019, 07:16 PM)DeaD_EyE Wrote: Inside the process you should use contextvars for asyncio.

If I use contextvars, I get the error from topic.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  How to assign a module to a variable even if it's not defined? mandaxyz 5 3,161 Aug-12-2020, 10:34 PM
Last Post: snippsat
  Multiprocessing share variable catosp 0 2,130 Jul-15-2020, 10:45 AM
Last Post: catosp
  using module variable jacklee26 4 2,184 Nov-13-2019, 01:36 AM
Last Post: jacklee26
  Multiprocessing Module Running An Infinite Child Process Even After Completion hsikora 0 3,835 Dec-19-2018, 08:01 AM
Last Post: hsikora
  Unwanted variable change in module dannyH 2 2,632 May-08-2018, 05:33 PM
Last Post: dannyH
  python import module or namespace from namepsace variable ? harun2525 11 8,515 May-23-2017, 05:39 PM
Last Post: Larz60+

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020