Python Forum
How I can use multiprocessing with upickled module variable?
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
How I can use multiprocessing with upickled module variable?
#1
Python 3.7.1

I use multiprocessing.Pool for paralleling my code, and it work good. But when I add contextvars in working module (for my coroutines, which I created 2 for each process):
import contextvars
line_var = contextvars.ContextVar('my_var')

def my_func():
    line = 1
    line_by_ticker_var.set(line)
    line2 = line_var.get()
I got this error:
Error:
multiprocessing.pool.MaybeEncodingError: Error sending result: '<multiprocessing.pool.ExceptionWithTraceback object at 0x7fe233cccf90>'. Reason: 'TypeError("can't pickle ContextVar objects")'
How can I use multiprocessing and asynchrony (with the necessary module contextvars) at the same time?
Reply
#2
https://docs.python.org/3.7/library/mult...guidelines
https://docs.python.org/3.7/library/mult...-processes

Objects shared between different processes need to be pickleable. ContextVars are apparently not pickleable. So I think the answer to your question is simply: "you can't".
Reply
#3
Mixing multiprocessing and asyncio is not very easy.
(contextvars was made for asyncio)

Use for process communication Queues, Manager (with namespace, list, dict).
Inside the process you should use contextvars for asyncio.
Almost dead, but too lazy to die: https://sourceserver.info
All humans together. We don't need politicians!
Reply
#4
(Oct-30-2019, 07:16 PM)DeaD_EyE Wrote: Inside the process you should use contextvars for asyncio.

If I use contextvars, I get the error from topic.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  How to assign a module to a variable even if it's not defined? mandaxyz 5 3,166 Aug-12-2020, 10:34 PM
Last Post: snippsat
  Multiprocessing share variable catosp 0 2,131 Jul-15-2020, 10:45 AM
Last Post: catosp
  using module variable jacklee26 4 2,186 Nov-13-2019, 01:36 AM
Last Post: jacklee26
  Multiprocessing Module Running An Infinite Child Process Even After Completion hsikora 0 3,837 Dec-19-2018, 08:01 AM
Last Post: hsikora
  Unwanted variable change in module dannyH 2 2,636 May-08-2018, 05:33 PM
Last Post: dannyH
  python import module or namespace from namepsace variable ? harun2525 11 8,523 May-23-2017, 05:39 PM
Last Post: Larz60+

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020