Python Forum
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Multithreading alternative
#1
Inside my project I have a loop of 13 iterations which includes some feature matching. The code needs to run real time, and the only way to achieve this is by doing this loop in parallel. I tried a couple of ways to implement parallel processing with ThreadPool but there was no improvement whatsoever.

from multiprocessing.pool import ThreadPool
import multiprocessing as multip

threads = multip.Pool(multip.cpu_count())
pool = ThreadPool(threads)


def calculateParallel():
    results = pool.map(temp_match, list(range(13)))
    pool.close()
    pool.join()
    return results
where temp_match is a 30 line image processing procedure which return a ndarray.
Is there a way to go around this? From what I read online GIL makes it almost impossible to run multiple CPU threads.
I have seen a lot of posts on forums and tutorials claiming their particular code does the job, but I have not seen any improvement with any of them.
Thanks in advance!
Reply
#2
ThreadPool isn't officially documented. That is, at least, one thing that could be considered suspicious.
You definitely have CPU-bound task here. From here you can find that ThreadPool isn't suitable for your task. Try to use Pool instead. Additionally, you can rewrite/optimize your 35-line image processing procedure using Cython (with no-gil clause), use numba-jit.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  multithreading xlwings error Equivocal 0 347 Jan-25-2025, 05:10 PM
Last Post: Equivocal
  multithreading Hanyx 4 2,100 Jul-29-2022, 07:28 AM
Last Post: Larz60+
Question Problems with variables in multithreading Wombaz 2 2,021 Mar-08-2022, 03:32 PM
Last Post: Wombaz
  Multithreading question amadeok 0 2,288 Oct-17-2020, 12:54 PM
Last Post: amadeok
  How can i add multithreading in this example WoodyWoodpecker1 3 3,340 Aug-11-2020, 05:30 PM
Last Post: deanhystad
  matplotlib multithreading catosp 0 3,537 Jul-03-2020, 09:33 AM
Last Post: catosp
  Multithreading dynamically syncronism Rodrigo 0 1,993 Nov-08-2019, 02:33 AM
Last Post: Rodrigo
  Locks in Multithreading Chuonon 0 2,290 Oct-03-2019, 04:16 PM
Last Post: Chuonon
  multithreading issue with output mr_byte31 4 4,411 Sep-11-2019, 12:04 PM
Last Post: stullis
  using locks in multithreading in python3 srm 2 4,543 Jul-13-2019, 11:35 AM
Last Post: noisefloor

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020