Python Forum
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Multithreading alternative
#1
Inside my project I have a loop of 13 iterations which includes some feature matching. The code needs to run real time, and the only way to achieve this is by doing this loop in parallel. I tried a couple of ways to implement parallel processing with ThreadPool but there was no improvement whatsoever.

from multiprocessing.pool import ThreadPool
import multiprocessing as multip

threads = multip.Pool(multip.cpu_count())
pool = ThreadPool(threads)


def calculateParallel():
    results = pool.map(temp_match, list(range(13)))
    pool.close()
    pool.join()
    return results
where temp_match is a 30 line image processing procedure which return a ndarray.
Is there a way to go around this? From what I read online GIL makes it almost impossible to run multiple CPU threads.
I have seen a lot of posts on forums and tutorials claiming their particular code does the job, but I have not seen any improvement with any of them.
Thanks in advance!
Reply
#2
ThreadPool isn't officially documented. That is, at least, one thing that could be considered suspicious.
You definitely have CPU-bound task here. From here you can find that ThreadPool isn't suitable for your task. Try to use Pool instead. Additionally, you can rewrite/optimize your 35-line image processing procedure using Cython (with no-gil clause), use numba-jit.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Twilio alternative jmair 3 3,886 Feb-08-2024, 01:55 PM
Last Post: Sharmi
  Pillow alternative? kucingkembar 4 864 Jul-27-2023, 10:50 AM
Last Post: Larz60+
  multithreading Hanyx 4 1,319 Jul-29-2022, 07:28 AM
Last Post: Larz60+
Question Problems with variables in multithreading Wombaz 2 1,320 Mar-08-2022, 03:32 PM
Last Post: Wombaz
  Alternative for Cairosvg? Maryan 0 2,448 Oct-26-2020, 01:27 PM
Last Post: Maryan
  Multithreading question amadeok 0 1,772 Oct-17-2020, 12:54 PM
Last Post: amadeok
  How can i add multithreading in this example WoodyWoodpecker1 3 2,500 Aug-11-2020, 05:30 PM
Last Post: deanhystad
  matplotlib multithreading catosp 0 2,939 Jul-03-2020, 09:33 AM
Last Post: catosp
  Multithreading dynamically syncronism Rodrigo 0 1,527 Nov-08-2019, 02:33 AM
Last Post: Rodrigo
  Locks in Multithreading Chuonon 0 1,836 Oct-03-2019, 04:16 PM
Last Post: Chuonon

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020