Python Forum
python multiprocessing import Pool, cpu_count: causes forever loop | help to remove
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
python multiprocessing import Pool, cpu_count: causes forever loop | help to remove
#1
The code using multiprocessing causes a forever loop.

I'm using a building an iris recognition system. this is the matching function. everything works fine until the multiprocessing the part.

I'm attaching screenshot of the error output below so that you get a better idea.


    RuntimeError:
            An attempt has been made to start a new process before the
            current process has finished its bootstrapping phase.
    
            This probably means that you are not using fork to start your
            child processes and you have forgotten to use the proper idiom
            in the main module:
    
                if __name__ == '__main__':
                    freeze_support()
                    ...
    
            The "freeze_support()" line can be omitted if the program
            is not going to be frozen to produce an executable.
Code


    ##-----------------------------------------------------------------------------
    ##  Import
    ##-----------------------------------------------------------------------------
    import numpy as np
    from os import listdir
    from fnmatch import filter
    import scipy.io as sio
    from multiprocessing import Pool, cpu_count
    from itertools import repeat
    
    import warnings
    warnings.filterwarnings("ignore")
    
    
    ##-----------------------------------------------------------------------------
    ##  Function
    ##-----------------------------------------------------------------------------
    def matching(template_extr, mask_extr, temp_dir, threshold=0.38):
        """
        Description:
            Match the extracted template with database.
    
        Input:
            template_extr   - Extracted template.
            mask_extr       - Extracted mask.
            threshold       - Threshold of distance.
            temp_dir        - Directory contains templates.
    
        Output:
            List of strings of matched files, 0 if not, -1 if no registered sample.
        """
        # Get the number of accounts in the database
        n_files = len(filter(listdir(temp_dir), '*.mat'))
        if n_files == 0:
            return -1
    
        # Use all cores to calculate Hamming distances
        args = zip(
            sorted(listdir(temp_dir)),
            repeat(template_extr),
            repeat(mask_extr),
            repeat(temp_dir),
        )
        with Pool(processes=cpu_count()) as pools:
            result_list = pools.starmap(matchingPool, args)
    
        filenames = [result_list[i][0] for i in range(len(result_list))]
        hm_dists = np.array([result_list[i][1] for i in range(len(result_list))])
    
        # Remove NaN elements
        ind_valid = np.where(hm_dists>0)[0]
        hm_dists = hm_dists[ind_valid]
        filenames = [filenames[idx] for idx in ind_valid]
    
        # Threshold and give the result ID
        ind_thres = np.where(hm_dists<=threshold)[0]
    
        # Return
        if len(ind_thres)==0:
            return 0
        else:
            hm_dists = hm_dists[ind_thres]
            filenames = [filenames[idx] for idx in ind_thres]
            ind_sort = np.argsort(hm_dists)
            return [filenames[idx] for idx in ind_sort]
    
    
    #------------------------------------------------------------------------------
    def calHammingDist(template1, mask1, template2, mask2):
        """
        Description:
            Calculate the Hamming distance between two iris templates.
    
        Input:
            template1   - The first template.
            mask1       - The first noise mask.
            template2   - The second template.
            mask2       - The second noise mask.
    
        Output:
            hd          - The Hamming distance as a ratio.
        """
        # Initialize
        hd = np.nan
    
        # Shift template left and right, use the lowest Hamming distance
        for shifts in range(-8,9):
            template1s = shiftbits(template1, shifts)
            mask1s = shiftbits(mask1, shifts)
    
            mask = np.logical_or(mask1s, mask2)
            nummaskbits = np.sum(mask==1)
            totalbits = template1s.size - nummaskbits
    
            C = np.logical_xor(template1s, template2)
            C = np.logical_and(C, np.logical_not(mask))
            bitsdiff = np.sum(C==1)
    
            if totalbits==0:
                hd = np.nan
            else:
                hd1 = bitsdiff / totalbits
                if hd1 < hd or np.isnan(hd):
                    hd = hd1
    
        # Return
        return hd
    
    
    #------------------------------------------------------------------------------
    def shiftbits(template, noshifts):
        """
        Description:
            Shift the bit-wise iris patterns.
    
        Input:
            template    - The template to be shifted.
            noshifts    - The number of shift operators, positive for right
                          direction and negative for left direction.
    
        Output:
            templatenew - The shifted template.
        """
        # Initialize
        templatenew = np.zeros(template.shape)
        width = template.shape[1]
        s = 2 * np.abs(noshifts)
        p = width - s
    
        # Shift
        if noshifts == 0:
            templatenew = template
    
        elif noshifts < 0:
            x = np.arange(p)
            templatenew[:, x] = template[:, s + x]
            x = np.arange(p, width)
            templatenew[:, x] = template[:, x - p]
    
        else:
            x = np.arange(s, width)
            templatenew[:, x] = template[:, x - s]
            x = np.arange(s)
            templatenew[:, x] = template[:, p + x]
    
        # Return
        return templatenew
    
    
    #------------------------------------------------------------------------------
    def matchingPool(file_temp_name, template_extr, mask_extr, temp_dir):
        """
        Description:
            Perform matching session within a Pool of parallel computation
    
        Input:
            file_temp_name  - File name of the examining template
            template_extr   - Extracted template
            mask_extr       - Extracted mask of noise
    
        Output:
            hm_dist         - Hamming distance
        """
        # Load each account
        data_template = sio.loadmat('%s%s'% (temp_dir, file_temp_name))
        template = data_template['template']
        mask = data_template['mask']
    
        # Calculate the Hamming distance
        hm_dist = calHammingDist(template_extr, mask_extr, template, mask)
        return (file_temp_name, hm_dist)
How can I remove multiprocessing and make code still work fine?
screenshots dropbox link
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Is this a multiprocessing bug in Python - or am I doing something wrong? haimat 1 1,128 Oct-18-2023, 06:07 AM
Last Post: absalom1
  python multiprocessing help -- to extract 10 sql table into csv mg24 3 1,313 Nov-20-2022, 11:50 PM
Last Post: mg24
  Multiprocessing Pool Multiple Instances How to Kill by Pool ID sunny9495 0 735 Nov-16-2022, 05:57 AM
Last Post: sunny9495
  python multiprocessing to download sql table mg24 5 1,407 Oct-31-2022, 03:53 PM
Last Post: Larz60+
  PyRun_SimpleFile calling multiprocessing Python Class cause endless init loop Xeno 2 988 Sep-19-2022, 02:32 AM
Last Post: Xeno
  Python multiprocessing Pool apply async wait for process to complete sunny9495 6 6,216 Apr-02-2022, 06:31 AM
Last Post: sunny9495
  generating random string unique forever Skaperen 5 2,286 Aug-20-2021, 07:15 AM
Last Post: Gribouillis
  Pool multiprocessing - know current status in loop? korenron 0 1,604 Jul-28-2021, 08:49 AM
Last Post: korenron
  pool mysql error - not catch by try\except? korenron 1 2,101 Jul-05-2021, 11:26 AM
Last Post: ibreeden
  Python - Import file sequence into Media Pool jensenni 1 2,080 Feb-02-2021, 05:11 PM
Last Post: buran

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020