Python Forum

I have the following function that builds a file list.

    import os
    import fnmatch
    import threading

    def getFiles(root='.', pattern='*.*', recurse=False):

        results = []
        if recurse:
            for base, dirs, files in os.walk(root):
                match = fnmatch.filter(files, pattern)
                results.extend(os.path.join(base, f) for f in match)
        else:
            results = fnmatch.filter(os.listdir(root),pattern)
            for i in range(0, len(results)): results[i] = os.path.join(root,results[i])

        return results

I want to run this function in a thread so that it can build the file list while the script is busy doing other things. How can I get the returned list from the main script? Here is a stripped down version.

    #Create the thread and start parallel execution

    t = threading.Thread(target=getFiles, args=['.', '*', False], daemon=False)
    t.start()

    #Here's where I do the "other things"

    # Done other things. Wait for the file list thread to complete

    t.join()

How do I get the returned list? I have found several examples but none that are not obfuscated by irrelevant bells and whistles.

With a little more effort I found the solution although it didn't involve adding to my existing code. The full solution is below

import os
import fnmatch
from   multiprocessing.pool import ThreadPool

def getFiles(root='.', pattern='*.*', recurse=False):
    """
    Return a list of all files in the given root that match the
    given pattern. If recurse=True the also do all sub-folders.
    """
    results = []
    
    if recurse:
        for base, dirs, files in os.walk(root):
            match = fnmatch.filter(files, pattern)
            results.extend(os.path.join(base, f) for f in match)
    else:
        results = fnmatch.filter(os.listdir(root),pattern)
        for i in range(0, len(results)): results[i] = os.path.join(root,results[i])

    return results
             
#Create the thread and start parallel execution

pool = ThreadPool(processes=1)
thread = pool.apply_async(getFiles, ('.','*.*',False))

#The code below this will run in parallel with the above thread

print("continue on with other code in parallel")

#This will pause until the thread completes.

files = thread.get()

for file in files:
    print(file)

Key here is that you have the wrong idea. You do not "return" a value from a thread. The thread runs, and eventually, terminates. There is no concept of "returning" anything.

However, you can ideally send a message between two threads, or have a shared queue into which you place results. Ultimately, the "main thread" can decide to examine this queue and find whatever elements you have put in it. I am not familiar enough with Python to say how this is done, but this is how I do it in other languages. Threads never "return" values because there is no place that they can return them to. But you can use shared resources, which you have to make sure are properly locked and synchronized, to pass information between threads. In some systems and libraries these capabilities are built in. I have not tried multithreading in Python, although I have depended on it for dozens of projects done in C/C++ and other languages (I've been using multithreading since about 1969).

If that is the case then how do you explain that the line "files = thread.get()" gets me the value that is returned by the threaded function?