Multiprocessing.Queue Issues (Missing/Corrupted Items/No Output)

[python3noob]

Hardware: Raspberry Pi 4B (1GB) & Macbook Pro

OS: Raspbian Buster & OSX 10.14.5

Python Version: 3.7.3

I'm having issues with multiprocessing.Queue(), and I think they are related, but I don't know how to resolve it.

Issue 1: multiprocessing.Queue() is skipping the first item that is placed into the queue. After some testing I have figured out that I can keep this from happening if I add additional code (time.sleep(.0001), print(''), anything except for commented code) between the subsequent q.put() commands. q.get will always skip the first item in the queue and start on the second item without a delay (see "additional code" mentioned above) between adding items to the queue, and when a delay is added, it will always get the first item in the queue. I trimmed down the code to the important part to show an example, but now it is presenting as a primarily as a different issue (see "Issue 2a" and "Issue 2b"), but also shows slightly different (see "Issue 2c"). Here is a small output example from the actual code where I first realized the issue.

Loading /home/pi/xxx/xxx/xxx/xxx/0004.xxx
Loading /home/pi/xxx/xxx/xxx/xxx/0005.xxx
Loading /home/pi/xxx/xxx/xxx/xxx/0006.xxx

The first files that were queued were 0001, 0002, and 0003.


Issue 2: This example code that I'm posting below gives me three different issues which I believe is related to issue one, just presenting itself differently. This code will sometimes display different issues depending on a couple variables (see below).
Issue 2a: No output at all. This is primarily seen on my Macbook Pro, but also seen on the Raspberry Pi with the code exactly as shown.
Issue 2b: Partial output. Stops at different points before reaching the end. Note: Output should continue all the way to and end with Entry: 33 Data 99. Most commonly seen on the Raspberry Pi, but also happens occasionally on the Macbook Pro. Happens with slight changes to the code such as replacing data = [data_list, set_size, entry] with data = [data_list.copy(), set_size, entry] or adding some kind of print statement to the spawned process such as print('got from queue', data) directly following data = q.get(.

Entry: 1 Data: 1
Entry: 1 Data: 2
Entry: 1 Data: 3
Entry: 2 Data: 4
Entry: 2 Data: 5
Entry: 2 Data: 6
Entry: 3 Data: 7
Entry: 3 Data: 8
Entry: 3 Data: 9
Entry: 4 Data: 10
Entry: 4 Data: 11
Entry: 4 Data: 12
Entry: 5 Data: 13
Entry: 5 Data: 14
Entry: 5 Data: 15
Entry: 6 Data: 16
Entry: 6 Data: 17
Entry: 6 Data: 18

Issue 2c: Output corrupted/out of order. Happens most often on the Raspberry Pi, but have been able to reproduce on the Macbook Pro. An examples of the full output is below.

Issue 2d: No output. Only happens on the Macbook Pro, always with the sample code as shown below, sometimes with additional code (time.sleep or print) added between each q.put() or after q.get().

Entry: 1 Data: 16
Entry: 1 Data: 17
Entry: 1 Data: 18

Entry: 1 should contain Data 1-3, not 16-18.


On to the test code:

import multiprocessing
import time

set_size = 3

def process_queueing():
    entry = 1
    data_list = []
    for i in range(1,100):
        data_list.append(i)
        if i % set_size == 0:
            data = [data_list, set_size, entry]
            q.put(data)
            #time.sleep(.001) #Uncomment to fix problem
            entry = entry + 1
            data_list.clear()

def process_data():
    while True:
        data = q.get()
        for i in data[0]:
            print('Entry: ' + str(data[2]) + ' Data: ' + str(i))
            

q = multiprocessing.Queue()
process = multiprocessing.Process(target=process_data, daemon=True)
process.start()

process_queueing()

Simply adding a very short wait or printing something between each q.put() will make everything start working, and even adding something as simple as print('got from queue', data) after data = q.get() in the spawned process will change the results (often resulting in partial output ("Issue 2b") or no output "issue 2d"). The addition of simple lines of code like that which change the output have made it hard for me to track down this problem.

I've also tried replacing data = [data_list, set_size, entry] with data = [data_list.copy(), set_size, entry], but that has not resolved the issue.

Any help in figuring out a proper fix to this issue and/or why this is happening would be great!

Reddit thread and Stackoverflow topic on the issue if that helps.