Did subprocess.Popen() causes main routine to pause stdout?

Did subprocess.Popen() causes main routine to pause stdout? - Printable Version

+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: General Coding Help (https://python-forum.io/forum-8.html)
+--- Thread: Did subprocess.Popen() causes main routine to pause stdout? (/thread-33522.html)

Did subprocess.Popen() causes main routine to pause stdout? - liudr - May-01-2021

Here is what I wanted to do:
In nix terminal emulator, run my script.
Call subprocess.Popen() to spawn a number of processes, make sure the stdout printouts won't fill my screen.
Once all sub processes have been spawn, I poll them every second to collect return codes. Once all return codes have been collected, I move on to print a summary which process succeeded/failed in the main process. Every second, I print out a progress dot on the screen so I know the main process is running, before printing out the summary.

What I did:
I developed my script in IDLE and ran in IDLE. In IDLE the stdout was not sent to IDLE's Python shell so everything goes as expected, on Raspberry pi.

What happened:
Once I started running the same script in a terminal emulator, the printouts started flooding my terminal. So I added stdout=subprocess.DEVNULL so the printouts are suppressed. BUT, big BUT, at the same time, my main process stdout printing of the progress dots are put on hold, until all processes were over and all progress dots and the summary printed out at once.

I'd like to know how to keep printing progress dots in my main process without being blocked.

The following function has everything I described. The first for-loop spawns all the sub processes. Then the while-loop that follows keeps polling the sub processes and printing out the progress dots.

'''Flash all firmware partitions for a new system'''
def flash_all(paras,flash_all_commands):
    processes=[]
    paras['return_code']=list(paras['name'])
    for Flash_ESP32_command in flash_all_commands:
        # print(Flash_ESP32_command)
        processes.append(subprocess.Popen(shlex.split(Flash_ESP32_command),stdout=subprocess.DEVNULL))
    while(True):
        i=0
        done=True
        for process in processes:
            return_code=process.poll()
            if return_code is None:
                done=False
                print('.',end='')
                break
                print("\n%s has not completed" %(paras['name'][i]))
            else :
                paras['return_code'][i]=return_code
            i=i+1
        if done: break
        time.sleep(1)
    i=0
    print()
    for jig_id in paras['id']:
        print("%s\t: %s" %(jig_id,'Success' if paras['return_code'][i]==0 else 'Failure'))
        i=i+1

In case you want to see some screen shots, mind you the screen is very small!
[Image: f83f2cf4b1c0cb1c78fa885f8268f13b85112dc7.jpeg]

RE: Did subprocess.Popen() causes main routine to pause stdout? - DeaD_EyE - May-03-2021

I guess the buffers are flushed at the end of the for-loop.
You can enforce the flush with the print function.

Instead of calling print('.',end=''), you call print('.',end='', flush=True).

RE: Did subprocess.Popen() causes main routine to pause stdout? - liudr - May-04-2021

That was dead on! Thankd you DeaD_EyE!

So I guess this is some sort of Python internal optimization for speed maybe?

RE: Did subprocess.Popen() causes main routine to pause stdout? - Gribouillis - May-04-2021

I think sys.stdout is flushed when a newline is printed, so print('.') flushes stdout but not print('.', end='')

RE: Did subprocess.Popen() causes main routine to pause stdout? - liudr - May-04-2021

(May-04-2021, 08:07 AM)Gribouillis Wrote: I think sys.stdout is flushed when a newline is printed, so print('.') flushes stdout but not print('.', end='')

Thanks but I guess IDLE Python shell handles it differently. In IDLE I didn't have to do anything including the flushing to see the dots and no need to suppress stdout either.