Python Forum
Did subprocess.Popen() causes main routine to pause stdout?
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Did subprocess.Popen() causes main routine to pause stdout?
#1
Here is what I wanted to do:
In nix terminal emulator, run my script.
Call subprocess.Popen() to spawn a number of processes, make sure the stdout printouts won't fill my screen.
Once all sub processes have been spawn, I poll them every second to collect return codes. Once all return codes have been collected, I move on to print a summary which process succeeded/failed in the main process. Every second, I print out a progress dot on the screen so I know the main process is running, before printing out the summary.

What I did:
I developed my script in IDLE and ran in IDLE. In IDLE the stdout was not sent to IDLE's Python shell so everything goes as expected, on Raspberry pi.

What happened:
Once I started running the same script in a terminal emulator, the printouts started flooding my terminal. So I added stdout=subprocess.DEVNULL so the printouts are suppressed. BUT, big BUT, at the same time, my main process stdout printing of the progress dots are put on hold, until all processes were over and all progress dots and the summary printed out at once.

I'd like to know how to keep printing progress dots in my main process without being blocked.

The following function has everything I described. The first for-loop spawns all the sub processes. Then the while-loop that follows keeps polling the sub processes and printing out the progress dots.

'''Flash all firmware partitions for a new system'''
def flash_all(paras,flash_all_commands):
    processes=[]
    paras['return_code']=list(paras['name'])
    for Flash_ESP32_command in flash_all_commands:
        # print(Flash_ESP32_command)
        processes.append(subprocess.Popen(shlex.split(Flash_ESP32_command),stdout=subprocess.DEVNULL))
    while(True):
        i=0
        done=True
        for process in processes:
            return_code=process.poll()
            if return_code is None:
                done=False
                print('.',end='')
                break
                print("\n%s has not completed" %(paras['name'][i]))
            else :
                paras['return_code'][i]=return_code
            i=i+1
        if done: break
        time.sleep(1)
    i=0
    print()
    for jig_id in paras['id']:
        print("%s\t: %s" %(jig_id,'Success' if paras['return_code'][i]==0 else 'Failure'))
        i=i+1
In case you want to see some screen shots, mind you the screen is very small!
[Image: f83f2cf4b1c0cb1c78fa885f8268f13b85112dc7.jpeg]
Reply
#2
I guess the buffers are flushed at the end of the for-loop.
You can enforce the flush with the print function.

Instead of calling print('.',end=''), you call print('.',end='', flush=True).
Gribouillis likes this post
Almost dead, but too lazy to die: https://sourceserver.info
All humans together. We don't need politicians!
Reply
#3
That was dead on! Thankd you DeaD_EyE!

So I guess this is some sort of Python internal optimization for speed maybe?
Reply
#4
I think sys.stdout is flushed when a newline is printed, so print('.') flushes stdout but not print('.', end='')
ibreeden likes this post
Reply
#5
(May-04-2021, 08:07 AM)Gribouillis Wrote: I think sys.stdout is flushed when a newline is printed, so print('.') flushes stdout but not print('.', end='')

Thanks but I guess IDLE Python shell handles it differently. In IDLE I didn't have to do anything including the flushing to see the dots and no need to suppress stdout either.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  How can I pause only one object? actualpy 1 322 Feb-01-2024, 07:43 PM
Last Post: deanhystad
  [subprocess] Why stdout sent to stderr? Winfried 3 415 Jan-26-2024, 07:26 PM
Last Post: snippsat
Information subprocess.Popen() suddenly giving me grief? davecotter 3 529 Dec-13-2023, 10:49 PM
Last Post: davecotter
  How to pause execution (on windows)? pstein 1 528 Jun-23-2023, 06:58 PM
Last Post: rob101
  Is it Possible to pause a script? tester_V 6 1,417 Apr-05-2023, 06:11 AM
Last Post: tester_V
  Performance options for sys.stdout.writelines dgrunwal 11 2,992 Aug-23-2022, 10:32 PM
Last Post: Pedroski55
  Use subprocess.Popen and time.sleep chucky831 2 1,890 Aug-11-2022, 07:53 PM
Last Post: carecavoador
  Subprocess.Popen() not working when reading file path from csv file herwin 13 14,615 May-07-2021, 03:26 PM
Last Post: herwin
  disable subprocess.popen prompt echo paul18fr 1 1,966 Feb-04-2021, 02:50 AM
Last Post: Larz60+
  A dynamic link library (DLL) initialization routine failed ish93 0 1,748 Jan-11-2021, 08:22 PM
Last Post: ish93

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020