Python Forum
How to parse a live feed in Python?
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
How to parse a live feed in Python?
#1
I am using the subprocess.run() to run handbrakecli to compress a media folder. Whenever I run the command I get a live data feed every 1/2 second in my output. Is there a way to capture the live stdout to get the ETA, Progress value, etc. to create a progress bar? I know how to pase a string to get that data, its just getting those lines from the live output and getting them fast enough to be useful is the hardest part.

Here's an example of what the live feed looks like when you run the code below.

from time import sleep
for feed in range(0,101,20):
    p = f'''
    Progress: {{
        "State":"WORKING",
        "Working":{{
            "ETASeconds":265,
            "Hours":0,
            "Minutes":4,
            "Pass":1,
            "PassCount":1,
            "PassID":0,
            "Paused":0,
            "Progress":{feed}.00000000000000000,
            "Rate":110.10037994384766,
            "RateAvg":124.60877227783203,
            "Seconds":{feed/10},
            "SequenceID":1
        }}
    }}
    '''
    sleep(.5)
    print(p)
Reply
#2
You're not showing how you're calling subprocess. subprocess.run is the easy way, but waits for the process to exit. If you want to read from stdout before it exits, you'll have to use subprocess.Popen directly.

If I change your program above to have it print with flush=True and call it "sender.py", then the following program will display the progress. You can do what you need with the data.

Note that this one should work okay because you're constantly getting output and you shouldn't need to do anything between the progress updates. The readline() is going to block until you get more data. If you had to do work while waiting for output, you'd need a different method.

import subprocess

cmd = ["python3", "sender.py"]

p = subprocess.Popen(cmd, stdout=subprocess.PIPE, text=True)

while (line := p.stdout.readline()) != "":
    line = line.rstrip()
    if '"Progress"' in line:
        progress = float(line.split(":")[1].rstrip(","))
        print(f"Progress: {progress:.2f}")

print(f"End of output.  Return code: {p.wait()}")
Output:
Progress: 0.00 Progress: 20.00 Progress: 40.00 Progress: 60.00 Progress: 80.00 Progress: 100.00 End of output. Return code: 0
Reply
#3
(Jan-18-2022, 05:26 AM)bowlofred Wrote: You're not showing how you're calling subprocess. subprocess.run is the easy way, but waits for the process to exit. If you want to read from stdout before it exits, you'll have to use subprocess.Popen directly.

If I change your program above to have it print with flush=True and call it "sender.py", then the following program will display the progress. You can do what you need with the data.

Note that this one should work okay because you're constantly getting output and you shouldn't need to do anything between the progress updates. The readline() is going to block until you get more data. If you had to do work while waiting for output, you'd need a different method.

import subprocess

cmd = ["python3", "sender.py"]

p = subprocess.Popen(cmd, stdout=subprocess.PIPE, text=True)

while (line := p.stdout.readline()) != "":
    line = line.rstrip()
    if '"Progress"' in line:
        progress = float(line.split(":")[1].rstrip(","))
        print(f"Progress: {progress:.2f}")

print(f"End of output.  Return code: {p.wait()}")
Output:
Progress: 0.00 Progress: 20.00 Progress: 40.00 Progress: 60.00 Progress: 80.00 Progress: 100.00 End of output. Return code: 0

That will work perfectly! Thank you for the help!

I hadn't responded for a while since I'm a bit of an issue with my Python interpreters, but I'll fix that later.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  How to parse and group hierarchical list items from an unindented string in Python? ann23fr 0 99 Mar-27-2024, 01:16 PM
Last Post: ann23fr
  Draw bounding boxes on live feed Jerome 0 228 Jan-20-2024, 10:50 PM
Last Post: Jerome
  Is it possible to capture live running waveforms in an oscilloscope using python? Girishbabu_27 4 924 Jul-29-2023, 06:05 AM
Last Post: Girishbabu_27
  [split] Parse Nested JSON String in Python mmm07 4 1,429 Mar-28-2023, 06:07 PM
Last Post: snippsat
  python read iperf log and parse throughput jacklee26 4 2,649 Aug-27-2022, 07:04 AM
Last Post: Yoriz
  Feed List items with Integer euras 9 3,851 May-19-2021, 07:45 PM
Last Post: snippsat
  splitting UAV/sat images to smaller pieces in order to feed a CNN hobbyist 0 1,505 Dec-08-2020, 11:48 AM
Last Post: hobbyist
  Rotation Effect on live Webcam Feed Leziiy 0 1,576 Sep-12-2020, 04:25 PM
Last Post: Leziiy
  Replace every 20th space with line feed T_Lafferty 3 2,331 Jul-19-2020, 10:37 PM
Last Post: T_Lafferty
  Parse a REST API call using Python GKT 1 1,874 May-07-2020, 04:15 AM
Last Post: buran

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020