Python Forum
no coding= option fo subprocess.Popen() - Printable Version

+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: General Coding Help (https://python-forum.io/forum-8.html)
+--- Thread: no coding= option fo subprocess.Popen() (/thread-19291.html)

no coding= option fo subprocess.Popen() - Skaperen - Jun-21-2019

i want to have utf8 decoding on the stdout pipe with subprocess.Popen() so when it is read i get str instead of bytes, but i see no option to set that. any suggestions?

RE: no coding= option fo subprocess.Popen() - snippsat - Jun-21-2019

Have you looked at doc?
See parameter encoding=None setting to encoding='utf-8' stdout will by string(Unicode default) Python 3.

Work in run() to which should be used in all cases it can handle.
subprocess.run() Wrote:The recommended approach to invoking subprocesses is to use the run() function for all use cases it can handle.
For more advanced use cases, the underlying Popen interface can be used directly.
import subprocess
 
output = subprocess.run(['ping', '-n', '4', 'google.com'], encoding='utf-8', capture_output=True)
print(output.stdout)
Output:
Pinging google.com [2a00:1450:400f:80b::200e] with 32 bytes of data:
Reply from 2a00:1450:400f:80b::200e: time=42ms 
Reply from 2a00:1450:400f:80b::200e: time=33ms 
Reply from 2a00:1450:400f:80b::200e: time=46ms 
Reply from 2a00:1450:400f:80b::200e: time=50ms 

Ping statistics for 2a00:1450:400f:80b::200e:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 33ms, Maximum = 50ms, Average = 42ms


RE: no coding= option fo subprocess.Popen() - Skaperen - Jun-21-2019

i need to read the data as it arrives, hence the need for the pipe. it had previously written the data to a temporary file and read the file and that got str from read, but now i need to read from a pipe, and now i get bytes. i will look at another possible design, but i do want to solve this with Popen(). on your ping command test, try doing it without the option -n 4 on the command and get your pings out as they arrive.
i just did try run() and it looks like (from the trace dump) that it calls Popen() itself, passing all the keyword options. my documentation does not show an encoding= option for either run() or Popen().

Output:
Traceback (most recent call last):
  File "test1.py", line 10, in <module>
    out = run(['aws','s3','ls'],encoding='utf-8',capture_output=True)
  File "/usr/lib/python3.5/subprocess.py", line 693, in run
    with Popen(*popenargs, **kwargs) as process:
TypeError: __init__() got an unexpected keyword argument 'encoding'
from the source code:
Output:
lt2a/forums /home/forums 8> cat -n /usr/lib/python3.5/subprocess.py|lines 830 874
   830	class Popen(object):
   831	
   832	    _child_created = False  # Set here since __del__ checks it
   833	
   834	    def __init__(self, args, bufsize=-1, executable=None,
   835	                 stdin=None, stdout=None, stderr=None,
   836	                 preexec_fn=None, close_fds=_PLATFORM_DEFAULT_CLOSE_FDS,
   837	                 shell=False, cwd=None, env=None, universal_newlines=False,
   838	                 startupinfo=None, creationflags=0,
   839	                 restore_signals=True, start_new_session=False,
   840	                 pass_fds=()):
   841	        """Create new Popen instance."""
   842	        _cleanup()
   843	        # Held while anything is calling waitpid before returncode has been
   844	        # updated to prevent clobbering returncode if wait() or poll() are
   845	        # called from multiple threads at once.  After acquiring the lock,
   846	        # code must re-check self.returncode to see if another thread just
   847	        # finished a waitpid() call.
   848	        self._waitpid_lock = threading.Lock()
   849	
   850	        self._input = None
   851	        self._communication_started = False
   852	        if bufsize is None:
   853	            bufsize = -1  # Restore default
   854	        if not isinstance(bufsize, int):
   855	            raise TypeError("bufsize must be an integer")
   856	
   857	        if _mswindows:
   858	            if preexec_fn is not None:
   859	                raise ValueError("preexec_fn is not supported on Windows "
   860	                                 "platforms")
   861	            any_stdio_set = (stdin is not None or stdout is not None or
   862	                             stderr is not None)
   863	            if close_fds is _PLATFORM_DEFAULT_CLOSE_FDS:
   864	                if any_stdio_set:
   865	                    close_fds = False
   866	                else:
   867	                    close_fds = True
   868	            elif close_fds and any_stdio_set:
   869	                raise ValueError(
   870	                        "close_fds is not supported on Windows platforms"
   871	                        " if you redirect stdin/stdout/stderr")
   872	        else:
   873	            # POSIX
lt2a/forums /home/forums 9>


RE: no coding= option fo subprocess.Popen() - snippsat - Jun-21-2019

(Jun-21-2019, 08:07 PM)Skaperen Wrote: my documentation does not show an encoding= option for either run() or Popen().
Upgrade your Python version,what show here don't work for Python 3.5.
In Python 3.5 has to use decode() on stdout output.

RE: no coding= option fo subprocess.Popen() - Skaperen - Jun-21-2019

i found that run() can capture stdout when given stdout=PIPE but that capture is bytes and i still need str.
i'm still trying to find a way to upgrade it. pyenv doesn't do it.
a function to convert bytes to str would solve this. oh ... bytes.decode().

RE: no coding= option fo subprocess.Popen() - Skaperen - Jun-21-2019

i got a new design working that could handle the output all at once.

another coming program will need to do that ... it can handle all the output all at once ... from 16 concurrent processes. it needs to make sure the outputs don't get mixed up, although it can take the 16 outputs in any order.