subprocess.Popen() and encodings

subprocess.Popen() and encodings - Printable Version

+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: General Coding Help (https://python-forum.io/forum-8.html)
+--- Thread: subprocess.Popen() and encodings (/thread-24580.html)

subprocess.Popen() and encodings - voltron - Feb-20-2020

I need to call 3rd party command-line tool from Python and communicate with it: pass commands and read their results. Tool started with the subprocess.Popen() and then I write to stdin and read from stdout. Here is simplified code

import subprocess

command = ['/path/to/executable', 'arg1', 'arg2', 'arg3']

instance = subprocess.Popen(command,
                            stdin=subprocess.PIPE,
                            stdout=subprocess.PIPE,
                            stderr=subprocess.DEVNULL,
                            universal_newlines=True)

# pass data and command to the tool
instance.stdin.write('/path/to/data\n-command\n')
instance.stdin.flush()

# get results
instance.stdout.readlines()

This works fine on Linux and Mac, but when I try to run it inside Windows command-line (cmd.exe) with file name containing non-ASCII characters, I get

Quote:UnicodeEncodeError: 'charmap' codec can't encode characters in position 53-59: character maps to <undefined>

As I understand this is because universal_newlines=True activates text mode and all input and output is decoded into some encoding (UTF-8 by default) while another encoding is used (e.g. CP1251 or other depending on the system locale). I tried to pass encoding='cp1251' but this does not help, filename turned into question marks and file can not be by commandline tool.

If I remove universal_newlines=True and pass all parameters as bytes objects encoded using UTF-8 encoding, it still does not work. Non-ASCII filenames encoded using \xNN escape sequences and can not be found by commandline tool.

I'm a bit lost here and probably missing something obvious. Is there a cross-paltform way to pass and read non-ASCII strings using subprocess.Popen() which will work for any locale?

I'm using Python 3.7.0 and Python 3.8.0.