argparse io issue

[pyDream]

Hi all, I'm quite new to Python, so please forgive me if this issue can be fixed with the debugger. I don't know how to debug packages installed from remote repositories. If you could point me to a tutorial on how to do that, it would be nice, but that is besides the main question.

I'm trying to use the package lingvoreader, to work on lingvo dictionaries. For that I've installed the said package, and running the main module of it from the command line. But whatever arguments do I provide to the program, it fails with an error that seems to me like it's caused by I/O issues.
To test that the issue is caused by I/O I've installed the same package on another environment. For reference currently the main environment that I use for my experiments is Windows 10, that runs Python 3.12.2. But I've installed another OS on the VirtualBox, that would be Ubuntu 23.10.1-desktop-amd64, with Python 3.11.6, and within that environment the package runs perfectly fine.

To reproduce the issue, you first would need to install the package lingvoreader.

Here's the commands that I've used to install it:

Output:
pip install setuptools -U
pip install lingvoreader

Then you can run the main module with any arguments whatsoever, it doesn't matter as long as you use correct syntax. The error is going to be reproduced either way as long as you run the program on Windows. As I've already tried to reproduce the described steps on a fresh install of a virtualized Windows machine.
Here's an example of what you can run:

Output:
lsdreader --help

There's really no need to provide the program with any dictionaries to work with to reproduce the issue, because the error occurs when the program is trying to print text to console.

Here's the full output:

Output:
E:\dict\X5>lsdreader --help
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\LENOVO\AppData\Local\Programs\Python\Python312\Scripts\lsdreader.exe\__main__.py", line 7, in <module>
  File "C:\Users\LENOVO\AppData\Local\Programs\Python\Python312\Lib\site-packages\lingvoreader\lsdreader.py", line 128, in main
    args = get_arg_parser().parse_args()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\LENOVO\AppData\Local\Programs\Python\Python312\Lib\argparse.py", line 1891, in parse_args
    args, argv = self.parse_known_args(args, namespace)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\LENOVO\AppData\Local\Programs\Python\Python312\Lib\argparse.py", line 1924, in parse_known_args
    namespace, args = self._parse_known_args(args, namespace)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\LENOVO\AppData\Local\Programs\Python\Python312\Lib\argparse.py", line 2136, in _parse_known_args
    start_index = consume_optional(start_index)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\LENOVO\AppData\Local\Programs\Python\Python312\Lib\argparse.py", line 2076, in consume_optional
    take_action(action, args, option_string)
  File "C:\Users\LENOVO\AppData\Local\Programs\Python\Python312\Lib\argparse.py", line 2000, in take_action
    action(self, namespace, argument_values, option_string)
  File "C:\Users\LENOVO\AppData\Local\Programs\Python\Python312\Lib\argparse.py", line 1141, in __call__
    parser.print_help()
  File "C:\Users\LENOVO\AppData\Local\Programs\Python\Python312\Lib\argparse.py", line 2626, in print_help
    self._print_message(self.format_help(), file)
  File "C:\Users\LENOVO\AppData\Local\Programs\Python\Python312\Lib\argparse.py", line 2632, in _print_message
    file.write(message)
  File "<frozen codecs>", line 378, in write
TypeError: write() argument must be str, not bytes

And for comparison, here's what the output looks like on Ubuntu VM:

Output:
vmadmin@vmadmin-1-2:~$ ~/.local/bin/python -m lingvoreader.lsdreader --help
usage: lsdreader.py [-h] (-i INPUT | -a) [--header] [-o OUTDIR] [-c] [-v]
                    [--version]

Decode Lingvo lsd dictionary to dsl

options:
  -h, --help            show this help message and exit
  -i INPUT, --input INPUT
                        Dictionary to decode
  -a, --all             All dictionary in current directory
  --header              Print dictionary header and exit
  -o OUTDIR, --outdir OUTDIR
                        Output directory
  -c, --codecs          print supported languages and their codes
  -v, --verbose
  --version             show program's version number and exit

Lastly, I will provide you with the steps that I took to install the same package on Ubuntu:

Output:
sudo apt install python3.11-venv

python3 -m venv ~/.local --system-site-packages

~/.local/bin/pip install setuptools -U

 ~/.local/bin/pip install lingvoreader

To recap, the question is, why does this issue occur on Windows, and not on Linux, and how do I fix it?

[snippsat]

The lsdreader dos not work later version of Python.
It's made a long time ago and the Python 3 compatibility they added is not testet good enough on Windows.
To make it work download Python 3.9(do not add to Path),go to Scripts folder and do pip install lingvoreader
Open C:\Python39\Lib\codecs.py line 378 change to this.

self.stream.write(data.decode())  # Add .decode()

Teste that it works.

C:\Python39\Scripts
λ lsdreader --help
usage: lsdreader [-h] (-i INPUT | -a) [--header] [-o OUTDIR] [-c] [-v] [--version]

Decode Lingvo lsd dictionary to dsl

optional arguments:
  -h, --help            show this help message and exit
  -i INPUT, --input INPUT
                        Dictionary to decode
  -a, --all             All dictionary in current directory
  --header              Print dictionary header and exit
  -o OUTDIR, --outdir OUTDIR
                        Output directory
  -c, --codecs          print supported languages and their codes
  -v, --verbose
  --version             show program's version number and exit

Can not use newer version because they have frozen codecs.py into Python(python.exe).

[pyDream]

The previous commenter had helped me.

I've created another VM to test this and installed the python 3.8.10 just to be sure the version was low enough. I also did include python to path, but that didn't create any problems. Installed the package in question, and then updated the codecs.py script. After doing these steps, everything worked correctly.

But I'm still curious how a novice like me could debug this issue on their own. Could I have used a debugger to step through the library function? Or what other way could be the easiest for this task?

[snippsat]

But I'm still curious how a novice like me could debug this issue on their own. Could I have used a debugger to step through the library function? Or what other way could be the easiest for this task?

This case is not so easy as need to now what <frozen codecs> mean and history why codecs was used in Python 2.

If i take a look at code for lsdreader code i can make a change so it work for eg for Python 3.12.
This mean change in setup.py and build a new wheel.
So codecs.open is is geared towards Python 2,in Python 3 the default encoding for open is UTF-8, and the handling of encodings is generally more robust.
Can write read function like this.

def read(*parts, encoding='utf-8'):
    here = os.path.abspath(os.path.dirname(__file__))
    with open(os.path.join(here, *parts), 'r', encoding=encoding) as f:
        return f.read()

After build and pip install of wheel,now it work in Python 3.12.

 G:\div_code\reader_env
(reader_env) λ python --version
Python 3.12.0

G:\div_code\reader_env
(reader_env) λ lsdreader --help
usage: lsdreader [-h] (-i INPUT | -a) [--header] [-o OUTDIR] [-c] [-v] [--version]

Decode Lingvo lsd dictionary to dsl

options:
  -h, --help            show this help message and exit
  -i INPUT, --input INPUT
                        Dictionary to decode
  -a, --all             All dictionary in current directory
  --header              Print dictionary header and exit
  -o OUTDIR, --outdir OUTDIR
                        Output directory
  -c, --codecs          print supported languages and their codes
  -v, --verbose
  --version             show program's version number and exit

If you want to try this yourself i can post step on how to do this.

[pyDream]

Thanks for the offer, but the information that you've provided was enough for me to fix the issue.

I'm mainly tinkering with this package to get better at Python.
So I'm really curious on how you understood what the

<frozen codecs>

means. In my understanding this refers to the feature in Python to 'freeze' a package for easy distribution, but I've updated the code on my local machine, not in the frozen package as I understand, why is that?

And why the read() function has anything to do with the program printing to console, as the function works on reading text files, not writing to console?

[snippsat]

So I'm really curious on how you understood what the

<frozen codecs>

In newer version of Python the have frozen codecs.py into python.exe then can not change codec.py(as it will have no effect).

And why the read() function has anything to do with the program printing to console, as the function works on reading text files, not writing to console?

read() is what make the error as i written with codec,so i rewrite to just use Python 3.
As read() is in setup.py then need to rebuild the module(also make a new wheel)
So this is the real fix that author could do to make it work for all Python version in Windows.

[pyDream]

I've misunderstood what that line did that I've modified, but I think I got it now.

def read(*parts, encoding='utf-8'):
    here = os.path.abspath(os.path.dirname(__file__))
    with open(os.path.join(here, *parts), 'r', encoding=encoding) as f:
        return f.read()

So this part not only defines how the script would read a dictionary, but also how it will write to the console, correct?
Just to be clear.

[snippsat]

So this part not only defines how the script would read a dictionary, but also how it will write to the console, correct?
Just to be clear.

No,the error happens before all the reading and use of argparse.
The problem happens in setup.py where read function is reading files like,
README.rst or __init__.py that contain metadata or other information needed in the setup script.
This can be hard to understand if you have not deal with building a package in Python.

If you want try as a excerice,here is step how it did rebuild this so it work in eg Python 3.12

# Make Virtual enviroemnt 
G:\div_code
λ python -m venv lsdreader_env

G:\div_code
λ cd lsdreader_env\

G:\div_code\lsdreader_env
λ G:\div_code\lsdreader_env\Scripts\activate

# Install build tools
G:\div_code\lsdreader_env
(lsdreader_env) λ pip install setuptools wheel
.....
Installing collected packages: wheel, setuptools
Successfully installed setuptools-69.2.0 wheel-0.43.0

# Clone down Repo
G:\div_code\lsdreader_env
(lsdreader_env) λ git clone https://github.com/sv99/lsdreader.git
Cloning into 'lsdreader'...
remote: Enumerating objects: 153, done.
remote: Counting objects: 100% (17/17), done.
remote: Compressing objects: 100% (16/16), done.
Receiving objects:  88% (135/153)
Receiving objects: 100% (153/153), 1.05 MiB | 3.56 MiB/s, done.
Resolving deltas: 100% (77/77), done.

G:\div_code\lsdreader_env
(lsdreader_env) λ cd lsdreader\

# Make the changes to <read> in setup.py
G:\div_code\lsdreader_env\lsdreader (master -> origin)
(lsdreader_env) λ ls
decoder.rar  LICENSE.md  lingvoreader/  README.rst  setup.py*  test/  testdata/

# Make the wheel
G:\div_code\lsdreader_env\lsdreader (master -> origin)
(lsdreader_env) λ python setup.py bdist_wheel
running bdist_wheel
running build
....

# Cd to dist folder
G:\div_code\lsdreader_env\lsdreader (master -> origin)
(lsdreader_env) λ cd dist

G:\div_code\lsdreader_env\lsdreader\dist (master -> origin)
(lsdreader_env) λ ls
lingvoreader-0.2.15-py3-none-any.whl

# Install wheel
G:\div_code\lsdreader_env\lsdreader\dist (master -> origin)
(lsdreader_env) λ pip install lingvoreader-0.2.15-py3-none-any.whl
Processing g:\div_code\lsdreader_env\lsdreader\dist\lingvoreader-0.2.15-py3-none-any.whl
Installing collected packages: lingvoreader
Successfully installed lingvoreader-0.2.15

# Test that it work
G:\div_code\lsdreader_env
(lsdreader_env) λ lsdreader --help
usage: lsdreader [-h] (-i INPUT | -a) [--header] [-o OUTDIR] [-c] [-v] [--version]

Decode Lingvo lsd dictionary to dsl

options:
  -h, --help            show this help message and exit
  -i INPUT, --input INPUT
                        Dictionary to decode
  -a, --all             All dictionary in current directory
  --header              Print dictionary header and exit
  -o OUTDIR, --outdir OUTDIR
                        Output directory
  -c, --codecs          print supported languages and their codes
  -v, --verbose
  --version             show program's version number and exit

[pyDream]

I'm actually very close to that part of documentation where they explain how the packaging works. But you're right, for now I don't know how packaging works well enough to fully understand your explanation. But you've managed to break down the issue the best you could to a newcomer like me.

As for the exercise, I'm not ready to handle it for now, but I will definitely be returning to it later.

I think I'll be returning with a report on my progress with this exercise. I just don't know if I should post another thread for that.