os.path.exists fails with accented characters

[glenndrives]

Looking at this further you have hit on something I was wondering about.

I used command completion in the shell to get the location of the file and took a look at the path returned with Hexedit. The OS is representing the characters as:

U+00F5 õ 0xc3 0xb5 LATIN SMALL LETTER O WITH TILDE

Python is representing the characters as:

U+006F o 0x6f LATIN SMALL LETTER O, U+0303 ̃ 0xcc 0x83 COMBINING TILDE

Right now I am trying to figure out how to either get Linux Mint to accept the Python encoding or to get Python to output what Linux Mint is expecting.

I am using the Bash shell. I get the same results with the Fish shell.

Cheers!

It could be a problem with wrong encoding of the file name itself.
I had often issues with Chinese songs which caused a UnicodeEncodeError.

for p in Path().rglob("*"):
    print(p)

Error:
UnicodeEncodeError: 'utf-8' codec can't encode character '\udcfc' in position 118: surrogates not allowed

To get a list of bad paths, then run in your home directory following code:

import time
from pathlib import Path

files_with_bad_encoding = []

for p in Path().rglob("*"):
    try:
        p.name.encode()
    except UnicodeEncodeError:
        files_with_bad_encoding.append(p)


print(f"Found {len(files_with_bad_encoding)} paths with wrong encoding")
time.sleep(2)

print("Listing affected files")
for path in files_with_bad_encoding:
    print(path.parts)

You can't print the Path or the affected part of the path directly, because it raises the UnicodeEncodeError again. Instead, the code prints the parts of the path, which is a tuple. This shows its representation of parts and is the cause of not raising the Exception again.

This kind of error is the most annoying in the whole Python Eco system!
I don't understand, why this issue is not fixed.

[glenndrives]

It looks like I have solved my problem.

Using the

unicodedata

library with

unicodedata.normalize

I was able to get Python3 to output the Normal Form C version of the UTF-8 characters that the underlying OS was wanting.

From https://docs.python.org/3/library/unicod....normalize:

The Unicode standard defines various normalization forms of a Unicode string, based on the definition of canonical equivalence and compatibility equivalence. In Unicode, several characters can be expressed in various way. For example, the character U+00C7 (LATIN CAPITAL LETTER C WITH CEDILLA) can also be expressed as the sequence U+0043 (LATIN CAPITAL LETTER C) U+0327 (COMBINING CEDILLA).

For each character, there are two normal forms: normal form C and normal form D. Normal form D (NFD) is also known as canonical decomposition, and translates each character into its decomposed form. Normal form C (NFC) first applies a canonical decomposition, then composes pre-combined characters again.

Changing line

commandLineOptions["path"] = finalPath

to

commandLineOptions["path"] = unicodedata.normalize('NFC', finalPath)

resulted in

Output:
(Path Exists): /var/altmusic/Snõõper/Super Snõõper/12  TownTopic_TMM_M1_D_16_44.1.mp3

Thank you all for your help.

Cheers!
Glenn