UnicodeEncodeError - Dealing with Japanese Characters

UnicodeEncodeError - Dealing with Japanese Characters - Printable Version

+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: General Coding Help (https://python-forum.io/forum-8.html)
+--- Thread: UnicodeEncodeError - Dealing with Japanese Characters (/thread-37668.html)

UnicodeEncodeError - Dealing with Japanese Characters - fioranosnake - Jul-07-2022

Hi All,

Ive got a small Python app that sends a bunch of codes to a checking API and returns the results.

Im getting errors when the the app tries to right non-english characters to a "results.csv" resulting in the following logging error:

File "C:\Users\PyUser\.conda\envs\testenv\lib\encodings\cp1252.py", line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode characters in position 49-59: character maps to <undefined>

The code that writes the results is

logging.info("create output")
    with open(pathlib.Path(f'{RESULTS_PATH}/api_results.csv'), 'w', newline='') as f:
        fieldnames = list(response_data["TestResults"].values())[0].keys()
        wrtr = csv.DictWriter(f, fieldnames=fieldnames)
        wrtr.writeheader()
        wrtr.writerows(response_data["TestResults"].values())

When running the codes through postman the resulting JSON is :

"82": {
"PASS-FAIL": "PASS",
"IDENTITY": "ニコン（日本）",
"ORIGIN": "Japan"
},
"83": {
"PASS-FAIL": "PASS",
"IDENTITY": "MICROSOFT",
"ORIGIN": "USA"
},

Any advice greatly appreciated

Thanks so much!

RE: UnicodeEncodeError - Dealing with Japanese Characters - snippsat - Jul-07-2022

(Jul-07-2022, 04:30 PM)fioranosnake Wrote: UnicodeEncodeError: 'charmap' codec can't encode

Windows choose wrong encoding,force it use utf-8.

with open(pathlib.Path(f'{RESULTS_PATH}/api_results.csv'), 'w', newline='', encoding='utf-8') as f:

RE: UnicodeEncodeError - Dealing with Japanese Characters - fioranosnake - Jul-07-2022

Perfect - working now!

Very Very much appreciated