UnicodeEncodeError - Dealing with Japanese Characters

[fioranosnake]

Hi All,

Ive got a small Python app that sends a bunch of codes to a checking API and returns the results.

Im getting errors when the the app tries to right non-english characters to a "results.csv" resulting in the following logging error:

File "C:\Users\PyUser\.conda\envs\testenv\lib\encodings\cp1252.py", line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode characters in position 49-59: character maps to <undefined>


The code that writes the results is

logging.info("create output")
    with open(pathlib.Path(f'{RESULTS_PATH}/api_results.csv'), 'w', newline='') as f:
        fieldnames = list(response_data["TestResults"].values())[0].keys()
        wrtr = csv.DictWriter(f, fieldnames=fieldnames)
        wrtr.writeheader()
        wrtr.writerows(response_data["TestResults"].values())

When running the codes through postman the resulting JSON is :


"82": {
"PASS-FAIL": "PASS",
"IDENTITY": "ニコン（日本）",
"ORIGIN": "Japan"
},
"83": {
"PASS-FAIL": "PASS",
"IDENTITY": "MICROSOFT",
"ORIGIN": "USA"
},

Any advice greatly appreciated

Thanks so much!

[snippsat]

UnicodeEncodeError: 'charmap' codec can't encode

Windows choose wrong encoding,force it use utf-8.

with open(pathlib.Path(f'{RESULTS_PATH}/api_results.csv'), 'w', newline='', encoding='utf-8') as f:

[fioranosnake]

Perfect - working now!

Very Very much appreciated