Python Forum
UnicodeEncodeError - Dealing with Japanese Characters
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
UnicodeEncodeError - Dealing with Japanese Characters
#1
Hi All,

Ive got a small Python app that sends a bunch of codes to a checking API and returns the results.

Im getting errors when the the app tries to right non-english characters to a "results.csv" resulting in the following logging error:

File "C:\Users\PyUser\.conda\envs\testenv\lib\encodings\cp1252.py", line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode characters in position 49-59: character maps to <undefined>


The code that writes the results is

logging.info("create output")
    with open(pathlib.Path(f'{RESULTS_PATH}/api_results.csv'), 'w', newline='') as f:
        fieldnames = list(response_data["TestResults"].values())[0].keys()
        wrtr = csv.DictWriter(f, fieldnames=fieldnames)
        wrtr.writeheader()
        wrtr.writerows(response_data["TestResults"].values())
When running the codes through postman the resulting JSON is :


"82": {
"PASS-FAIL": "PASS",
"IDENTITY": "ニコン(日本)",
"ORIGIN": "Japan"
},
"83": {
"PASS-FAIL": "PASS",
"IDENTITY": "MICROSOFT",
"ORIGIN": "USA"
},

Any advice greatly appreciated

Thanks so much!
Reply
#2
(Jul-07-2022, 04:30 PM)fioranosnake Wrote: UnicodeEncodeError: 'charmap' codec can't encode
Windows choose wrong encoding,force it use utf-8.
with open(pathlib.Path(f'{RESULTS_PATH}/api_results.csv'), 'w', newline='', encoding='utf-8') as f:
fioranosnake likes this post
Reply
#3
Perfect - working now!

Very Very much appreciated
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  [SOLVED] [Debian] UnicodeEncodeError: 'ascii' codec Winfried 1 289 Nov-16-2022, 11:41 AM
Last Post: Winfried
  UnicodeEncodeError: 'ascii' codec can't encode character '\xfd' in position 14: ordin Armandito 6 1,164 Apr-29-2022, 12:36 PM
Last Post: Armandito
  UnicodeEncodeError caused by print when program runs from Popen SheeppOSU 5 1,335 Jan-13-2022, 08:11 AM
Last Post: SheeppOSU
  [UnicodeEncodeError from smtplib] yoohooos 0 2,327 Sep-25-2021, 04:27 AM
Last Post: yoohooos
  Dealing with duplicated data in a CSV file bts001 10 5,494 Sep-06-2021, 12:11 AM
Last Post: SamHobbs
  Remove escape characters / Unicode characters from string DreamingInsanity 5 8,869 May-15-2020, 01:37 PM
Last Post: snippsat
  how json dump Japanese yunkai 2 6,248 Mar-17-2020, 07:06 AM
Last Post: vishalhule
  Dealing with a .json nightmare... ideas? t4keheart 10 2,949 Jan-28-2020, 10:12 PM
Last Post: t4keheart
  Dealing with Exponential data parthi1705 11 7,132 May-30-2019, 10:16 AM
Last Post: buran
  UnicodeEncodeError: 'charmap' codec can't encode character '\u2019' in position 23: c kapilan15 2 3,636 Jan-14-2019, 09:11 PM
Last Post: Gribouillis

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020