Python Forum
python2 string formatting - old and new - different for unicode
Thread Rating:
  • 2 Vote(s) - 2.5 Average
  • 1
  • 2
  • 3
  • 4
  • 5
python2 string formatting - old and new - different for unicode
#1
replying to another thread, I noticed something strange. following script, running python 2.7


import forecastio

def main():
    api_key = "API KEY"
    lat = -31.967819
    lng = 115.87718
    forecast = forecastio.load_forecast(api_key, lat, lng) 
    by_day = forecast.daily()

    print "===========Daily Data========="
    print "Daily Summary: %s" %(by_day.summary)
    
    print "===========Daily Data========="
    print "Daily Summary: {}".format(by_day.summary)

if __name__ == "__main__":
    main()
and the result

Output:
===========Daily Data========= Daily Summary: Light rain on Friday through Monday, with temperatures falling to 15°C on Saturday. ===========Daily Data========= Traceback (most recent call last):  File "dark.py", line 53, in <module>    main()  File "dark.py", line 46, in main    print "Daily Summary: {}".format(by_day.summary) UnicodeEncodeError: 'ascii' codec can't encode character u'\xb0' in position 68: ordinal not in range(128)
so old-style string formatting has no problem with unicode char, while new one - using format, raise error. The behavior still the same if explicitly specify {:s}. I find it odd that there is difference. Do I miss something?
Reply
#2
From the docs:
Quote:Converting to Bytes

The opposite method of bytes.decode() is str.encode(), which returns a bytes representation of the Unicode string, encoded in the requested encoding.

The errors parameter is the same as the parameter of the decode() method but supports a few more possible handlers. As well as 'strict', 'ignore', and 'replace' (which in this case inserts a question mark instead of the unencodable character), there is also 'xmlcharrefreplace' (inserts an XML character reference), backslashreplace (inserts a \uNNNN escape sequence) and namereplace (inserts a \N{...} escape sequence).

The following example shows the different results:
>>>

>>> u = chr(40960) + 'abcd' + chr(1972)
>>> u.encode('utf-8')
b'\xea\x80\x80abcd\xde\xb4'
>>> u.encode('ascii')  
Traceback (most recent call last):
   ...
UnicodeEncodeError: 'ascii' codec can't encode character '\ua000' in
 position 0: ordinal not in range(128)
>>> u.encode('ascii', 'ignore')
b'abcd'
>>> u.encode('ascii', 'replace')
b'?abcd?'
>>> u.encode('ascii', 'xmlcharrefreplace')
b'ꀀabcd޴'
>>> u.encode('ascii', 'backslashreplace')
b'\\ua000abcd\\u07b4'
>>> u.encode('ascii', 'namereplace')
b'\\N{YI SYLLABLE IT}abcd\\u07b4'

The low-level routines for registering and accessing the available encodings are found in the codecs module. Implementing new encodings also requires understanding the codecs module. However, the encoding and decoding functions returned by this module are usually more low-level than is comfortable, and writing new encodings is a specialized task, so the module won’t be covered in this HOWTO.
https://docs.python.org/3/howto/unicode.html
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Formatting a date time string read from a csv file DosAtPython 5 1,160 Jun-19-2023, 02:12 PM
Last Post: DosAtPython
  String formatting (strptime) issues Henrio 2 811 Jan-06-2023, 06:57 PM
Last Post: deanhystad
  confused about string formatting barryjo 7 1,917 Mar-06-2022, 02:03 AM
Last Post: snippsat
  string formatting barryjo 7 1,988 Jan-02-2022, 02:08 AM
Last Post: snippsat
  Help with string formatting in classes brthurr 6 7,752 Dec-17-2021, 04:35 PM
Last Post: Jeff900
  Question on HTML formatting with set string in message Cknutson575 3 3,410 Mar-09-2021, 08:11 AM
Last Post: Cknutson575
  Remove escape characters / Unicode characters from string DreamingInsanity 5 13,419 May-15-2020, 01:37 PM
Last Post: snippsat
  smtplib: string formatting not carrying over to email ClassicalSoul 1 2,612 Apr-22-2020, 09:58 PM
Last Post: bowlofred
  Unicode string index problem luoheng 6 2,941 Nov-23-2019, 03:04 PM
Last Post: luoheng
  Trying to run a python2 script dagamer1991 3 2,484 Aug-12-2019, 12:33 PM
Last Post: buran

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020