Python Forum
python2 string formatting - old and new - different for unicode - Printable Version

+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: General Coding Help (https://python-forum.io/forum-8.html)
+--- Thread: python2 string formatting - old and new - different for unicode (/thread-3351.html)



python2 string formatting - old and new - different for unicode - buran - May-16-2017

replying to another thread, I noticed something strange. following script, running python 2.7


import forecastio

def main():
    api_key = "API KEY"
    lat = -31.967819
    lng = 115.87718
    forecast = forecastio.load_forecast(api_key, lat, lng) 
    by_day = forecast.daily()

    print "===========Daily Data========="
    print "Daily Summary: %s" %(by_day.summary)
    
    print "===========Daily Data========="
    print "Daily Summary: {}".format(by_day.summary)

if __name__ == "__main__":
    main()
and the result

Output:
===========Daily Data========= Daily Summary: Light rain on Friday through Monday, with temperatures falling to 15°C on Saturday. ===========Daily Data========= Traceback (most recent call last):  File "dark.py", line 53, in <module>    main()  File "dark.py", line 46, in main    print "Daily Summary: {}".format(by_day.summary) UnicodeEncodeError: 'ascii' codec can't encode character u'\xb0' in position 68: ordinal not in range(128)
so old-style string formatting has no problem with unicode char, while new one - using format, raise error. The behavior still the same if explicitly specify {:s}. I find it odd that there is difference. Do I miss something?


RE: python2 string formatting - old and new - different for unicode - Larz60+ - May-16-2017

From the docs:
Quote:Converting to Bytes

The opposite method of bytes.decode() is str.encode(), which returns a bytes representation of the Unicode string, encoded in the requested encoding.

The errors parameter is the same as the parameter of the decode() method but supports a few more possible handlers. As well as 'strict', 'ignore', and 'replace' (which in this case inserts a question mark instead of the unencodable character), there is also 'xmlcharrefreplace' (inserts an XML character reference), backslashreplace (inserts a \uNNNN escape sequence) and namereplace (inserts a \N{...} escape sequence).

The following example shows the different results:
>>>

>>> u = chr(40960) + 'abcd' + chr(1972)
>>> u.encode('utf-8')
b'\xea\x80\x80abcd\xde\xb4'
>>> u.encode('ascii')  
Traceback (most recent call last):
   ...
UnicodeEncodeError: 'ascii' codec can't encode character '\ua000' in
 position 0: ordinal not in range(128)
>>> u.encode('ascii', 'ignore')
b'abcd'
>>> u.encode('ascii', 'replace')
b'?abcd?'
>>> u.encode('ascii', 'xmlcharrefreplace')
b'ꀀabcd޴'
>>> u.encode('ascii', 'backslashreplace')
b'\\ua000abcd\\u07b4'
>>> u.encode('ascii', 'namereplace')
b'\\N{YI SYLLABLE IT}abcd\\u07b4'

The low-level routines for registering and accessing the available encodings are found in the codecs module. Implementing new encodings also requires understanding the codecs module. However, the encoding and decoding functions returned by this module are usually more low-level than is comfortable, and writing new encodings is a specialized task, so the module won’t be covered in this HOWTO.
https://docs.python.org/3/howto/unicode.html