Python Forum
Want a list utf8 formatted but bytestrings found
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Want a list utf8 formatted but bytestrings found
#22
(Feb-16-2019, 08:31 PM)nikos Wrote: Still i cannot understand WHY it reads the database names like that, instead of their normal Greek names.
Is it you have put data into DB,or it is an exiting DB?
MySQL i think store as latin-1 default,if not doing anything like charset='utf8mb4'.
Yes MySQL has own stupid encoding utf8mb4 to have proper utf8 support.
Look at SO and utf8 data on latin1 tables.
I do not use MySQL if have choice,always PostgreSQL or SQLite for simpler stuff.

Quote:Raw output of names is:
The it should not be a problem in you running Python 3,as done a couple of times in this post.
Put in this in script:
import sys

print(sys.executable)
Python 3.7:
items = ['Alexander Lepsveridze', 'John Comeau', '\xce\x86\xce\xba\xce\xb7\xcf\x82 \xce\xa4\xcf\x83\xce\xb9\xce\xac\xce\xbc\xce\xb7\xcf\x82', '\xce\x8c\xce\xbc\xce\xb9\xce\xbb\xce\xbf\xcf\x82 \xce\xa4\xcf\x83\xce\xbf\xcf\x84\xcf\x85\xce\xbb\xce\xaf\xce\xbf\xcf\x85']
names = [s.encode('latin1').decode('utf8') for s in items]
print(names)
Output:
λ python uni1.py C:\python37\python.exe ['Alexander Lepsveridze', 'John Comeau', 'Άκης Τσιάμης', 'Όμιλος Τσοτυλίου']
Running same code in Python 2.7,and it fails.
λ py -2.7 uni1.py
C:\Python27\python.exe
Traceback (most recent call last):
  File "uni1.py", line 7, in <module>
    names = [s.encode('latin1').decode('utf8') for s in items]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xce in position 0: ordinal not in range(128)
Fix so it also work in Python 2.7.
# -*- coding: utf-8 -*-
import sys

print(sys.executable)
items = ['Alexander Lepsveridze', 'John Comeau', '\xce\x86\xce\xba\xce\xb7\xcf\x82 \xce\xa4\xcf\x83\xce\xb9\xce\xac\xce\xbc\xce\xb7\xcf\x82', '\xce\x8c\xce\xbc\xce\xb9\xce\xbb\xce\xbf\xcf\x82 \xce\xa4\xcf\x83\xce\xbf\xcf\x84\xcf\x85\xce\xbb\xce\xaf\xce\xbf\xcf\x85']
names = [s.decode('utf8') for s in items]
print(names)
for name in names:
    print(name)
Output:
py -2.7 uni1.py C:\Python27\python.exe [u'Alexander Lepsveridze', u'John Comeau', u'\u0386\u03ba\u03b7\u03c2 \u03a4\u03c3\u03b9\u03ac\u03bc\u03b7\u03c2', u'\u038c\u03bc\u03b9\u03bb\u03bf\u03c2 \u03a4\u03c3\u03bf\u03c4\u03c5\u03bb\u03af\u03bf\u03c5'] Alexander Lepsveridze John Comeau Άκης Τσιάμης Όμιλος Τσοτυλίου
Reply


Messages In This Thread
RE: Want a list utf8 formatted but bytestrings found - by snippsat - Feb-16-2019, 09:30 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
  [SOLVED] [Windows] Converting filename to UTF8? Winfried 5 2,614 Sep-06-2022, 10:47 PM
Last Post: snippsat
  Formatted string not translated by gettext YvanM 10 2,041 Sep-02-2022, 08:46 PM
Last Post: YvanM
  Split string using variable found in a list japo85 2 1,327 Jul-11-2022, 08:52 AM
Last Post: japo85
  How can I found how many numbers are there in a Collatz Sequence that I found? cananb 2 2,574 Nov-23-2020, 05:15 PM
Last Post: cananb
  How to run a method on an argument in a formatted string Exsul 1 1,701 Aug-30-2019, 01:57 AM
Last Post: Exsul
  How work with formatted text in Python? AlekseyPython 3 2,846 Mar-18-2019, 05:00 AM
Last Post: AlekseyPython
  Who converts data when writing to a database with an encoding different from utf8? AlekseyPython 1 2,397 Mar-04-2019, 08:26 AM
Last Post: DeaD_EyE
  modify line in file if pattern found in list. kttan 1 2,240 Dec-10-2018, 08:45 AM
Last Post: Gribouillis
  How to detect and tell user that no matches were found in a list RedSkeleton007 6 3,938 Jul-19-2018, 06:27 PM
Last Post: woooee
  How can I write formatted (i.e. bold, italic, change font size, etc.) text to a file? JohnJSal 6 24,147 Jun-19-2018, 03:43 PM
Last Post: JohnJSal

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020