Python Forum
Want a list utf8 formatted but bytestrings found
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Want a list utf8 formatted but bytestrings found
#21
Raw output of names is:
Output:
['Alexander Lepsveridze', 'John Comeau', '\xce\x86\xce\xba\xce\xb7\xcf\x82 \xce\xa4\xcf\x83\xce\xb9\xce\xac\xce\xbc\xce\xb7\xcf\x82', '\xce\x8c\xce\xbc\xce\xb9\xce\xbb\xce\xbf\xcf\x82 \xce\xa4\xcf\x83\xce\xbf\xcf\x84\xcf\x85\xce\xbb\xce\xaf\xce\xbf\xcf\x85']
Still i cannot understand WHY it reads the database names like that, instead of their normal Greek names.
Here is the code concerning names:
Output:
names = [] cur.execute( '''SELECT name, phone, hits, money FROM clients ORDER BY hits DESC''' ) data = cur.fetchall() #print each and every client record for row in data: (name, phone, hits, money) = row #populate client listing into list names.append( name ) ...... ...... # populate names, months, years names.append( '' ) names.sort() names = [s.encode('latin1').decode('utf8') for s in names]
And here is when it fail to do the encoding providing the error of:

Output:
UnicodeEncodeError('latin-1', 'Άκης Τσιάμης', 0, 4, 'ordinal not in range(256)')
I use Apache + Bottle + PyMySQL. Before chnaging to Bottle from Flask in the exact same database this error does not appear.

ps. I can give you access to my server if you want to see/try something for yourself.
Reply
#22
(Feb-16-2019, 08:31 PM)nikos Wrote: Still i cannot understand WHY it reads the database names like that, instead of their normal Greek names.
Is it you have put data into DB,or it is an exiting DB?
MySQL i think store as latin-1 default,if not doing anything like charset='utf8mb4'.
Yes MySQL has own stupid encoding utf8mb4 to have proper utf8 support.
Look at SO and utf8 data on latin1 tables.
I do not use MySQL if have choice,always PostgreSQL or SQLite for simpler stuff.

Quote:Raw output of names is:
The it should not be a problem in you running Python 3,as done a couple of times in this post.
Put in this in script:
import sys

print(sys.executable)
Python 3.7:
items = ['Alexander Lepsveridze', 'John Comeau', '\xce\x86\xce\xba\xce\xb7\xcf\x82 \xce\xa4\xcf\x83\xce\xb9\xce\xac\xce\xbc\xce\xb7\xcf\x82', '\xce\x8c\xce\xbc\xce\xb9\xce\xbb\xce\xbf\xcf\x82 \xce\xa4\xcf\x83\xce\xbf\xcf\x84\xcf\x85\xce\xbb\xce\xaf\xce\xbf\xcf\x85']
names = [s.encode('latin1').decode('utf8') for s in items]
print(names)
Output:
λ python uni1.py C:\python37\python.exe ['Alexander Lepsveridze', 'John Comeau', 'Άκης Τσιάμης', 'Όμιλος Τσοτυλίου']
Running same code in Python 2.7,and it fails.
λ py -2.7 uni1.py
C:\Python27\python.exe
Traceback (most recent call last):
  File "uni1.py", line 7, in <module>
    names = [s.encode('latin1').decode('utf8') for s in items]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xce in position 0: ordinal not in range(128)
Fix so it also work in Python 2.7.
# -*- coding: utf-8 -*-
import sys

print(sys.executable)
items = ['Alexander Lepsveridze', 'John Comeau', '\xce\x86\xce\xba\xce\xb7\xcf\x82 \xce\xa4\xcf\x83\xce\xb9\xce\xac\xce\xbc\xce\xb7\xcf\x82', '\xce\x8c\xce\xbc\xce\xb9\xce\xbb\xce\xbf\xcf\x82 \xce\xa4\xcf\x83\xce\xbf\xcf\x84\xcf\x85\xce\xbb\xce\xaf\xce\xbf\xcf\x85']
names = [s.decode('utf8') for s in items]
print(names)
for name in names:
    print(name)
Output:
py -2.7 uni1.py C:\Python27\python.exe [u'Alexander Lepsveridze', u'John Comeau', u'\u0386\u03ba\u03b7\u03c2 \u03a4\u03c3\u03b9\u03ac\u03bc\u03b7\u03c2', u'\u038c\u03bc\u03b9\u03bb\u03bf\u03c2 \u03a4\u03c3\u03bf\u03c4\u03c5\u03bb\u03af\u03bf\u03c5'] Alexander Lepsveridze John Comeau Άκης Τσιάμης Όμιλος Τσοτυλίου
Reply
#23
I just changed framework from Bottle => Flask and now the names are being read form the db normally as Greek names. I have not touched the db i just changes the framework!
Reply
#24
(Feb-16-2019, 09:34 PM)nikos Wrote: I just changed framework from Bottle => Flask and now the names are being read form the db normally as Greek names. I have not touched the db i just changes the framework!
Yes a better choice,i also mention other tools in Thread that can be smart to look at/use for a better web-development experience in Python.
Reply
#25
But can you explain why the chnage of the framework to Flask didnt presnt the same issue?
Reply
#26
(Feb-16-2019, 10:18 PM)nikos Wrote: But can you explain why the chnage of the framework to Flask didnt presnt the same issue?
No,it can be that Flask is doing encoding internal automatic.
The advantage with using Python most popular web-framework Flask GitHub(41,975 stars).
Is that a lot error get reported and fixed,33 issues open and 1614 closed.
So if someone in the past has report encoding problem form DB,it can have been fixed.
Reply
#27
I see. Well i should stick with Flask i guess.
You said something about mod_wsgi before. Well when the app is ready i have to stop using the development server and use a production httpd like Apache that iam using.

You said i shouldnt be using wsgi but then how would i run my application?
Reply
#28
(Feb-17-2019, 01:16 PM)nikos Wrote: Well when the app is ready i have to stop using the development server and use a production httpd like Apache that iam using.
Yes,and there are much better choice than Apache(httpd) and mod_wsgi.
As i mention before Gunicorn with NGINX(better and easier to setup than Apache).

DO has good tutorial as start on blank Droplet eg with Ubuntu 18.04.
How To Serve Flask Applications with Gunicorn and Nginx on Ubuntu 18.04.

Other way is the servers less way with eg ZAPPA,which use AWS Lambda as host.
Quote:AWS Lambda lets you run code without provisioning or managing servers.
You pay only for the compute time you consume
Reply
#29
Thank you very much for all your help.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  [SOLVED] [Windows] Converting filename to UTF8? Winfried 5 2,447 Sep-06-2022, 10:47 PM
Last Post: snippsat
  Formatted string not translated by gettext YvanM 10 1,918 Sep-02-2022, 08:46 PM
Last Post: YvanM
  Split string using variable found in a list japo85 2 1,238 Jul-11-2022, 08:52 AM
Last Post: japo85
  How can I found how many numbers are there in a Collatz Sequence that I found? cananb 2 2,506 Nov-23-2020, 05:15 PM
Last Post: cananb
  How to run a method on an argument in a formatted string Exsul 1 1,649 Aug-30-2019, 01:57 AM
Last Post: Exsul
  How work with formatted text in Python? AlekseyPython 3 2,776 Mar-18-2019, 05:00 AM
Last Post: AlekseyPython
  Who converts data when writing to a database with an encoding different from utf8? AlekseyPython 1 2,332 Mar-04-2019, 08:26 AM
Last Post: DeaD_EyE
  modify line in file if pattern found in list. kttan 1 2,188 Dec-10-2018, 08:45 AM
Last Post: Gribouillis
  How to detect and tell user that no matches were found in a list RedSkeleton007 6 3,813 Jul-19-2018, 06:27 PM
Last Post: woooee
  How can I write formatted (i.e. bold, italic, change font size, etc.) text to a file? JohnJSal 6 23,961 Jun-19-2018, 03:43 PM
Last Post: JohnJSal

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020