Bottom Page

Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
 Who converts data when writing to a database with an encoding different from utf8?
#1
Python 3.7.2

I write the strings from my Python code into my database. My strings contain Latin and Cyrillic characters, so in the database I use 1-byte encoding koi8-r. The miracle is that my strings without distortion are written to the database, although utf8 and koi8r have completely different sequence of characters (for example, as in ascii and utf8). Sometimes characters of other layouts appear in the text and then write errors appear.

Therefore, the questions appear:
1. Who converts strings: the database or the aiomysql library, that I use to write to the database.
2. How quickly in Python / MariaDB to remove non-koi8-r characters to avoid errors.

Thank you in advance for participating in the conversation.

Perhaps, there are databases that support an "economical" (for my case) multibyte encoding, which stores the Latin and Cyrillic characters in the first byte, and other layouts in other bytes?
Quote
#2
If I understood the code right, there happens an implicit encoding to utf8mb4, if no other default encoding has been set.
If I'm right, this means the encoding takes 4 byte for each character. This can be something for internal optimization of MySQL itself.

Where the encoding happens: https://github.com/aio-libs/aiomysql/blo...on.py#L426
DEFAULT_CHARSET from external dependency: https://github.com/PyMySQL/PyMySQL/blob/...ons.py#L91

If you have a str, it's already encoded internally with utf8.
If your input was made with a koi8-r encoding, somewhere must happen a conversion.
For example if you enter form data on a web page, you should receive the parameters as raw bytes.
Then they need to be decoded, to be represented as str.

If the query is a str, then it is automatically encoded to utf8mb4, if no other encoding has been set somewhere.
My code examples are always for Python >=3.6.0
Almost dead, but too lazy to die: https://sourceserver.info
All humans together. We don't need politicians!
Quote

Top Page

Possibly Related Threads...
Thread Author Replies Views Last Post
  Reading blob data from database by python and store it in .zip format Adityasi 2 146 Nov-18-2019, 05:22 PM
Last Post: ibreeden
  python openpyxl writing data Bikram 2 220 Oct-17-2019, 03:33 PM
Last Post: Bikram
  How to import data from database to excel nick123 1 322 Oct-09-2019, 04:39 PM
Last Post: sm51251
  Writing Data to CSV abhaydd 2 394 Aug-09-2019, 01:08 AM
Last Post: DeaD_EyE
  Want a list utf8 formatted but bytestrings found nikos 28 1,968 Feb-18-2019, 08:26 AM
Last Post: nikos
  Help with Code (Writing Data File) caroline_d_124 3 501 Jan-06-2019, 09:17 PM
Last Post: caroline_d_124
  Dataframe Data Query Database ab0217 0 543 Oct-16-2018, 02:26 AM
Last Post: ab0217
  How to update only two colums in database from csv without touching data in other col Prince_Bhatia 0 407 Aug-21-2018, 09:13 AM
Last Post: Prince_Bhatia
  Need help saving data into MySQL database reezalaq 0 638 Jun-03-2018, 07:50 PM
Last Post: reezalaq
  Recommend an Excel library for writing data? jonesin1974 1 854 Feb-09-2018, 10:28 PM
Last Post: Gribouillis

Forum Jump:


Users browsing this thread: 1 Guest(s)