Python Forum
Why does Python/gpx save data as Latin1?
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Why does Python/gpx save data as Latin1?
#1
Hello,

For some reason, the input data that are in UTF8 in SQLite end up as Latin1 in the output file:

import gpxpy.gpx
import sqlite3

gpx = gpxpy.gpx.GPX()
con = sqlite3.connect('input.sqlite')

con.row_factory = sqlite3.Row
cur = con.cursor()
cur.execute("SELECT name,latitude,longitude FROM table1");
results = cur.fetchall()
for row in results:
	w1 = gpxpy.gpx.GPXWaypoint(row["latitude"],row["longitude"])
	w1.name = row["name"]
	gpx.waypoints.append(w1)
	print("Waypoint: ",w1.name)

with open("map.gpx", "w") as f:
    f.write( gpx.to_xml())
I'm pretty sure data in SQLIte are in UTF8 because a run of ".output test.csv ; SELECT …" shows data in UTF8, so it looks like the issue lies in the code above.

Am I missing a switch somewhere?

Thank you.
Reply
#2
From Python built-in help (>>> help('open')):

Quote:open(file, mode='r', buffering=-1, encoding=None, errors=None, newline=None, closefd=True, opene
r=None)

/.../

encoding is the name of the encoding used to decode or encode the
file. This should only be used in text mode. The default encoding is
platform dependent, but any encoding supported by Python can be
passed.

One can check (and set) default encoding in locale:

import locale
locale.getlocale()
However, more robust is to specify encoding while writing into file:

with open("map.gpx", "w", encoding='UTF-8') as f:
I'm not 'in'-sane. Indeed, I am so far 'out' of sane that you appear a tiny blip on the distant coast of sanity. Bucky Katt, Get Fuzzy

Da Bishop: There's a dead bishop on the landing. I don't know who keeps bringing them in here. ....but society is to blame.
Reply
#3
You nailed it. Thank you very much.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  how to save to multiple locations during save cubangt 1 509 Oct-23-2023, 10:16 PM
Last Post: deanhystad
  Save data frame to .csv df.to.csv() mcva 1 1,497 Feb-03-2022, 07:05 PM
Last Post: mcva
  SaltStack: MySQL returner save less data into Database table columns xtc14 2 2,115 Jul-02-2021, 02:19 PM
Last Post: xtc14
  How to save json data in a dataframe shantanu97 1 2,122 Apr-15-2021, 02:44 PM
Last Post: klllmmm
  sqlite3 database does not save data across restarting the program SheeppOSU 1 3,405 Jul-24-2020, 05:53 AM
Last Post: SheeppOSU
  How to save Python Requests data sent to server? RedLeonard 5 4,799 Jul-05-2020, 10:33 AM
Last Post: RedLeonard
  How to save CSV file data into the Azure Data Lake Storage Gen2 table? Mangesh121 0 2,079 Jun-26-2020, 11:59 AM
Last Post: Mangesh121
  Save Arduino data in mysql server via raspberrypi rithikvg 1 3,348 Jun-24-2020, 10:59 PM
Last Post: Larz60+
  save data in .txt after certain interval Shaswat 1 2,057 Oct-13-2019, 07:07 AM
Last Post: Gribouillis
  Looking to Save Data Corwin 3 2,572 Oct-27-2018, 08:40 AM
Last Post: Corwin

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020