Python Forum
Unable to read decimal datatype using pandasql
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Unable to read decimal datatype using pandasql
#1
I have a parquet file with 4 columns. It looks something like below.


TYPE | ID | SRNO | AMT

D | 123456 | 1 | 100.00
D | 123457 | 2 | 200.00
D | 123459 | 3 | 500.00
D | 123458 | 4 | 1000.00


The Schema for this file is

dataframe.printSchema
Output:
|-- TYPE: string (nullable = true) |-- ID: integer (nullable = true) |-- SRNO: integer (nullable = true) |-- AMT: decimal(15,2) (nullable = true)
NOTE : When I read this file in pandas the schema changes for decimal and is represented as object

pandas_dataframe.dtypes
Output:
TYPE object ID int32 SRNO int32 AMT object
I get the below error when I try to run a sql on the Dataframe.

ps.sqldf("select * from pandas_dataframe")
Error:
Traceback (most recent call last): File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1229, in _execute_context cursor, statement, parameters, context File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/engine/default.py", line 577, in do_executemany cursor.executemany(statement, parameters) sqlite3.InterfaceError: Error binding parameter 3 - probably unsupported type.
FYI .. I tried casting the Decimal field to String , Double and it works fine .

Does it mean Pandasql cannot handle a file that has Decimal Datatypes ? Is there a better programatic alternatives to handles this than Typecasting explicitly .

Thanks in Advance
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  How to change the datatype of list elements? mHosseinDS86 9 1,955 Aug-24-2022, 05:26 PM
Last Post: deanhystad
  datatype check arkiboys 1 1,175 Jan-18-2022, 12:46 PM
Last Post: ndc85430
  Error when Excelwriter saving a dataframe with datetime datatype with timezone klllmmm 3 13,335 Dec-08-2020, 11:37 AM
Last Post: Larz60+
  Unable to read from serial port br0kenpixel 1 2,472 Aug-08-2020, 10:03 PM
Last Post: Larz60+
  I need to get only string datatype and report in excel file. akshay3210 3 2,361 Dec-12-2019, 09:53 AM
Last Post: akshay3210
  Unable to read update access DB saaketh 1 1,964 Sep-24-2019, 11:37 AM
Last Post: Malt
  Add two number and variable datatype is int8 ift38375 3 2,918 Aug-05-2019, 05:41 PM
Last Post: DeaD_EyE
  testing for Decimal w/o importing decimal every time Skaperen 7 4,432 May-06-2019, 10:23 PM
Last Post: Skaperen
  Unable to read data from string base64 representing a .xlsx file max 2 15,302 Apr-05-2018, 07:58 PM
Last Post: max
  Unable to read csv file inside full code snippet, at the same time its working seprat sreeraj 2 3,002 Jan-28-2018, 07:25 PM
Last Post: sreeraj

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020