Python Forum
Pyspark "mismatched input FIELDS"
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Pyspark "mismatched input FIELDS"
I am looking for help.
I am trying to use SerDes with Hive in pySpark.sql.
Here is my SQL:

user_id VARCHAR(36),
weekstartdate date, 
user_name VARCHAR(36), 
user_age int, ... )
                       ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
                       FIELDS TERMINATED BY '|t' 
                       STORED AS TEXTFILE
                       LOCATION 's3://stx-apollo-pr-datascience-shared/unloads/testdata/v1/mphd/customer_attributes_weekly'
                       TBLPROPERTIES ('hive.lazysimple.extended_boolean_literal'='true')
With that, I receive the error:
pyspark.sql.utils.ParseException: "\nmismatched input 'FIELDS' expecting <EOF>... " FIELDS TERMINATED BY '|t' -----------------------^^^
If instead of
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
I put something like
-- there is no error.

Obviously I use some wrong syntax but I cannot find out what's exactly wrong:
I took the line
Quote:ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
from Hive documentation.

Any ideas? I would really, really appreciate any help.
Thank you!

By accident, posted the same twice. Don't know how to delete a post.

... Please disregard. Posted twice by mistake (I am new to this forum),
couldn't find how to remove post..

Possibly Related Threads…
Thread Author Replies Views Last Post
  how to filter two fields in json using python python_student 4 717 Mar-15-2021, 05:58 PM
Last Post: python_student
Photo Integration of apache spark and Kafka on eclipse pyspark aupres 1 980 Feb-27-2021, 08:38 AM
Last Post: Serafim
  KafkaUtils module not found on spark 3 pyspark aupres 2 1,447 Feb-17-2021, 09:40 AM
Last Post: Larz60+
  PySpark Coding Challenge cpatte7372 3 1,198 Feb-14-2021, 04:49 PM
Last Post: ndc85430
  pyspark dataframe to json without header vijz 0 518 Nov-28-2020, 05:36 PM
Last Post: vijz
  Pyspark SQL Error - mismatched input 'FROM' expecting <EOF> Ariean 3 10,299 Nov-20-2020, 03:49 PM
Last Post: Ariean
  Json fields not being retrieved mrcurious2020 4 746 Sep-14-2020, 06:24 AM
Last Post: bowlofred
  ValueError: shape mismatched: objects cannot be broadcast to a single shape Laplace12 0 2,218 Jul-14-2020, 11:45 AM
Last Post: Laplace12
  How Does pyspark deal with Spaces in Queries cpatte7372 3 1,403 Jul-31-2018, 09:53 PM
Last Post: micseydel
  pyspark sql unable to recognize SQL query command cpatte7372 6 7,043 Jul-31-2018, 04:17 PM
Last Post: micseydel

Forum Jump:

User Panel Messages

Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020