Python Forum
Pyspark SQL Error - mismatched input 'FROM' expecting <EOF>
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Pyspark SQL Error - mismatched input 'FROM' expecting <EOF>
#1
Hello All,
I am executing a python script in AWS EMR (Linux) which executes a sql inside or below snippet of code and erroring out. May i please know what mistake i am doing here or how to fix this?


edc_hc_final_7_sql="""
SELECT
DISTINCT
ldim.fnm_ln_id,
ldim.ln_aqsn_prd,
COALESCE(CAST(CASE WHEN ldfact.ln_entp_paid_mi_cvrg_ind='Y' THEN ehc.edc_hc_epmi ELSE eh.edc_hc END AS DECIMAL(14,10)),0) as edc_hc_final,
ldfact.ln_entp_paid_mi_cvrg_ind
FROM LN_DIM_7 ldim LEFT JOIN LN_DL_GRP_FACT_7 ldfact ON ldim.fnm_ln_id=ldfact.fnm_ln_id AND LOWER(ldfact.agmt_drvd_rnkg_typ)='primary'
AND ldfact.dl_grp_acvy_rptg_prd=cast(date_format(add_months(to_date('{0}','yyyyMM'),-1),'yyyyMM')as integer)
LEFT JOIN EDC_HC_EPMI_7 ehc ON ehc.fnm_ln_id=ldim.fnm_ln_id
LEFT JOIN EDC_HC_7 eh ON eh.fnm_ln_id=ldim.fnm_ln_id
""".format(sys.argv[3])

edc_hc_final_7_df = spark.sql(edc_hc_final_7_sql)
Error Message:

Error:
pyspark.sql.utils.ParseException: u"\nmismatched input 'FROM' expecting <EOF>(line 8, pos 0)\n\n== SQL ==\n\nSELECT\nDISTINCT\nldim.fnm_ln_id,\nldim.ln_aqsn_prd,\nCOALESCE(CAST(CASE WHEN ldfact.ln_entp_paid_mi_cvrg_ind='Y' THEN ehc.edc_hc_epmi ELSE eh.edc_hc END AS DECIMAL(14,10)),0) as edc_hc_final,\nldfact.ln_entp_paid_mi_cvrg_ind\nFROM LN_DIM_7 ldim LEFT JOIN LN_DL_GRP_FACT_7 ldfact ON ldim.fnm_ln_id=ldfact.fnm_ln_id AND LOWER(ldfact.agmt_drvd_rnkg_typ)='primary'\n^^^\nAND ldfact.dl_grp_acvy_rptg_prd=cast(date_format(add_months(to_date(''202002'','yyyyMM'),-1),'yyyyMM')as integer)\nLEFT JOIN EDC_HC_EPMI_7 ehc ON ehc.fnm_ln_id=ldim.fnm_ln_id\nLEFT JOIN EDC_HC_7 eh ON eh.fnm_ln_id=ldim.fnm_ln_id\n"
Reply
#2
That is really weird. Could you try it with a simpler query?
The only thing that strikes me is that there are two pairs of single quotes around the year-month: ''202002''. That is not correct but it does not explain the error you get.
Ariean likes this post
Reply
#3
i tried with simpler query as well but no luck.
Reply
#4
Thanks much actually that fixed the issue, double single quotes causing the issue it ran fine when i made it to single quotes.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  pyspark dataframe to json without header vijz 0 114 Nov-28-2020, 05:36 PM
Last Post: vijz
  EOF error while taking input ShishirModi 1 325 Sep-27-2020, 11:28 AM
Last Post: jefsummers
  ValueError: shape mismatched: objects cannot be broadcast to a single shape Laplace12 0 798 Jul-14-2020, 11:45 AM
Last Post: Laplace12
  Input Error Dream 2 512 Jul-12-2020, 05:41 PM
Last Post: bowlofred
  Getting an error while using input function dcsethia 5 486 May-11-2020, 04:59 PM
Last Post: buran
  TypeError: size; expecting a recognized type filling string dict a11_m11 0 560 Feb-10-2020, 08:26 AM
Last Post: a11_m11
  Pyspark "mismatched input FIELDS" Mabooka 1 887 Aug-31-2019, 08:51 AM
Last Post: Mabooka
  catch input type error mcmxl22 5 809 Aug-11-2019, 07:33 AM
Last Post: wavic
  inserting input gives me error message RubberNuggets 3 761 Jan-15-2019, 06:17 PM
Last Post: buran
  an input error or elif error , how can i fix this the_fire_pharaoh 1 856 Dec-15-2018, 09:47 PM
Last Post: Gribouillis

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020