Python Forum
Pyspark SQL Error - mismatched input 'FROM' expecting <EOF>
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Pyspark SQL Error - mismatched input 'FROM' expecting <EOF>
#1
Hello All,
I am executing a python script in AWS EMR (Linux) which executes a sql inside or below snippet of code and erroring out. May i please know what mistake i am doing here or how to fix this?


edc_hc_final_7_sql="""
SELECT
DISTINCT
ldim.fnm_ln_id,
ldim.ln_aqsn_prd,
COALESCE(CAST(CASE WHEN ldfact.ln_entp_paid_mi_cvrg_ind='Y' THEN ehc.edc_hc_epmi ELSE eh.edc_hc END AS DECIMAL(14,10)),0) as edc_hc_final,
ldfact.ln_entp_paid_mi_cvrg_ind
FROM LN_DIM_7 ldim LEFT JOIN LN_DL_GRP_FACT_7 ldfact ON ldim.fnm_ln_id=ldfact.fnm_ln_id AND LOWER(ldfact.agmt_drvd_rnkg_typ)='primary'
AND ldfact.dl_grp_acvy_rptg_prd=cast(date_format(add_months(to_date('{0}','yyyyMM'),-1),'yyyyMM')as integer)
LEFT JOIN EDC_HC_EPMI_7 ehc ON ehc.fnm_ln_id=ldim.fnm_ln_id
LEFT JOIN EDC_HC_7 eh ON eh.fnm_ln_id=ldim.fnm_ln_id
""".format(sys.argv[3])

edc_hc_final_7_df = spark.sql(edc_hc_final_7_sql)
Error Message:

Error:
pyspark.sql.utils.ParseException: u"\nmismatched input 'FROM' expecting <EOF>(line 8, pos 0)\n\n== SQL ==\n\nSELECT\nDISTINCT\nldim.fnm_ln_id,\nldim.ln_aqsn_prd,\nCOALESCE(CAST(CASE WHEN ldfact.ln_entp_paid_mi_cvrg_ind='Y' THEN ehc.edc_hc_epmi ELSE eh.edc_hc END AS DECIMAL(14,10)),0) as edc_hc_final,\nldfact.ln_entp_paid_mi_cvrg_ind\nFROM LN_DIM_7 ldim LEFT JOIN LN_DL_GRP_FACT_7 ldfact ON ldim.fnm_ln_id=ldfact.fnm_ln_id AND LOWER(ldfact.agmt_drvd_rnkg_typ)='primary'\n^^^\nAND ldfact.dl_grp_acvy_rptg_prd=cast(date_format(add_months(to_date(''202002'','yyyyMM'),-1),'yyyyMM')as integer)\nLEFT JOIN EDC_HC_EPMI_7 ehc ON ehc.fnm_ln_id=ldim.fnm_ln_id\nLEFT JOIN EDC_HC_7 eh ON eh.fnm_ln_id=ldim.fnm_ln_id\n"
Reply
#2
That is really weird. Could you try it with a simpler query?
The only thing that strikes me is that there are two pairs of single quotes around the year-month: ''202002''. That is not correct but it does not explain the error you get.
Ariean likes this post
Reply
#3
i tried with simpler query as well but no luck.
Reply
#4
Thanks much actually that fixed the issue, double single quotes causing the issue it ran fine when i made it to single quotes.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Error in Using INPUT statement gunwaba 1 171 Jul-03-2022, 10:22 PM
Last Post: deanhystad
  JSONDecodeError: Expecting value mehtamonita 1 508 Mar-07-2022, 04:24 PM
Last Post: bowlofred
  How to iterate Groupby in Python/PySpark DrData82 2 684 Feb-05-2022, 09:59 PM
Last Post: DrData82
  PySpark Equivalent Code cpatte7372 0 517 Jan-14-2022, 08:59 PM
Last Post: cpatte7372
  Pyspark - my code works but I want to make it better Kevin 1 875 Dec-01-2021, 05:04 AM
Last Post: Kevin
  pyspark parallel write operation not working aliyesami 1 886 Oct-16-2021, 05:18 PM
Last Post: aliyesami
  pyspark creating temp files in /tmp folder aliyesami 1 1,500 Oct-16-2021, 05:15 PM
Last Post: aliyesami
Star I'm getting syntax error while using input function in def. yecktmpmbyrv 1 986 Oct-06-2021, 09:39 AM
Last Post: menator01
Photo Integration of apache spark and Kafka on eclipse pyspark aupres 1 2,254 Feb-27-2021, 08:38 AM
Last Post: Serafim
  KafkaUtils module not found on spark 3 pyspark aupres 2 3,934 Feb-17-2021, 09:40 AM
Last Post: Larz60+

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020