+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: General Coding Help (https://python-forum.io/forum-8.html)
+--- Thread: Pyspark SQL Error - mismatched input 'FROM' expecting <EOF> (/thread-31056.html)
Pyspark SQL Error - mismatched input 'FROM' expecting <EOF> - Ariean - Nov-20-2020
Hello All,
I am executing a python script in AWS EMR (Linux) which executes a sql inside or below snippet of code and erroring out. May i please know what mistake i am doing here or how to fix this?
edc_hc_final_7_sql="""
SELECT
DISTINCT
ldim.fnm_ln_id,
ldim.ln_aqsn_prd,
COALESCE(CAST(CASE WHEN ldfact.ln_entp_paid_mi_cvrg_ind='Y' THEN ehc.edc_hc_epmi ELSE eh.edc_hc END AS DECIMAL(14,10)),0) as edc_hc_final,
ldfact.ln_entp_paid_mi_cvrg_ind
FROM LN_DIM_7 ldim LEFT JOIN LN_DL_GRP_FACT_7 ldfact ON ldim.fnm_ln_id=ldfact.fnm_ln_id AND LOWER(ldfact.agmt_drvd_rnkg_typ)='primary'
AND ldfact.dl_grp_acvy_rptg_prd=cast(date_format(add_months(to_date('{0}','yyyyMM'),-1),'yyyyMM')as integer)
LEFT JOIN EDC_HC_EPMI_7 ehc ON ehc.fnm_ln_id=ldim.fnm_ln_id
LEFT JOIN EDC_HC_7 eh ON eh.fnm_ln_id=ldim.fnm_ln_id
""".format(sys.argv[3])
edc_hc_final_7_df = spark.sql(edc_hc_final_7_sql)Error Message:Error:pyspark.sql.utils.ParseException: u"\nmismatched input 'FROM' expecting <EOF>(line 8, pos 0)\n\n== SQL ==\n\nSELECT\nDISTINCT\nldim.fnm_ln_id,\nldim.ln_aqsn_prd,\nCOALESCE(CAST(CASE WHEN ldfact.ln_entp_paid_mi_cvrg_ind='Y' THEN ehc.edc_hc_epmi ELSE eh.edc_hc END AS DECIMAL(14,10)),0) as edc_hc_final,\nldfact.ln_entp_paid_mi_cvrg_ind\nFROM LN_DIM_7 ldim LEFT JOIN LN_DL_GRP_FACT_7 ldfact ON ldim.fnm_ln_id=ldfact.fnm_ln_id AND LOWER(ldfact.agmt_drvd_rnkg_typ)='primary'\n^^^\nAND ldfact.dl_grp_acvy_rptg_prd=cast(date_format(add_months(to_date(''202002'','yyyyMM'),-1),'yyyyMM')as integer)\nLEFT JOIN EDC_HC_EPMI_7 ehc ON ehc.fnm_ln_id=ldim.fnm_ln_id\nLEFT JOIN EDC_HC_7 eh ON eh.fnm_ln_id=ldim.fnm_ln_id\n"RE: Pyspark SQL Error - mismatched input 'FROM' expecting <EOF> - ibreeden - Nov-20-2020
That is really weird. Could you try it with a simpler query?
The only thing that strikes me is that there are two pairs of single quotes around the year-month:
''202002''. That is not correct but it does not explain the error you get.
RE: Pyspark SQL Error - mismatched input 'FROM' expecting <EOF> - Ariean - Nov-20-2020
i tried with simpler query as well but no luck.
RE: Pyspark SQL Error - mismatched input 'FROM' expecting <EOF> - Ariean - Nov-20-2020
Thanks much actually that fixed the issue, double single quotes causing the issue it ran fine when i made it to single quotes.