Pyspark SQL Error - mismatched input 'FROM' expecting <EOF>

[Ariean]

Hello All,
I am executing a python script in AWS EMR (Linux) which executes a sql inside or below snippet of code and erroring out. May i please know what mistake i am doing here or how to fix this?

1 2 3 4 5 6 7 8 9 10 11 12 13 14

edc_hc_final_7_sql=""" SELECT DISTINCT ldim.fnm_ln_id, ldim.ln_aqsn_prd, COALESCE(CAST(CASE WHEN ldfact.ln_entp_paid_mi_cvrg_ind='Y' THEN ehc.edc_hc_epmi ELSE eh.edc_hc END AS DECIMAL(14,10)),0) as edc_hc_final, ldfact.ln_entp_paid_mi_cvrg_ind FROM LN_DIM_7 ldim LEFT JOIN LN_DL_GRP_FACT_7 ldfact ON ldim.fnm_ln_id=ldfact.fnm_ln_id AND LOWER(ldfact.agmt_drvd_rnkg_typ)='primary' AND ldfact.dl_grp_acvy_rptg_prd=cast(date_format(add_months(to_date('{0}','yyyyMM'),-1),'yyyyMM')as integer) LEFT JOIN EDC_HC_EPMI_7 ehc ON ehc.fnm_ln_id=ldim.fnm_ln_id LEFT JOIN EDC_HC_7 eh ON eh.fnm_ln_id=ldim.fnm_ln_id """.format(sys.argv[3])   edc_hc_final_7_df = spark.sql(edc_hc_final_7_sql)

Error Message:

Error:
pyspark.sql.utils.ParseException: u"\nmismatched input 'FROM' expecting <EOF>(line 8, pos 0)\n\n== SQL ==\n\nSELECT\nDISTINCT\nldim.fnm_ln_id,\nldim.ln_aqsn_prd,\nCOALESCE(CAST(CASE WHEN ldfact.ln_entp_paid_mi_cvrg_ind='Y' THEN ehc.edc_hc_epmi ELSE eh.edc_hc END AS DECIMAL(14,10)),0) as edc_hc_final,\nldfact.ln_entp_paid_mi_cvrg_ind\nFROM LN_DIM_7 ldim LEFT JOIN LN_DL_GRP_FACT_7 ldfact ON ldim.fnm_ln_id=ldfact.fnm_ln_id AND LOWER(ldfact.agmt_drvd_rnkg_typ)='primary'\n^^^\nAND ldfact.dl_grp_acvy_rptg_prd=cast(date_format(add_months(to_date(''202002'','yyyyMM'),-1),'yyyyMM')as integer)\nLEFT JOIN EDC_HC_EPMI_7 ehc ON ehc.fnm_ln_id=ldim.fnm_ln_id\nLEFT JOIN EDC_HC_7 eh ON eh.fnm_ln_id=ldim.fnm_ln_id\n"

[ibreeden]

That is really weird. Could you try it with a simpler query?
The only thing that strikes me is that there are two pairs of single quotes around the year-month: ''202002''. That is not correct but it does not explain the error you get.

[Ariean]

i tried with simpler query as well but no luck.

[Ariean]

Thanks much actually that fixed the issue, double single quotes causing the issue it ran fine when i made it to single quotes.