+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: General Coding Help (https://python-forum.io/forum-8.html)
+--- Thread: pyspark database exception handling (/thread-11882.html)
pyspark database exception handling - lsanthan - Jul-30-2018
We are replacing our datastage ETL tool with PySpark code.We want to mimic the error handling code in our Python script. Our ETL program fetches rows from source databases(Oracle) and then inserts the final transformed dataset to Oracle database. We are using dataframes and temporary tables in spark for enriching the data. When we insert/update rows to Oracle table we want to log the bad records(rows which failed due to database exceptions) to a text file and continue processing the remaining records. I am looking for some code examples which will help me to achieve this functionality .
RE: pyspark database exception handling - micseydel - Jul-30-2018
This seems too straightforward for an "example". Literally just wrap the line(s) in a try/except, and either print to the file or use the logging module.