Jan-29-2020, 01:42 PM
Hi All,
I have a case where i need to find any wrong in the csv file.
Detailed info:
There are some 20 set of CSV files are there each has unique columns.
Csvfile1 has 10 columns mixed of integer and string and datetime.
c1int,c2int,c3str,c4str,c5str,c6str,c7str,c8str,c9str,c10date
same way i have table in my database with
C1int,c2int,c3varchar(20),c4varchar(20),c5varchar(10),c6varchar(100),c7varchar(45),c8varchar(25),c9varchar(20),c10datetime
I need to find in the csv table if my rows has the data defined by my table , say in csvfile1 if c3column has more than 20 character , how can we check that ?
Each csv file has more than 10 million records, these is just a primarily check , just to validate the csv file.
Can we load the data set in DF and do , if so How can we achieve this else is there any option to validate the data ?
If we have any issue in the csv file, then it should not go for the further step in my process.
Thanks!
I have a case where i need to find any wrong in the csv file.
Detailed info:
There are some 20 set of CSV files are there each has unique columns.
Csvfile1 has 10 columns mixed of integer and string and datetime.
c1int,c2int,c3str,c4str,c5str,c6str,c7str,c8str,c9str,c10date
same way i have table in my database with
C1int,c2int,c3varchar(20),c4varchar(20),c5varchar(10),c6varchar(100),c7varchar(45),c8varchar(25),c9varchar(20),c10datetime
I need to find in the csv table if my rows has the data defined by my table , say in csvfile1 if c3column has more than 20 character , how can we check that ?
Each csv file has more than 10 million records, these is just a primarily check , just to validate the csv file.
Can we load the data set in DF and do , if so How can we achieve this else is there any option to validate the data ?
If we have any issue in the csv file, then it should not go for the further step in my process.
Thanks!