Not sure where to start?

[torz]

Hey guys,

Sorry if yet another noob question - hoping you are all kind enough to help out / point me in the right direction because I'm not even sure where to start with what I'm trying to do.

My script / program is downloading a bunch of csv files off a ftp server and I'm using sqlldr via a Popen process to import them into an oracle DB. This is all working perfectly along with some other validation checks after the load, and now for the part where I'm stuck

I have the code below which is running without an issue and is giving me the expected results when executed with print statements.

import linecache
import os

ctl_log_dir = "log_directory"

for file in os.listdir(ctl_log_dir):
    print("Table import results for " + file[:-8] + ": ")
    find_lines = ['Total logical records read:','Total logical records rejected:','Total logical records skipped:','Total logical records discarded:','Rows successfully loaded.']
    with open(ctl_log_dir + file) as f:
        for i, line in enumerate(f, 1):
            for find in find_lines:
                if find in line:
                    print(linecache.getline(ctl_log_dir + file, i).strip())

The output of the print statements is:

Output:
Table import results for TABLENAME1: 
703 Rows successfully loaded.
Total logical records skipped:          0
Total logical records read:           704
Total logical records rejected:         1
Total logical records discarded:        0
Table import results for TABLENAME2: 
17973 Rows successfully loaded.
Total logical records skipped:          0
Total logical records read:         17974
Total logical records rejected:         1
Total logical records discarded:        0
Table import results for TABLENAME3: 
1151 Rows successfully loaded.
Total logical records skipped:          0
Total logical records read:          1152
Total logical records rejected:         1
Total logical records discarded:        0
Table import results for TABLENAME4: 
4301 Rows successfully loaded.
Total logical records skipped:          0
Total logical records read:          4302
Total logical records rejected:         1
Total logical records discarded:        0
Table import results for TABLENAME5: 
5704 Rows successfully loaded.
Total logical records skipped:          0
Total logical records read:          5705
Total logical records rejected:         1
Total logical records discarded:        0
Table import results for TABLENAME6: 
57 Rows successfully loaded.
Total logical records skipped:          0
Total logical records read:            58
Total logical records rejected:         1
Total logical records discarded:        0
Table import results for TABLENAME7: 
57 Rows successfully loaded.
Total logical records skipped:          0
Total logical records read:            58
Total logical records rejected:         1
Total logical records discarded:        0
Table import results for TABLENAME8: 
51 Rows successfully loaded.
Total logical records skipped:          0
Total logical records read:            52
Total logical records rejected:         1
Total logical records discarded:        0
Table import results for TABLENAME9: 
11094 Rows successfully loaded.
Total logical records skipped:          0
Total logical records read:         11095
Total logical records rejected:         1
Total logical records discarded:        0
Table import results for TABLENAME10: 
151 Rows successfully loaded.
Total logical records skipped:          0
Total logical records read:           152
Total logical records rejected:         1
Total logical records discarded:        0
Table import results for TABLENAME11: 
161 Rows successfully loaded.
Total logical records skipped:          0
Total logical records read:           162
Total logical records rejected:         1
Total logical records discarded:        0

What I'm wanting to do with this output is either:
1. Get it all into 1 long sting OR
2. Get 1 string for each table (eg Table import results for TABLENAME1: + the 5 lines below it)

I thought that I may be able to just concatenate it all into 1 string or a string for each table? But I wasn't to sure what the best way to approach it would be...

Any help will be greatly appreciated!

[Mekire]

In both cases it sounds like you should just create a list and append each valid entry to it.  Then at the end if you want one long string you just use str.join.

Have you tried this?

[torz]

I haven't tried that - it was what I was thinking of as I was writing the post...

So being rather new to python - I'm assuming if I was to go with a the list append approach, I'd create and empty list above the initial for and then I basically change the print commands to append to the list instead of printing - Is that correct?

Then once the whole list has the same as what is in the output section above I can just split that into 1 string for each table?

Sorry if these are all really stupid questions...

[Mekire]

Then once the whole list has the same as what is in the output section above I can just split that into 1 string for each table?

Probably just make a list of lists. Create a main list before you start iterating through files. Then create a list inside for each file and append the lines to it. Then finally append that list onto the main list.

Try to write it and then we can work out the details.

[torz]

Thanks Mekire!
Greatly appreciated!

I'm pretty sure this is not quite what you were suggesting, but it is working the way I needed it to - so there is probably no need to have 1 string per table in the end...

import linecache
import os

ctl_log_dir = "log_directory"

list1 = []
for file in os.listdir(ctl_log_dir):
    list1.append("<b>Table import results for " + file[:-8] + ": </b><br>")
    find_lines = ['Total logical records read:','Total logical records rejected:','Total logical records skipped:','Total logical records discarded:','Rows successfully loaded.']
    with open(ctl_log_dir + file) as f:
        for i, line in enumerate(f, 1):
            for find in find_lines:
                if find in line:
                    list1.append(linecache.getline(ctl_log_dir + file, i) + "<br>".strip())
str1 = ''.join(list1)

In case anyone is wondering why there are a couple of html formatting things being appended, it is because I'm passing the final string to a procedure in our oracle database which will does a bunch of things based on some conditions that it gets passed - one of which is sending an email & scheduling some jobs if some of those conditions are met.