(Aug-13-2023, 05:00 PM)cubangt Wrote: So i am having to process a few different text files, for each file i process, i need too capture runtime duration, record count and timestampYou should link to other Thread or continue there,because i guess all these adds should work with code already written.
Quote:Since i'm already using the scan_files() function is it possible to incorporate the mapcount() functionality into the main function?Have to change scan_files() function and also try to add this in with exiting code.
So start could be something like this.
import time, os import subprocess from concurrent.futures import ThreadPoolExecutor import pandas as pd def ping(ip): return ( ip, subprocess.run( f"ping {ip} -n 1", stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL ).returncode, ) def scan_files(): directory = 'G:/div_code/egg/ping' ip_files = [] for entry in os.scandir(directory): if entry.is_file() and entry.name.endswith('.txt'): if 'ip_list' in entry.name: pt = directory + '/' + entry.name ip_files.append(pt) return ip_files if __name__ == '__main__': df_lst = [] for fname in scan_files(): start = time.time() with open(fname) as file: park = [ip.strip() for ip in file] executor = ThreadPoolExecutor(12) df = pd.DataFrame(executor.map(ping, park)) #df.to_csv(r'ip_output.csv', header=False, index=False, quoting=None) print(df) end = time.time() time_used = end - start df_lst.append(df) df_lst.append(f'File <{fname}> used {time_used:.2f} sec') print(f'File <{fname}> used {time_used:.2f} sec')
Output: 0 1
0 python-forum.io 0
1 youtube.com 0
2 youtube.com99 1
3 www.vg.no 0
4 python-forum.io99 1
File <G:/div_code/egg/ping/ip_list1.txt> used 0.13 sec
0 1
0 python-forum.io 0
1 youtube.com 0
2 youtube.com99 1
3 www.vg.no 0
4 python-forum.io99 1
File <G:/div_code/egg/ping/ip_list2.txt> used 0.07 sec
Output over is also in list df_lst
.So it will take time on each files,and Pandas index will work as file count for each file.
As what's in
df_lst
is still Pandas so can eg use count lines in files or if dive be 2 will get file count.>>> df_lst[0] 0 1 0 python-forum.io 0 1 youtube.com 0 2 youtube.com99 1 3 www.vg.no 0 4 python-forum.io99 1 >>> df_lst[0].count() 0 5 1 5 dtype: int64 >>> len(df_lst) // 2 2 >>> df_lst[0].count() + df_lst[2].count() 0 10 1 10 dtype: int64