Python Forum
How can i combine these two functions so i only open the file once?
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
How can i combine these two functions so i only open the file once?
#3
(Aug-13-2023, 05:00 PM)cubangt Wrote: So i am having to process a few different text files, for each file i process, i need too capture runtime duration, record count and timestamp
You should link to other Thread or continue there,because i guess all these adds should work with code already written.
Quote:Since i'm already using the scan_files() function is it possible to incorporate the mapcount() functionality into the main function?
Have to change scan_files() function and also try to add this in with exiting code.
So start could be something like this.
import time, os
import subprocess
from concurrent.futures import ThreadPoolExecutor
import pandas as pd

def ping(ip):
    return (
        ip,
        subprocess.run(
            f"ping {ip} -n 1", stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL
        ).returncode,
    )

def scan_files():
    directory = 'G:/div_code/egg/ping'
    ip_files = []
    for entry in os.scandir(directory):
        if entry.is_file() and entry.name.endswith('.txt'):
            if 'ip_list' in entry.name:
                pt = directory + '/' + entry.name
                ip_files.append(pt)
    return ip_files

if __name__ == '__main__':
    df_lst = []
    for fname in scan_files():
        start = time.time()
        with open(fname) as file:
            park = [ip.strip() for ip in file]
            executor = ThreadPoolExecutor(12)
            df = pd.DataFrame(executor.map(ping, park))
            #df.to_csv(r'ip_output.csv', header=False, index=False, quoting=None)
            print(df)
            end = time.time()
            time_used = end - start
            df_lst.append(df)
            df_lst.append(f'File <{fname}> used {time_used:.2f} sec')
            print(f'File <{fname}> used {time_used:.2f} sec')
Output:
0 1 0 python-forum.io 0 1 youtube.com 0 2 youtube.com99 1 3 www.vg.no 0 4 python-forum.io99 1 File <G:/div_code/egg/ping/ip_list1.txt> used 0.13 sec 0 1 0 python-forum.io 0 1 youtube.com 0 2 youtube.com99 1 3 www.vg.no 0 4 python-forum.io99 1 File <G:/div_code/egg/ping/ip_list2.txt> used 0.07 sec
Output over is also in list df_lst.
So it will take time on each files,and Pandas index will work as file count for each file.
As what's in df_lst is still Pandas so can eg use count lines in files or if dive be 2 will get file count.
>>> df_lst[0]
                   0  1
0    python-forum.io  0
1        youtube.com  0
2      youtube.com99  1
3          www.vg.no  0
4  python-forum.io99  1
>>> df_lst[0].count()
0    5
1    5
dtype: int64
>>> len(df_lst) // 2
2
>>> df_lst[0].count() + df_lst[2].count()
0    10
1    10
dtype: int64
cubangt likes this post
Reply


Messages In This Thread
RE: How can i combine these two functions so i only open the file once? - by snippsat - Aug-13-2023, 08:56 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
  Open/save file on Android frohr 0 410 Jan-24-2024, 06:28 PM
Last Post: frohr
  file open "file not found error" shanoger 8 1,387 Dec-14-2023, 08:03 AM
Last Post: shanoger
  I cannot able open a file in python ? ted 5 3,756 Feb-11-2023, 02:38 AM
Last Post: ted
  testing an open file Skaperen 7 1,509 Dec-20-2022, 02:19 AM
Last Post: Skaperen
  I get an FileNotFouerror while try to open(file,"rt"). My goal is to replace str decoded 1 1,464 May-06-2022, 01:44 PM
Last Post: Larz60+
  Dynamic File Name to a shared folder with open command in python sjcsvatt 9 6,341 Jan-07-2022, 04:55 PM
Last Post: bowlofred
  Functions to consider for file renaming and moving around directories cubangt 2 1,856 Jan-07-2022, 02:16 PM
Last Post: cubangt
  Open an excel file Newbie1114 1 2,388 Jun-16-2021, 09:11 PM
Last Post: Gribouillis
  How to open MIDI-file and get events in a list? philipbergwerf 7 5,174 May-29-2021, 08:24 AM
Last Post: j.crater
  Error on open of file created with tempfile.TemporaryDirectory() Brian177 4 6,497 Apr-05-2021, 07:12 PM
Last Post: Brian177

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020