How can i combine these two functions so i only open the file once?

[cubangt]

So i am having to process a few different text files, for each file i process, i need too capture runtime duration, record count and timestamp

Since i'm already using the scan_files() function is it possible to incorporate the mapcount() functionality into the main function?
I mean if im already opening up the first text file, why not get the count returned while its open. so far just adding the readline logic below the strip() in the main logic isnt working..

# ROUTINE TO OPEN THE APPROPRIATE TEXT FILES TO PROCESS THE IP LIST
def scan_files():
    directory = '.'
    for entry in os.scandir(directory):
        if entry.is_file() and entry.name.endswith('.txt'):
            if 'ip_list' in entry.name:
                pt = directory + '/' + entry.name
                with open(pt) as file:
                    for ip in file:
                        yield ip.strip()

# ROUTINE TO GET FILE COUNT                        
def mapcount(filename):
    with open(filename, "r+") as f:
        buf = mmap.mmap(f.fileno(), 0)
        lines = 0
        readline = buf.readline
        while readline():
            lines += 1
        return lines  

I want to open the file once, get the count and move on to the rest of the script.

[Gribouillis]

There is a problem with the interface: to client code scan_files() is nothing but an iterable of IPs. Suppose it also computes the number of lines in the files, how is it going to output that number?

Also the time taken by the generator to run depends on client code. If client code requests the next IP every 10 minutes, the generator will be very long to consume.

[snippsat]

So i am having to process a few different text files, for each file i process, i need too capture runtime duration, record count and timestamp

You should link to other Thread or continue there,because i guess all these adds should work with code already written.

Since i'm already using the scan_files() function is it possible to incorporate the mapcount() functionality into the main function?

Have to change scan_files() function and also try to add this in with exiting code.
So start could be something like this.

import time, os
import subprocess
from concurrent.futures import ThreadPoolExecutor
import pandas as pd

def ping(ip):
    return (
        ip,
        subprocess.run(
            f"ping {ip} -n 1", stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL
        ).returncode,
    )

def scan_files():
    directory = 'G:/div_code/egg/ping'
    ip_files = []
    for entry in os.scandir(directory):
        if entry.is_file() and entry.name.endswith('.txt'):
            if 'ip_list' in entry.name:
                pt = directory + '/' + entry.name
                ip_files.append(pt)
    return ip_files

if __name__ == '__main__':
    df_lst = []
    for fname in scan_files():
        start = time.time()
        with open(fname) as file:
            park = [ip.strip() for ip in file]
            executor = ThreadPoolExecutor(12)
            df = pd.DataFrame(executor.map(ping, park))
            #df.to_csv(r'ip_output.csv', header=False, index=False, quoting=None)
            print(df)
            end = time.time()
            time_used = end - start
            df_lst.append(df)
            df_lst.append(f'File <{fname}> used {time_used:.2f} sec')
            print(f'File <{fname}> used {time_used:.2f} sec')

Output:
                   0  1
0    python-forum.io  0
1        youtube.com  0
2      youtube.com99  1
3          www.vg.no  0
4  python-forum.io99  1
File <G:/div_code/egg/ping/ip_list1.txt> used 0.13 sec
                   0  1
0    python-forum.io  0
1        youtube.com  0
2      youtube.com99  1
3          www.vg.no  0
4  python-forum.io99  1
File <G:/div_code/egg/ping/ip_list2.txt> used 0.07 sec

Output over is also in list df_lst.
So it will take time on each files,and Pandas index will work as file count for each file.
As what's in df_lst is still Pandas so can eg use count lines in files or if dive be 2 will get file count.

>>> df_lst[0]
                   0  1
0    python-forum.io  0
1        youtube.com  0
2      youtube.com99  1
3          www.vg.no  0
4  python-forum.io99  1
>>> df_lst[0].count()
0    5
1    5
dtype: int64
>>> len(df_lst) // 2
2
>>> df_lst[0].count() + df_lst[2].count()
0    10
1    10
dtype: int64

[deanhystad]

I don't think it applies to this problem, but you can also pass information through the iterable and user function. In this example I pass the filename and the line number of the IP address in the file. The user function passes all this info along with the ping status and a timestamp.

def ping(args):
    ip, file, counter = args
    returncode = subprocess.run(
        f"ping {ip} -n 1", stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL
    ).returncode
    return ip, file, counter, datetime.now(), returncode


def scan_files():
    directory = Path('.')
    for name in directory.glob("*ip_list*.txt"):
        count = 0
        with open(name, "r") as file:
            for ip in file:
                count += 1
                yield ip.strip(), name.name, count


executor = ThreadPoolExecutor(125)
df = pd.DataFrame(executor.map(ping, scan_files()))
print(df)

Output:
                   0            1  2                          3  4
0    python-forum.io  ip_list.txt  1 2023-08-13 21:53:42.935575  0
1        youtube.com  ip_list.txt  2 2023-08-13 21:53:42.929570  0
2      youtube.com99  ip_list.txt  3 2023-08-13 21:53:42.910397  1
3          www.vg.no  ip_list.txt  4 2023-08-13 21:53:43.117031  0
4  python-forum.io99  ip_list.txt  5 2023-08-13 21:53:42.911398  1

[snippsat]

Here code where i use Loguru one of my favorite Python libraries.
So now logged all into ip.log file.
Now that also log a date/time stamp when file run,have line number,name of file and time used.
So also just bye looking at 19:00:07 and next file start 19:00:16,that time used is ca 9-sec.
If uncomment pd.set_option('display.max_rows', None) it will log all ip and not just make head/tail as shown now.

import time, os
import subprocess
from concurrent.futures import ThreadPoolExecutor
import pandas as pd
#pd.set_option('display.max_rows', None)
from loguru import logger
logger.remove() # Only info to file
logger.add("ip.log", rotation="2 day", format="{time:YYYY-MM-DD at HH:mm:ss}\n{message}")

def ping(ip):
    return (
        ip,
        subprocess.run(
            f"ping {ip} -n 1", stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL
        ).returncode,
    )

def scan_files():
    directory = 'G:/div_code/egg/ping'
    ip_files = []
    for entry in os.scandir(directory):
        if entry.is_file() and entry.name.endswith('.txt'):
            if 'ip_list' in entry.name:
                pt = directory + '/' + entry.name
                ip_files.append(pt)
    return ip_files

if __name__ == '__main__':
    for fname in scan_files():
        start = time.time()
        with open(fname) as file:
            park = [ip.strip() for ip in file] * 100
            executor = ThreadPoolExecutor(12)
            df = pd.DataFrame(executor.map(ping, park), columns=["address", "state"])
            df.index += 1
            end = time.time()
            time_used = end - start
            logger.info(f'{df}\n File <{fname}> used {time_used:.2f} sec\n')

Output:
2023-08-14 at 19:00:07
               address  state
1      python-forum.io      0
2          youtube.com      0
3        youtube.com99      1
4            www.vg.no      0
5    python-forum.io99      1
..                 ...    ...
496    python-forum.io      0
497        youtube.com      0
498      youtube.com99      1
499          www.vg.no      0
500  python-forum.io99      1

[500 rows x 2 columns]
 File <G:/div_code/egg/ping/ip_list1.txt> used 9.04 sec

2023-08-14 at 19:00:16
               address  state
1      python-forum.io      0
2          youtube.com      0
3        youtube.com99      1
4            www.vg.no      0
5    python-forum.io99      1
..                 ...    ...
496    python-forum.io      0
497        youtube.com      0
498      youtube.com99      1
499          www.vg.no      0
500  python-forum.io99      1

[500 rows x 2 columns]
 File <G:/div_code/egg/ping/ip_list2.txt> used 8.53 sec