Python Forum
Receiving this error in my "response" and causes script to return wrong status - Printable Version

+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: General Coding Help (https://python-forum.io/forum-8.html)
+--- Thread: Receiving this error in my "response" and causes script to return wrong status (/thread-40525.html)

Pages: 1 2


Receiving this error in my "response" and causes script to return wrong status - cubangt - Aug-11-2023

I have a script which i thought was working correctly until i purposely added some invalid IP's into my list so that i can validate the results file.
The "response" value is always this = Access denied. Option -c requires administrative privileges. and never has the true ping response. So now i have a solution that i thought was working and in fact not really working as all results returned are "Up" based on the condition.

So i have 2 questions:
#1 - what can be done to prevent that error, since i do not have admin rights on my work pc.
#2 - If there is no way around that error based on the script code below, is there any other method of pinging over 4000 IP's in a timely manner?

This script run against all 4000 ip's in under 4 minutes.. which currently is ok processing time.. IF it can be done more efficiently I'm all for improvements. I have another post about trying the icmp method, but cant seem to get that working and dont want to mess with the below yet.

The above message is returned in the "response" but still allows the script to run and return results.
response = os.popen(f"ping -c 1 {ip} ").read() using this method returns the message above and takes about 4 minutes to run thru 4000 ip's, but returns false positives.
response = os.popen(f"ping /n 1 {ip} ").read() using this method no message and takes about 21 minutes to run thru 4000 ip's but this time returns valid results.



import os
import time
from datetime import datetime

now = datetime.now() # current date and time
date_time = now.strftime("%m/%d/%Y,%H:%M:%S")

start = time.time()

# DIRECTORY WHERE ALL IP LISTS RESIDE
directory = 'Lists'

# RETURN LIST OF ALL FILES IN THE "DIRECTORY" PROVIDED ABOVE.
for entry in os.scandir(directory):
    
    # THIS WILL LOOP THROUGH ALL TEXT FILES IN DIRECTORY
    if entry.is_file() and entry.name.endswith('.txt'):
        
        # THIS WILL ONLY LOOK FOR AND RETURN FILES WITH IP_LIST IN THE NAME SINCE THOSE ARE THE ONES WITH THE SPECIFIC IP'S
        if 'ip_list' in entry.name:
            
            # FILE PATH FOR EACH FILE TO PROCESS(EX. Lists/Register_ip_list.txt)
            pt = directory + '/' + entry.name
            
            # THIS WILL PARSE OUT THE VALUE BEFORE THE FIRST '_' TO SET THE VARIABLE AND USE LATER FOR SAVING THE RESULTS.
            parts = entry.name.split('_')
            
            # OPEN EACH TEXT FILE THAT CONTAINS IPS TO PROCESS
            with open(pt) as file:
                park = file.read()
                park = park.splitlines()
                #print(" {park}  \n")
                
                # THIS IS TO CLEAR THE FILE BEFORE WRITING TO IT(WONT NEED THIS ONCE WE START INSERTING THE DATE INTO A DATABASE)
                file_to_clear = open("Lists/" + parts[0] +"_ip_output.txt",'w')
                file_to_clear.close()
                
            # PING EACH IP IN FILE LIST
            for ip in park:
                response = os.popen(f"ping -c 1 {ip} ").read()
                
                # SAVE PING STATUS INTO FILE FOR DASHBOARD (IP,STATUS,DATETIME)
                if("Request timed out." or "unreachable") in response:
                        #print(response)
                        f = open("Lists/" + parts[0] +"_ip_output.txt","a")
                        f.write(str(ip) + ',down,'+ date_time +'\n')
                        f.close() 
                else:
                        #print(response)
                        f = open("Lists/" + parts[0] +"_ip_output.txt","a")  
                        f.write(str(ip) + ',up,'+ date_time +'\n')
                        f.close() 
            # PRINT COMPLETE RESULTS TO CONSOLE WINDOW (DEBUGGING PURPOSES)
            # with open("Lists/" + parts[0] +"_ip_output.txt") as file:
            #     output = file.read()
            #     f.close()
            #     print(output)
            # THIS CLEARS THE FILE AFTER ITS WRITTEN TO THE CONSOLE
            #with open("Lists/ip_output.txt","w") as file:    
            #                pass

# CALCULATED THE RUNTIME OF THE SCRIPT (FUTURE UPDATES TO INCLUDE CAPTURING RUNTIMES IN SEPARATE FILE FOR REVIEW, TO INCLUDE FILENAME, ROW COUNT AND DURATION)
end = time.time()
print(end - start)



RE: Receiving this error in my "response" and causes script to return wrong status - deanhystad - Aug-11-2023

os.popen() is depreciated. Use subprocess.run(). I think os.popen actually uses subprocess.run(), so skip the middleman and your program will continue to run when os.popen has been retired.

subprocess.run('func').returncode gets the returncode from running func. For ping this will be 0 if successful, else some error code. This code pings multiple IPs at the same time using the ThreadPoolExecutor.
import subprocess
from concurrent.futures import ThreadPoolExecutor


ips = ["10.0.0.5", "127.0.0.1", "::1"]


def ping(ip):
    return (
        ip,
        subprocess.run(
            f"ping {ip} -n 1", stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL
        ).returncode,
    )


executor = ThreadPoolExecutor(4)
print(*executor.map(ping, ips))



RE: Receiving this error in my "response" and causes script to return wrong status - cubangt - Aug-11-2023

ok so im trying to replace your hardcoded list of IP's with my text file. Since my logic loops thru each ip to ping, can i still use my file and if i can, can it be used with formatting the IP's different than this: (my text file has 1 ip per line with no commas or quotes.
192.28.9.6
192.28.9.7
192.28.9.8
192.28.9.9
192.28.9.10
192.28.9.11
192.28.9.12
192.28.9.13


RE: Receiving this error in my "response" and causes script to return wrong status - deanhystad - Aug-11-2023

You need an iterable. Any iterable will do.


RE: Receiving this error in my "response" and causes script to return wrong status - cubangt - Aug-11-2023

Man oh man..
im so confused, but getting thru this.. i had one incorrect line, but now works based on list of my ips. But i cant seen to add a if condition on the results so that i can capture the status in my output file.

How can i get access to the data returned by this *executor.map(ping, ips) so i can capture the ip, status in my output?

i tried assigning it to a variable, but didnt like that.
Based on my original code above, i need to capture the ip, status, date, time into a text file to update our spreadsheet.

If the timestamp is correct, i just ran your suggested code against 4000 ips and returned just under 1 minute.. THATS AMAZING!!!
now just need to write that results to a file so i can consume into excel.


RE: Receiving this error in my "response" and causes script to return wrong status - cubangt - Aug-11-2023

I think i figured out how to get it into a output i can work with
changed the lines to this:

    park = file.readlines()
    
    #executor = ThreadPoolExecutor(4)
    with ThreadPoolExecutor(4) as executor:
        for result in executor.map(ping, park):
            print(result)
But still cant get access to do a IF condition to check for the 0 or 1


RE: Receiving this error in my "response" and causes script to return wrong status - deanhystad - Aug-11-2023

The thing returned by "executor.map(ping, park)" is a lazy iterator. executor.map() returns immediately, but if you want the values, it will make you wait until they are available. You can do this as you did in your last post, by iterating through the results. Or you could use something like list() to stuff them all in a list.

I am glad it is faster. Try increasing the number of threads. Results will eventually plateau.

I don't understand the writing problem. The 0, 1 is available right there in the result. Maybe you will find this helpful.
import subprocess
from concurrent.futures import ThreadPoolExecutor


ips = ["10.0.0.5", "127.0.0.1", "::1"]


def ping(ip):
    return (
        ip,
        subprocess.run(
            f"ping {ip} -n 1", stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL
        ).returncode,
    )


executor = ThreadPoolExecutor(4)
for result in executor.map(ping, ips):
    print(result[0], "is down" if result[1] else "is up")
\
Output:
10.0.0.5 is down 127.0.0.1 is up ::1 is up
Or you could stuff the results into a dataframe.
import subprocess
from concurrent.futures import ThreadPoolExecutor
import pandas as pd

ips = ["10.0.0.5", "127.0.0.1", "::1"]


def ping(ip):
    return (
        ip,
        subprocess.run(
            f"ping {ip} -n 1", stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL
        ).returncode,
    )


executor = ThreadPoolExecutor(4)
df = pd.DataFrame(executor.map(ping, ips), columns=["address", "state"])
print(df)
Output:
address state 0 10.0.0.5 1 1 127.0.0.1 0 2 ::1 0
And pandas has a function for writing the dataframe as an excel format file.


RE: Receiving this error in my "response" and causes script to return wrong status - cubangt - Aug-12-2023

So im trying the last suggestions and having some odd results in the output.. As seen below, that is the format the results are output.
#1 - not sure why or where the newline character is coming from in the dataframe
#2 - not sure why all except for the last site is considered down.. No matter how many sites i have on my list to process, the last one is always UP an everything before it is down.
#

Here is what i see when i write it to a dataframe:

            address  state
0  www.google.com\n      1
1   www.abc13.com\n      1
2   www.yahoo.com\n      1
3     www.cnn.com\n      1
4   www.walmart.com      0
0.17187809944152832
This is what i get when i use the example before dataframe:

www.google.com
,down
www.abc13.com
,down
www.yahoo.com
,down
www.cnn.com
,down
www.walmart.com,up
Here is the code used to produce the newline for each row in the 2nd example:

import time
import subprocess
from concurrent.futures import ThreadPoolExecutor
import pandas as pd

def ping(ip):
    return (
        ip,
        subprocess.run(
            f"ping {ip} -n 1", stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL
        ).returncode,
    )

start = time.time()

with open("ip_list.txt") as file:
    park = file.readlines()
    with ThreadPoolExecutor(4) as executor:
        for result in executor.map(ping, park):
            if result[1] == 0:
           	    f = open("ip_output.txt","a")
           	    f.write(str(result[0]) + ',up' + '\n')
           	    f.close() 
            else:
           	    f = open("ip_output.txt","a")
           	    f.write(str(result[0] + ',down' +'\n'))
           	    f.close()   
This is the dataframe code:

import time
import subprocess
from concurrent.futures import ThreadPoolExecutor
import pandas as pd

def ping(ip):
    return (
        ip,
        subprocess.run(
            f"ping {ip} -n 1", stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL
        ).returncode,
    )

start = time.time()

with open("ip_list.txt") as file:
    park = file.readlines()
 
    executor = ThreadPoolExecutor(4)
    df = pd.DataFrame(executor.map(ping, park), columns=["address", "state"])
    print(df)
If i manually ping the above sites from the command prompt, they are all up and online.


RE: Receiving this error in my "response" and causes script to return wrong status - deanhystad - Aug-12-2023

I bet google.com\n is not responding to your ping


RE: Receiving this error in my "response" and causes script to return wrong status - cubangt - Aug-12-2023

Little more testing

I changed to using the dataframe and the file it generates is not formatted correctly.

with open("ip_list.txt") as file:
    park = file.readlines()
 
    executor = ThreadPoolExecutor(5)
    df = pd.DataFrame(executor.map(ping, park), columns=['address', 'state'])
    df.to_csv(r'ip_output.csv',header=False, index=False) 
The output from the above is this, why is there " around all the sites except the last one and why are they all "Down" except the last one?
"www.google.com
",1
"www.abc13.com
",1
"www.yahoo.com
",1
"www.cnn.com
",1
www.walmart.com,0