Python Forum
Receiving this error in my "response" and causes script to return wrong status
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Receiving this error in my "response" and causes script to return wrong status
#1
I have a script which i thought was working correctly until i purposely added some invalid IP's into my list so that i can validate the results file.
The "response" value is always this = Access denied. Option -c requires administrative privileges. and never has the true ping response. So now i have a solution that i thought was working and in fact not really working as all results returned are "Up" based on the condition.

So i have 2 questions:
#1 - what can be done to prevent that error, since i do not have admin rights on my work pc.
#2 - If there is no way around that error based on the script code below, is there any other method of pinging over 4000 IP's in a timely manner?

This script run against all 4000 ip's in under 4 minutes.. which currently is ok processing time.. IF it can be done more efficiently I'm all for improvements. I have another post about trying the icmp method, but cant seem to get that working and dont want to mess with the below yet.

The above message is returned in the "response" but still allows the script to run and return results.
response = os.popen(f"ping -c 1 {ip} ").read() using this method returns the message above and takes about 4 minutes to run thru 4000 ip's, but returns false positives.
response = os.popen(f"ping /n 1 {ip} ").read() using this method no message and takes about 21 minutes to run thru 4000 ip's but this time returns valid results.



import os
import time
from datetime import datetime

now = datetime.now() # current date and time
date_time = now.strftime("%m/%d/%Y,%H:%M:%S")

start = time.time()

# DIRECTORY WHERE ALL IP LISTS RESIDE
directory = 'Lists'

# RETURN LIST OF ALL FILES IN THE "DIRECTORY" PROVIDED ABOVE.
for entry in os.scandir(directory):
    
    # THIS WILL LOOP THROUGH ALL TEXT FILES IN DIRECTORY
    if entry.is_file() and entry.name.endswith('.txt'):
        
        # THIS WILL ONLY LOOK FOR AND RETURN FILES WITH IP_LIST IN THE NAME SINCE THOSE ARE THE ONES WITH THE SPECIFIC IP'S
        if 'ip_list' in entry.name:
            
            # FILE PATH FOR EACH FILE TO PROCESS(EX. Lists/Register_ip_list.txt)
            pt = directory + '/' + entry.name
            
            # THIS WILL PARSE OUT THE VALUE BEFORE THE FIRST '_' TO SET THE VARIABLE AND USE LATER FOR SAVING THE RESULTS.
            parts = entry.name.split('_')
            
            # OPEN EACH TEXT FILE THAT CONTAINS IPS TO PROCESS
            with open(pt) as file:
                park = file.read()
                park = park.splitlines()
                #print(" {park}  \n")
                
                # THIS IS TO CLEAR THE FILE BEFORE WRITING TO IT(WONT NEED THIS ONCE WE START INSERTING THE DATE INTO A DATABASE)
                file_to_clear = open("Lists/" + parts[0] +"_ip_output.txt",'w')
                file_to_clear.close()
                
            # PING EACH IP IN FILE LIST
            for ip in park:
                response = os.popen(f"ping -c 1 {ip} ").read()
                
                # SAVE PING STATUS INTO FILE FOR DASHBOARD (IP,STATUS,DATETIME)
                if("Request timed out." or "unreachable") in response:
                        #print(response)
                        f = open("Lists/" + parts[0] +"_ip_output.txt","a")
                        f.write(str(ip) + ',down,'+ date_time +'\n')
                        f.close() 
                else:
                        #print(response)
                        f = open("Lists/" + parts[0] +"_ip_output.txt","a")  
                        f.write(str(ip) + ',up,'+ date_time +'\n')
                        f.close() 
            # PRINT COMPLETE RESULTS TO CONSOLE WINDOW (DEBUGGING PURPOSES)
            # with open("Lists/" + parts[0] +"_ip_output.txt") as file:
            #     output = file.read()
            #     f.close()
            #     print(output)
            # THIS CLEARS THE FILE AFTER ITS WRITTEN TO THE CONSOLE
            #with open("Lists/ip_output.txt","w") as file:    
            #                pass

# CALCULATED THE RUNTIME OF THE SCRIPT (FUTURE UPDATES TO INCLUDE CAPTURING RUNTIMES IN SEPARATE FILE FOR REVIEW, TO INCLUDE FILENAME, ROW COUNT AND DURATION)
end = time.time()
print(end - start)
Reply
#2
os.popen() is depreciated. Use subprocess.run(). I think os.popen actually uses subprocess.run(), so skip the middleman and your program will continue to run when os.popen has been retired.

subprocess.run('func').returncode gets the returncode from running func. For ping this will be 0 if successful, else some error code. This code pings multiple IPs at the same time using the ThreadPoolExecutor.
import subprocess
from concurrent.futures import ThreadPoolExecutor


ips = ["10.0.0.5", "127.0.0.1", "::1"]


def ping(ip):
    return (
        ip,
        subprocess.run(
            f"ping {ip} -n 1", stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL
        ).returncode,
    )


executor = ThreadPoolExecutor(4)
print(*executor.map(ping, ips))
Reply
#3
ok so im trying to replace your hardcoded list of IP's with my text file. Since my logic loops thru each ip to ping, can i still use my file and if i can, can it be used with formatting the IP's different than this: (my text file has 1 ip per line with no commas or quotes.
192.28.9.6
192.28.9.7
192.28.9.8
192.28.9.9
192.28.9.10
192.28.9.11
192.28.9.12
192.28.9.13
Reply
#4
You need an iterable. Any iterable will do.
Reply
#5
Man oh man..
im so confused, but getting thru this.. i had one incorrect line, but now works based on list of my ips. But i cant seen to add a if condition on the results so that i can capture the status in my output file.

How can i get access to the data returned by this *executor.map(ping, ips) so i can capture the ip, status in my output?

i tried assigning it to a variable, but didnt like that.
Based on my original code above, i need to capture the ip, status, date, time into a text file to update our spreadsheet.

If the timestamp is correct, i just ran your suggested code against 4000 ips and returned just under 1 minute.. THATS AMAZING!!!
now just need to write that results to a file so i can consume into excel.
Reply
#6
I think i figured out how to get it into a output i can work with
changed the lines to this:

    park = file.readlines()
    
    #executor = ThreadPoolExecutor(4)
    with ThreadPoolExecutor(4) as executor:
        for result in executor.map(ping, park):
            print(result)
But still cant get access to do a IF condition to check for the 0 or 1
Reply
#7
The thing returned by "executor.map(ping, park)" is a lazy iterator. executor.map() returns immediately, but if you want the values, it will make you wait until they are available. You can do this as you did in your last post, by iterating through the results. Or you could use something like list() to stuff them all in a list.

I am glad it is faster. Try increasing the number of threads. Results will eventually plateau.

I don't understand the writing problem. The 0, 1 is available right there in the result. Maybe you will find this helpful.
import subprocess
from concurrent.futures import ThreadPoolExecutor


ips = ["10.0.0.5", "127.0.0.1", "::1"]


def ping(ip):
    return (
        ip,
        subprocess.run(
            f"ping {ip} -n 1", stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL
        ).returncode,
    )


executor = ThreadPoolExecutor(4)
for result in executor.map(ping, ips):
    print(result[0], "is down" if result[1] else "is up")
\
Output:
10.0.0.5 is down 127.0.0.1 is up ::1 is up
Or you could stuff the results into a dataframe.
import subprocess
from concurrent.futures import ThreadPoolExecutor
import pandas as pd

ips = ["10.0.0.5", "127.0.0.1", "::1"]


def ping(ip):
    return (
        ip,
        subprocess.run(
            f"ping {ip} -n 1", stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL
        ).returncode,
    )


executor = ThreadPoolExecutor(4)
df = pd.DataFrame(executor.map(ping, ips), columns=["address", "state"])
print(df)
Output:
address state 0 10.0.0.5 1 1 127.0.0.1 0 2 ::1 0
And pandas has a function for writing the dataframe as an excel format file.
Reply
#8
So im trying the last suggestions and having some odd results in the output.. As seen below, that is the format the results are output.
#1 - not sure why or where the newline character is coming from in the dataframe
#2 - not sure why all except for the last site is considered down.. No matter how many sites i have on my list to process, the last one is always UP an everything before it is down.
#

Here is what i see when i write it to a dataframe:

            address  state
0  www.google.com\n      1
1   www.abc13.com\n      1
2   www.yahoo.com\n      1
3     www.cnn.com\n      1
4   www.walmart.com      0
0.17187809944152832
This is what i get when i use the example before dataframe:

www.google.com
,down
www.abc13.com
,down
www.yahoo.com
,down
www.cnn.com
,down
www.walmart.com,up
Here is the code used to produce the newline for each row in the 2nd example:

import time
import subprocess
from concurrent.futures import ThreadPoolExecutor
import pandas as pd

def ping(ip):
    return (
        ip,
        subprocess.run(
            f"ping {ip} -n 1", stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL
        ).returncode,
    )

start = time.time()

with open("ip_list.txt") as file:
    park = file.readlines()
    with ThreadPoolExecutor(4) as executor:
        for result in executor.map(ping, park):
            if result[1] == 0:
           	    f = open("ip_output.txt","a")
           	    f.write(str(result[0]) + ',up' + '\n')
           	    f.close() 
            else:
           	    f = open("ip_output.txt","a")
           	    f.write(str(result[0] + ',down' +'\n'))
           	    f.close()   
This is the dataframe code:

import time
import subprocess
from concurrent.futures import ThreadPoolExecutor
import pandas as pd

def ping(ip):
    return (
        ip,
        subprocess.run(
            f"ping {ip} -n 1", stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL
        ).returncode,
    )

start = time.time()

with open("ip_list.txt") as file:
    park = file.readlines()
 
    executor = ThreadPoolExecutor(4)
    df = pd.DataFrame(executor.map(ping, park), columns=["address", "state"])
    print(df)
If i manually ping the above sites from the command prompt, they are all up and online.
Reply
#9
I bet google.com\n is not responding to your ping
Reply
#10
Little more testing

I changed to using the dataframe and the file it generates is not formatted correctly.

with open("ip_list.txt") as file:
    park = file.readlines()
 
    executor = ThreadPoolExecutor(5)
    df = pd.DataFrame(executor.map(ping, park), columns=['address', 'state'])
    df.to_csv(r'ip_output.csv',header=False, index=False) 
The output from the above is this, why is there " around all the sites except the last one and why are they all "Down" except the last one?
"www.google.com
",1
"www.abc13.com
",1
"www.yahoo.com
",1
"www.cnn.com
",1
www.walmart.com,0
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Wrong type error rowan_bradley 6 1,232 Aug-07-2023, 10:44 AM
Last Post: rowan_bradley
  Compiles Python code with no error but giving out no output - what's wrong with it? pythonflea 6 1,583 Mar-27-2023, 07:38 AM
Last Post: buran
  Am I wrong or is Udemy wrong? String Slicing! Mavoz 3 2,575 Nov-05-2022, 11:33 AM
Last Post: Mavoz
  SMA (simple moving avg) Not receiving Data (stock prices). gdbengo 2 1,467 Jul-31-2022, 08:20 PM
Last Post: paulyan
  Receiving snmp traps with more than one Community String ilknurg 0 2,232 Jan-19-2022, 09:02 AM
Last Post: ilknurg
  return out of loops syntax error felixf 7 3,462 Nov-03-2020, 01:00 PM
Last Post: perfringo
  ERROR: Command errored out with exit status 1 calesii 3 7,193 Oct-23-2020, 05:39 PM
Last Post: snippsat
  SystemError: error return without exception set!!! faryad13 3 3,700 Oct-23-2020, 02:32 PM
Last Post: ATARI_LIVE
  Coding error- Not sure where I have put error markers against the code that is wrong Username9 1 1,739 Sep-28-2020, 07:57 AM
Last Post: buran
  Empty response to request causing .json() to error t4keheart 1 10,088 Jun-26-2020, 08:35 PM
Last Post: bowlofred

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020