Multithreading with ssh connection

[harusin]

Hello,

i need help with implementation Queue and maybe Lock in multithreading operation. Code below will print results i want, but i want to return it as object for further operations. if there is option to solve this with multiprocessing please how to apply it to this concrete code.

self.query_sort() ## Returns Dictionary with ip addresses scope {"9" : "192.168.9.0", "192.168.9.1", ....} it's about 400 IP's in 7 different scopes

self.list_result = []

def get_network_data(self, ixx):
        for ix in ixx:
                sshh = SshConnection(ix, self.username, self.password) # Connecting to SSH ix is ip address
            try:
                out1, err1 = sshh.ssh_command("ioreg -c IOPlatformExpertDevice "
                                              "-d 2 | awk '/IOPlatformSerialNumber/'")  # Get computer  serial number
                serial_number = out1[0].strip().split('"')
                out2, err2 = sshh.ssh_command('echo $HOSTNAME') # Get computer name
                station_name = out2[0].strip().split(".")[0]
                new_rec = ix + ":" + serial_number[3] + ":" + station_name
                print(new_rec) # returns "192.168.9.1:DXXXXXXXXXX:MY_NAME"
                self.list_result.append(new_rec) # This will return None because writing to shared memory
            except:
                pass


    def thread_queue(self):
        dict_all = self.query_sort()
        for x in dict_all.values():
            t = threading.Thread(target=self.get_network_data, args=(x,))
            t.start()

Thanks.

[Larz60+]

change

                self.list_result.append(new_rec) # This will return None because writing to shared memory

to

                return new_rec

and work with that data outside of the function
In addition, you should handle your exception

[harusin]

Hi again,

thank you for your swift answer, i have updated code with what you said and result is same as before. You can see i have used list to append new values in loop which differs from what you wrote but it makes no difference result is 'None'.

Any other ideas where i am making mistake?

    def get_network_data(self, ixx):
        list_result = []
        for ix in ixx:
            if ix.startswith("10.172.9."):
                sshh = SshConnection(ix, "apple", "apple")
            else:
                sshh = SshConnection(ix, self.username, self.password)
            try:
                out1, err1 = sshh.ssh_command("ioreg -c IOPlatformExpertDevice "
                                              "-d 2 | awk '/IOPlatformSerialNumber/'")
                serial_number = out1[0].strip().split('"')
                out2, err2 = sshh.ssh_command('echo $HOSTNAME')
                station_name = out2[0].strip().split(".")[0]
                new_rec = ix + ":" + serial_number[3] + ":" + station_name
                # print(new_rec)
                list_result.append(new_rec)
                return list_result
            except (ssh_exception.AuthenticationException,
                    ssh_exception.NoValidConnectionsError,
                    ssh_exception.SSHException,
                    socket.timeout) as err:
                pass


    def thread_queue(self):
        dict_all = self.query_sort()
        for x in dict_all.values():
            t = threading.Thread(target=self.get_network_data, args=(x,))
            t.start()


new_nmap = NmapScan()
list_res = new_nmap.thread_queue() # >>>> Returns None
print(list_res)
for item in list_res:
    print(item) # >>>> TypeError: 'NoneType' object is not iterable

[Larz60+]

you load target with the return result, but you don't do anything with it.
How you process the data after you get it returned is up to you

[harusin]

Is it possible to write example how should i return list out of that function? I am not sure i understand what you mean.

[Larz60+]

First of all, since I see self in the argument list for your methods include self, I assume there is a class that this is operating from
(you should provide full listings or at least full classes when posting).
So under the class __init__ method, add a line self.target = None
then modify:

t = threading.Thread(target=self.get_network_data, args=(x,))
# to 
t = threading.Thread(self.target=self.get_network_data, args=(x,))

your data is now in self.target to do with as you please

[harusin]

Thank you for your response i have resolved it, it took me long enough to see the issue.
One thing still bothers me and it's speed of ssh over paramiko taking too long, when i am sending two separate requests with 3 sec timeouts. 400 computers 77 sec (I don't know if it is bad but i guess it's no good.

def get_network_data(self, ixx, qq):
        """Checking computers for Serial number and name"""
        self.set_result = set()
        qq.put(ixx)
        for ix in ixx:
            if ix.startswith("10.172.9."):
                sshh = SshConnection(ix, "apple", "apple")
            else:
                sshh = SshConnection(ix, self.username, self.password)
            try:
                out1, err1 = sshh.ssh_command("ioreg -c IOPlatformExpertDevice "
                                              "-d 2 | awk '/IOPlatformSerialNumber/'")
                serial_number = out1[0].strip().split('"')
                out2, err2 = sshh.ssh_command('echo $HOSTNAME')
                station_name = out2[0].strip().split(".")[0]
                new_rec = ix + ":" + serial_number[3] + ":" + station_name
                # print(new_rec)
                self.set_result.add(new_rec)
            except (ssh_exception.AuthenticationException,
                    ssh_exception.NoValidConnectionsError,
                    ssh_exception.SSHException,
                    socket.timeout) as err:
                pass
        qq.task_done()



    def thread_queue(self):
        qq = Queue()
        dict_all = self.query_sort()
        for x in dict_all.values():
            t = threading.Thread(self.target=self.get_network_data, args=(x, qq))
            t.start()
        t.join()
        qq.join()
        return self.set_result


new_nmap = NmapScan()
list_res = new_nmap.thread_queue() # >>>> Returns None