Python Forum

So I'm running some code and trying to thread it. I'm getting se same propery_id/ object_id for each thread when I run it, which to me says the same thread is being used for my program.

I've tried
print(Threading.thread.native_id)
print(Threading.thread.name)
print(Threading.thread.ident)
print(threading.get_native_id())


all with the same result of an object id

Python docs says this:


https://docs.python.org/3/library/threading.html
threading.get_native_id()
Return the native integral Thread ID of the current thread assigned by the kernel. This is a non-negative integer. Its value may be used to uniquely identify this particular thread system-wide (until the thread terminates, after which the value may be recycled by the OS).



so given I am getting the same values, this is telling me no new separate threads are being used for each target



Here's the code below, it isn't long. The Hardware variable is a class I imported to run against the lines of the input file (namely the ip address). They represent Cisco images in a gns3 environment.


In the code below I put each Connection Handler of a device into a queue, get from the queue, and my target = Hardware(host, ip), daemon=True).start()

if I leave ip out
target = Hardware(host), daemon=True).start()

there's no effect except ther log dictionary doesn't get updated properly (it only gets logged with the last ip, not all the ips we ran through)

Hardware = HardwareMaintenance.hardware_handler
thread_pool = Queue()

class Connect:
   

    def __init__(self,):
        intake_file = open('ip_list.txt', 'r')
        self.json_data = [json.loads(line) for line in intake_file]
        pass
    def connect_parser(self, ):
        # max_threads = 20
        # executor = ThreadPoolExecutor(max_threads)
        for data in self.json_data:
            ip = data["ip"]


            #res.join()

            # port = data["port"]
            username = data["username"] if data["username"] else ""
            password = data["password"] if data["password"] else ""
            secret = data["secret"] if "secret" in data else False
            device_type = data["device_type"] if data["device_type"] else ""

            if data["header"] == "Netmiko":
                print("The variables being passed:  " + ip, username, password, device_type)
                ConnectHandler = netmiko.ConnectHandler(
                    device_type=device_type,
                    host=ip,
                    username=username,
                    password=password,
                    port=22,
                    secret=data["secret"] if "secret" in data else False)
                if ConnectHandler:
                    try:
                        ConnectHandler.enable()
                    except Exception as e:
                        print("Could not connect to {}".format(ip))


                thread_pool.put(ConnectHandler)
            host = thread_pool.get()

            res = threading.Thread(name = ip, target=Hardware(host, ip), daemon=True).start()
            print(threading.get_native_id())
            #thread_pool.task_done()










if __name__ == "__main__":
    from main import Connect
    Connector = Connect()
    print(time.perf_counter())
    Connector.connect_parser()
    # with ThreadPoolExecutor(max_workers=15) as exe:
    #     exe.submit(Connector.connect_parser(inlist))
    print(time.perf_counter())
    print(time.perf_counter())

Judging from this code does someone see something i don't? I've tried context managers as well with no imporvement in time (although I didn't verify if they were running new threads or same one, I'm assuming same)

I tried print(threading.current_thread())

and the output is 'Main Thread' with the same OID for each thread. This tells me I am not threading properly. Here's the code below.

class Connect:

    def __init__(self,):
        intake_file = open('ip_list.txt', 'r')
        self.json_data = [json.loads(line) for line in intake_file]
        pass
    def connect_parser(self, ):
        # max_threads = 20
        # executor = ThreadPoolExecutor(max_threads)
        for data in self.json_data:
            ip = data["ip"]


            #res.join()

            # port = data["port"]
            username = data["username"] if data["username"] else ""
            password = data["password"] if data["password"] else ""
            secret = data["secret"] if "secret" in data else False
            device_type = data["device_type"] if data["device_type"] else ""

            if data["header"] == "Netmiko":
                print("The variables being passed:  " + ip, username, password, device_type)
                ConnectHandler = netmiko.ConnectHandler(
                    device_type=device_type,
                    host=ip,
                    username=username,
                    password=password,
                    port=22,
                    secret=data["secret"] if "secret" in data else False)
                if ConnectHandler:
                    try:
                        ConnectHandler.enable()
                    except Exception as e:
                        print("Could not connect to {}".format(ip))


                res = threading.Thread(target=Hardware(ConnectHandler, ip), daemon=True)

            res.start()
            print(threading.local())
            print(threading.get_ident())
            print(threading.current_thread())
            res.join()

The Hardware(ConnectHandler,ip)

is just a hardware class running against each connection. In this case it seems to be doing it sequentially. I want to see if I can process the hardware class more than one at a time, per ConnectHandler

So I got it to work with different threads. I'm looking at the debugger as it is pulling ConnetHandler objects from the queue, and ip objects from Queue. It seems to be running a different thread on each on. However the execution time has improved by an average of 3 seconds. Here is the code, I'm not sure if anyone has suggestions on how to improve the threading.

class Connect:

    def __init__(self,):
        intake_file = open('ip_list.txt', 'r')
        self.json_data = [json.loads(line) for line in intake_file]
        pass
    def connect_parser(self, ):
        # max_threads = 20
        # executor = ThreadPoolExecutor(max_threads)

        for data in self.json_data:
            ip = data["ip"]
            ip_pool.put(ip)


            #res.join()

            # port = data["port"]
            username = data["username"] if data["username"] else ""
            password = data["password"] if data["password"] else ""
            secret = data["secret"] if "secret" in data else False
            device_type = data["device_type"] if data["device_type"] else ""

            if data["header"] == "Netmiko":
                print("The variables being passed:  " + ip, username, password, device_type)
                ConnectHandler = netmiko.ConnectHandler(
                    device_type=device_type,
                    host=ip,
                    username=username,
                    password=password,
                    port=22,
                    secret=data["secret"] if "secret" in data else False)
                if ConnectHandler:
                    try:
                        ConnectHandler.enable()
                    except Exception as e:
                        print("Could not connect to {}".format(ip))
                thread_pool.put(ConnectHandler)
                print(list(thread_pool.queue))

        looper()



def looper():
    while not thread_pool.empty():
        conn = thread_pool.get(block=False)
        host = ip_pool.get(block=False)
        res = threading.Thread(target=Hardware(conn, host), daemon=True)
        conn.disconnect()
        res.start()
        res.join()
        print(res.__repr__())
        print(res.native_id)
        print(res.__sizeof__())
        print(list(thread_pool.queue))
            #print(threading.local())
            #print(threading.get_ident())





    #thread_pool.task_done()










if __name__ == "__main__":
    from _Timer_tes import Connect
    Connector = Connect()
    print(time.perf_counter())
    Connector.connect_parser()

    print(time.perf_counter())
    print(time.perf_counter())