Python Forum
Queuing the file content and run the thread.
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Queuing the file content and run the thread.
#1
Hello,
I have a list of network devices in a file (each line is a device). I am passing each device or each line of that file as an argument to a curl statement to get the interfaces from the devices. The problem is, since there are more than 20K devices - it is actually taking a lot of time to get all the interfaces. 
This led me into threading and queuing module. Now, I am queuing all the devices and then running the thread. The thread contains the function to pull the interfaces. 
with open("filtered_devices") as my_file:
   for line in my_file:
     q.put(line)

numofThreads = 10.
for i in range (numofThreads):
   print str(threading.activeCount())
   t = threading.Thread(target = queue_interfaces())
   t.start()
   #threadlist.append(t)
Here, queue_interfaces() is the function that passes each devices as as argument and gets the interfaces. Not sure if this is the correct way to do that. May be I am doing it wrong, since the time is still the same. Can somebody please advise?
Also, my active thread count is not incrementing.
Reply
#2
Are you joining on the queue?
Reply
#3
Yes, I am.

q.join()
Reply
#4
Can you provide enough code to reproduce the problem? You can reply the devices by curling Google or something. Also, I just noticed that the code you've posted passes a float to range() which is not valid. Don't take this personally, but oftentimes people post code that isn't terribly related to their problem, and it really helps us to get real runnable code. Otherwise it can be very difficult to help.
Reply
#5
I am sorry about that. Not sure how 10 became a floating - it was "10" when I posted. Here's my complete code. I have not included the url but you can assume it is an API call. Also, I have a list of devices in the file "filtered_devices".

import urllib
import urllib2
import requests
import threading
import Queue

#Queuing
q = Queue.LifoQueue()

url = 'x'


def queue_interfaces(q):    #grabs the queue and starts getting the interfaces from the list
    while not q.empty():
        headers = {"Content-Type": "application/json"}
        a = str(q.get()).strip('\n')
        data = {"data":{"Type":"Interfaces","Filters":{"device__name":a},"Attributes":["speed","status__name"],"page_size":10}}
        r = requests.post(url, headers=headers, json=data)
        #print (r.text)
        f = open("interfaceslistfinal", 'w')
        f.write(r.text)
        q.task_done()
    f.close()


with open("filtered_devices") as my_file:
    for line in my_file:
      q.put(line)

q.join()

#Threading
numofThreads = 10
#threadlist = []
for i in range (numofThreads):
    print i
    t = threading.Thread(target = queue_interfaces, args=(q,))
    t.setDaemon(True)
    t.start()
Reply
#6
I'm no expert on threading, but I'm fairly certain that at least one problem is that you need to join on the queue after starting the threads.

Another, more minor issue that probably isn't causing issues, is that f.close() should be inside your while-loop.

I don't have a chance at the moment, but I'll play with this code when I get a chance.
Reply
#7
Cool, Thanks micseydel. Despite joining the queue after starting the threads, it still doesn't do that good. The time is still the same. Yeah, f.close () is inside the while-loop. Please let me know when you get a chance to play around and find something that can help.
Reply
#8
Apologies for taking so long to get back, but I've been able to reproduce the issue very concisely
import Queue
q = Queue.LifoQueue()
q.put(None)
q.join()
This will hang. Here is the same thing with more prints
import Queue
q = Queue.LifoQueue()
print "Putting into the queue"
q.put(None)
print "About to join()"
q.join()
print "Done with join()"
Basically, the problem is q.join() will block until all the items in the queue have been completed. More specifically, the docs say that task_done() being called is required to resolve items, but when you join on the queue you haven't started any threads yet.

I haven't dug deeper into your code, but I'll let you integrate this for now. If you have followup questions I'm still happy to come back to this, though as you know that could take some time....
Reply
#9
Thanks a lot [b]micseydel ! [/b]BTW, I have already fixed my code. Sorry, that I didn't update the thread. [Part of my code:]

# Queuing
q = Queue.Queue(maxsize=0)
lock = threading.Lock()   # Lock the file while writing for 1 thread

#Threading
with open("filtered_devices") as my_file:
    for line in my_file:
      q.put(line)

numofThreads = 20

for i in range (numofThreads):
    print "Threading Count No:%d" %i
    t = threading.Thread(target = queue_interfaces, args=(q,))
    t.setDaemon(True)
    t.start()

q.join()
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Queuing Theory in Python NOLA_Saints 0 749 Feb-22-2023, 11:42 PM
Last Post: NOLA_Saints
  Extracting Specific Lines from text file based on content. jokerfmj 8 2,861 Mar-28-2022, 03:38 PM
Last Post: snippsat
Thumbs Up Parsing a YAML file without changing the string content..?, Flask - solved. SpongeB0B 2 2,222 Aug-05-2021, 08:02 AM
Last Post: SpongeB0B
  Opening file and outputting its text content example leodavinci1990 1 1,849 Oct-12-2020, 05:33 AM
Last Post: buran
  how to view file content of .docx while in Recycling bin NeoPy1 0 1,383 Sep-16-2020, 10:11 PM
Last Post: NeoPy1
  copy content of text file with three delimiter into excel sheet vinaykumar 0 2,304 Jul-12-2020, 01:27 PM
Last Post: vinaykumar
  Read content of GitHub file sridhar 5 9,527 Apr-29-2020, 11:49 PM
Last Post: sridhar
  Error SQLite objects created in a thread can only be used in that same thread. binhduonggttn 3 15,391 Jan-31-2020, 11:08 AM
Last Post: DeaD_EyE
  How do I read the HTML files in a directory and write the content into a CSV file? glittergirl 1 2,559 Sep-23-2019, 11:01 AM
Last Post: Larz60+
  save content of table into file atlass218 10 9,774 Aug-28-2019, 12:12 PM
Last Post: Gribouillis

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020