Python Forum
Unable to repopulate a queue in python 3
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Unable to repopulate a queue in python 3
#1
The code is getting exited when I run the following code
import mysql.connector
    import mysql.connector.pooling as x
    import time
    from threading import Thread
    from queue import Queue

    dbconfig = {
      "database": "mydb",
      "user":     "root",
      "password": "rootpass",
      "host":"127.0.0.1"
    }

    cnxpool =x.MySQLConnectionPool(pool_name = "mypool",pool_size = 2,**dbconfig)
    q = Queue(maxsize=20)

    threads = []
    limit = 0
    fetch_size = 20
    flag = True

    def populate():
        cnx = cnxpool.get_connection()
        cursor = cnx.cursor()
        global flag,fetch_size,limit
        while flag:
            cursor.execute("SELECT mycol from mytab limit " + str(limit)+","+ str(fetch_size))
            rows = cursor.fetchall()
            for row in rows:
                print("Putting "+ str(row[0]))
                q.put(row[0])
            if cursor.rowcount < 20 :    
                for i in range(4):
                    q.put(None)
                flag = False
            limit = limit + 20
        cursor.close()
        cnx.close()

    def consume():
        while True:
            item = q.get()
            if item is None:
                q.task_done()
                break
            time.sleep(1)
            print(item)
            q.task_done()

    for i in range(4):
        t = Thread(target=consume)
        t.start()
        threads.append(t)

    y = Thread(target=populate)
    y.start()

    q.join()
    y.join()

    for t in threads:
        t.join()

    print("Completed")
Code is getting completed in jupyter notebook

But when I try to repopulate the queue even after removing the variable it hangs.
        import mysql.connector
        import mysql.connector.pooling as x
        import time
        from threading import Thread
        from queue import Queue

        dbconfig = {
          "database": "mydb",
          "user":     "root",
          "password": "rootpass",
          "host":"127.0.0.1"
        }

        cnxpool =x.MySQLConnectionPool(pool_name = "mypool",pool_size = 2,**dbconfig)
        q = Queue(maxsize=20)

        threads = []
        limit = 0
        fetch_size = 20
        flag = True

        def populate():
            cnx = cnxpool.get_connection()
            cursor = cnx.cursor()
            global flag,fetch_size,limit
            while flag:
                cursor.execute("SELECT mycol from mytab limit " + str(limit)+","+ str(fetch_size))
                rows = cursor.fetchall()
                for row in rows:
                    print("Putting "+ str(row[0]))
                    q.put(row[0])
                if cursor.rowcount < 20 :    
                    for i in range(4):
                        q.put(None)
                    flag = False
                limit = limit + 20
            cursor.close()
            cnx.close()

        def consume():
            while True:
                item = q.get()
                if item is None:
                    q.task_done()
                    break
                time.sleep(1)
                print(item)
                q.task_done()

        for i in range(4):
            t = Thread(target=consume)
            t.start()
            threads.append(t)

        y = Thread(target=populate)
        y.start()

        q.join()
        y.join()

        for t in threads:
            t.join()

        print("Completed")

        del q

        q = Queue(maxsize=20)

        for i in range(4):
            t = Thread(target=consume)
            t.start()
            threads.append(t)

        y = Thread(target=populate)
        y.start()

        q.join()
        y.join()

        for t in threads:
            t.join()




        print("Completed")
It hangs for ever. Even creating a new queue doesn't help. Where am I going wrong. Kindly guide me.

Thanks in advance.

I am using Python 3.
Reply
#2
It is difficult to test your code (missing the DB) but I think the problem is that the Queue q that both methods populate and consume are seeing is the one they find when declared (the one at line 15) Once declared, it does not matter if you delete and declare again the variable q, it is a "new" variable and populate and consume does not known about it.

The 3 things I think I would try are:
  • Add q to a global declaration. In this way both consume and populate will look for the q variable in the upper namespace... but using global feels always dirty and in many occasions the result is a painful bug 3 days later.
  • Do not declare a new variable, empty the queue with something like:
    while not q.empty():
        try:
            q.get_nowait()
        except queue.Empty:
            break
  • modify populate and consume so they get the queue to work with from the argument list and pass the queue object with the args= of Thread.
I do not know which one works better -if any of them does-- but if I have to choose I prefer the 3rd one, I do not like the global variables...

I think the reason in jupyter works has to do with the fact that jupyter makes some dirty tricks with the global namespace to handle the different cells.
Reply
#3
I tried the approach told by you. It didn't help. If I try to populate again the same queue, it hangs.

(May-21-2018, 12:05 PM)killerrex Wrote: It is difficult to test your code (missing the DB) but I think the problem is that the Queue q that both methods populate and consume are seeing is the one they find when declared (the one at line 15) Once declared, it does not matter if you delete and declare again the variable q, it is a "new" variable and populate and consume does not known about it.

The 3 things I think I would try are:
  • Add q to a global declaration. In this way both consume and populate will look for the q variable in the upper namespace... but using global feels always dirty and in many occasions the result is a painful bug 3 days later.
  • Do not declare a new variable, empty the queue with something like:
    while not q.empty():
        try:
            q.get_nowait()
        except queue.Empty:
            break
  • modify populate and consume so they get the queue to work with from the argument list and pass the queue object with the args= of Thread.
I do not know which one works better -if any of them does-- but if I have to choose I prefer the 3rd one, I do not like the global variables...

I think the reason in jupyter works has to do with the fact that jupyter makes some dirty tricks with the global namespace to handle the different cells.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  queue for async function python telegram.ext noctious 0 1,516 Jun-11-2023, 02:58 PM
Last Post: noctious
  task queue Valon1981 8 3,518 Jul-07-2020, 07:41 AM
Last Post: freeman
  Unable to install python-wnck python-imaging on Void Linux linuxlight 1 3,155 Mar-08-2020, 03:53 AM
Last Post: Larz60+
  Queue in Pygame constantin01 1 3,640 Jan-07-2020, 04:02 PM
Last Post: metulburr
  Queue maxsize mr_byte31 2 4,501 Sep-03-2019, 07:02 PM
Last Post: mr_byte31
  Is there an iterable queue in python? AlekseyPython 3 15,143 Dec-05-2018, 02:19 PM
Last Post: AlekseyPython
  Queue.Queue() would not reduce capacity after get() yuan8421 9 10,964 Jan-02-2018, 09:38 PM
Last Post: Windspar
  Threading and Queue nexusfactor 5 4,221 Oct-16-2017, 04:14 PM
Last Post: Larz60+

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020