Python Forum
Queue.Queue() would not reduce capacity after get()
Thread Rating:
  • 1 Vote(s) - 5 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Queue.Queue() would not reduce capacity after get()
#1
Hi Community,

I need your help about Queue memory.
1) I choose Queue as my data structure because I have one thread to feed data to the queue and another thread will take the data
2) The two threads designed to run for days
3) I don't want to limit the queue size, the size of queue could be really long, say ~10k which occupy 10GB. This is fine
4) Problem is when I finish shrink the q size by get() to only 20 items which occupy only ~100MB in memory. I print the size and I'm sure there is only 20 items.
5) But in system level, the whole process still occupy ~10GB

I tried to call
gc.collect()
by myself, memory doesn't change. So my wild guess is that those item from get() is destroyed.
And the thread is always running and python would not decrease capacity of the queue.

My question is: Is there anyway to free those memory which the queue doesn't use for now? I can't find any api to do that.
Reply


Messages In This Thread
Queue.Queue() would not reduce capacity after get() - by yuan8421 - Dec-28-2017, 11:16 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
Question [redistribution] Reduce number + size of dependencies? Winfried 2 633 Jan-31-2025, 10:17 PM
Last Post: snippsat
  queue for async function python telegram.ext noctious 0 3,168 Jun-11-2023, 02:58 PM
Last Post: noctious
  reduce nested for-loops Phaze90 11 5,038 Mar-16-2023, 06:28 PM
Last Post: ndc85430
  Adding values with reduce() function from the list of tuples kinimod 10 5,389 Jan-24-2023, 08:22 AM
Last Post: perfringo
  Priority Queue with update functionality PythonNewbee 4 4,883 Dec-29-2022, 12:13 PM
Last Post: Yoriz
  Multiprocessing queue catch get timeout Pythocodras 1 4,315 Apr-22-2022, 06:01 PM
Last Post: Pythocodras
  Asyncio: Queue consumer gets out of while loop without break. Where exactly and how? saavedra29 2 4,412 Feb-07-2022, 07:24 PM
Last Post: saavedra29
  Clearing an asyncio queue if it's empty H84Gabor 0 3,242 Jan-28-2022, 02:27 AM
Last Post: H84Gabor
  Error when using Watchdog and Qt to keep open a Queue before and after sending first pyhill00 0 2,407 Oct-28-2021, 09:10 AM
Last Post: pyhill00
  Package cannot be loaded into PBS queue file emersonpl 1 2,842 Sep-09-2021, 08:06 PM
Last Post: emersonpl

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020