Python Forum

I am looking for some implementation of a queue that would work much like the multiprocessing.Queue but between different machines. It should have the following properties:

• distributed, can connect processes on multiple machines via the network
  
• thread / multiprocessing safe, can be written to or read from by multiple processes on multiple machines
  
• !!! Limited size, can specify maximum size and will block writers when limit is reached
  
• Blocking read, should block when empty until a new element arrives, unblock and allow reading from a random process of several that may be reading and block
  

The implementation could be based on redis or memcached.
Is there a library that supports this?

(I original posted this in the general discussions forum by mistake but I am unable to remove it from there, how can that mistake get cleaned up?)

The library ZMQ, which has bindings to Python (pyzmq), is a message queue.
You can use: SUB, PUB, PUSH, PULL, REQ, REP
The Conncetion is handled in the background automatically.
For example a publisher can be a server or a client.

Asyncio is supported. If you don't use asyncio, the read-function blocks, until a
new message arrived. But you can still use NOBLOCK. You can use a Poller, but you don't have to.

I don't know the maximum size, but I use this for sensor-data in a Radar-System.
Currently i am working with a shape of (3, 256, 256, 2) with datatype uint16.
This data is sent every 100ms. No problems with it.
I work with this library very low level: sending multipart messages, where the first one is always the topic, followed by other segments with bytes. But there is also an abstraction for Python data type.

I use websockets and in the past, I got more problems with websockets (fragmentation, lost messages, blocking) as with ZMQ.
What you need to know: Never expose the ZMQ-Services to the internet. This is bad.
Companies in financial sectors are using zmq. One million messages per second is a normal workload in the financial sector.