Sep-16-2020, 04:44 PM
I have a suggestion about the implementation of asyncio queues that could improve performance. I might be missing something, however. I am sort of new to Python. Below a short description of the problem I am facing.
I wrote a daemon in Python 3 (running in Linux) which test many devices at the same time, to be used in a factory environment. This daemon include multiple communication events to a back-end running in another country. I am using a class for each device I test, and embedded into the class I use asyncio. Due to the application itself and the number of devices tested simultaneously, I soon run out of file descriptor. Well, I increased the number of file descriptor in the application and then I started running into problems like “ValueError: filedescriptor out of range in select()”. I guess this problem is related to a package called serial_asyncio, and of course, that could be corrected. However I became curious about the number of open file descriptors opened: why so many?
Apparently asyncio Queues use a Linux pipe and each queue require 2 file descriptors. Am I correct? So I asked my self: if a asyncio queue is just a mechanism of piping information between two asyncio tasks, which should never run at the same time, why do I need the operating system in the middle of that? Isn’t the whole idea about asyncio that the operating system would be avoided whenever possible? No one will put anything into a queue if asyncio called epoll, because some Python code should be running to push things into the queue. If there is nothing in a particular queue, nothing will show up while asyncio is waiting for a file descriptor event. So, if I am correct, it would be more efficient to put the queue in a ready-queue list whenever something is pushed into it. Then, just before asyncio calls epoll (or select), it would check that ready queue, and it would process it before the epoll call. I mean that epoll would not be called unless all the queues have been properly processed. Queues would be implemented in a much simpler way, using local memory: a simple array may be enough to do the job. With that the OS would be avoided, and a much lower number of file descriptors would be necessary.
I wrote a daemon in Python 3 (running in Linux) which test many devices at the same time, to be used in a factory environment. This daemon include multiple communication events to a back-end running in another country. I am using a class for each device I test, and embedded into the class I use asyncio. Due to the application itself and the number of devices tested simultaneously, I soon run out of file descriptor. Well, I increased the number of file descriptor in the application and then I started running into problems like “ValueError: filedescriptor out of range in select()”. I guess this problem is related to a package called serial_asyncio, and of course, that could be corrected. However I became curious about the number of open file descriptors opened: why so many?
Apparently asyncio Queues use a Linux pipe and each queue require 2 file descriptors. Am I correct? So I asked my self: if a asyncio queue is just a mechanism of piping information between two asyncio tasks, which should never run at the same time, why do I need the operating system in the middle of that? Isn’t the whole idea about asyncio that the operating system would be avoided whenever possible? No one will put anything into a queue if asyncio called epoll, because some Python code should be running to push things into the queue. If there is nothing in a particular queue, nothing will show up while asyncio is waiting for a file descriptor event. So, if I am correct, it would be more efficient to put the queue in a ready-queue list whenever something is pushed into it. Then, just before asyncio calls epoll (or select), it would check that ready queue, and it would process it before the epoll call. I mean that epoll would not be called unless all the queues have been properly processed. Queues would be implemented in a much simpler way, using local memory: a simple array may be enough to do the job. With that the OS would be avoided, and a much lower number of file descriptors would be necessary.