Apr-12-2023, 03:36 PM
(This post was last modified: Apr-14-2023, 02:04 AM by Larz60+.
Edit Reason: fixed error tags
)
Hi,
I have a quite complex multi-threaded python script on a production environment (Debian 11.6 server) which regularly crashes using python3.10.10.
Sometimes it crashes with
Script architecture
The architecture of the script is the following one :
- A main thread which is used to control sub-threads (i.e. start, join, relaunch, exception handling, etc...)
This main thread initially launches a thread (let's call it the research thread) and waits for it to end, if it ends without any exception the main thread relaunch it indefinitely.
- The research thread makes calls to several APIs to get data and when the conditions are met it will launch a new thread (let's call it the ongoing thread)
Before launching a new ongoing thread, the research thread will setup several things for it. Firstly it will prepare the associated data for the thread, it will subscribe to websocket channels associated to the data, and it will setup queues for the ongoing thread to receive async event.
Once all this has been setup, the new ongoing thread is created and launched and the research thread continue to search for other conditions to be met.
There may be many parallel ongoing thread at the same time
- The ongoing thread just waits for an event to arrive on one of its queues to treat it, and depending on the event and the associated data, it will either exit or continue to receive events through queues.
- The main thread also manages ongoing threads execution, if one of them fails with an exception, it stops all the other threads, cleanup and exit.
Code presentation
I won't put the whole code wince there is a lot of it and it's quite specific to the application but I'll present some components in order to make things clearer.
I'm using a custom thread class in order to be able to detect if an exception occurred during the thread execution, this class also allows to stop the thread using an event, finally, this class also allows to retrieve a return code when joining a terminated thread
Here is a minimal pseudo-code example representing the architecture of the script into a Tester class
Debugging the crashes
Since the script crashes regularly, in order to understand what was happening, I recompiled python 3.10.10 from sources with debug support to be able to debug it with gdb.
The use of the file python3.10-gdb.py is just to be able to use gdb python extensions.
In order to analyze coredumps I do as follow:
Here are several backtrace I had from several coredumps
Since it seems to be memory related, I began to monitor the memory usage of the server and the process.
I have 8Gb available on the server and I'm always bellow 4Gb in global use
Even if the memory used by the process is slowly increasing, it's still acceptable (rarelly more than 200Mb), furthermore I have other python processes which consumes way more (For example I have one process which consumes up to 800Mb and runs for several days without never crashing)
Concluding
I don't know how to go further now, since the error doesn't seems to be consistent at all. This is the only script between many that crashes and even if it has a more complex architecture, it uses the same components.
I'm not really used to gdb with python so I don't fully understand the given backtrace and thus I don't understand where the error could come from. I do feel like this is related to a memory issue, but I can't go further into debugging and I'm beginning to lose faith.
That's why I'm coming here to see if someone would be able to help me to go through this issue.
If needed I can provide more gdb backtraces, make more runs with different python versions, etc...
Any help will be greatly appreciated.
Thanks in advance
I have a quite complex multi-threaded python script on a production environment (Debian 11.6 server) which regularly crashes using python3.10.10.
Sometimes it crashes with
malloc(): unsorted double linked list corrupted sometimes with Segmentation fault and I'm unable to understand what's happening into this script.Script architecture
The architecture of the script is the following one :
- A main thread which is used to control sub-threads (i.e. start, join, relaunch, exception handling, etc...)
This main thread initially launches a thread (let's call it the research thread) and waits for it to end, if it ends without any exception the main thread relaunch it indefinitely.
- The research thread makes calls to several APIs to get data and when the conditions are met it will launch a new thread (let's call it the ongoing thread)
Before launching a new ongoing thread, the research thread will setup several things for it. Firstly it will prepare the associated data for the thread, it will subscribe to websocket channels associated to the data, and it will setup queues for the ongoing thread to receive async event.
Once all this has been setup, the new ongoing thread is created and launched and the research thread continue to search for other conditions to be met.
There may be many parallel ongoing thread at the same time
- The ongoing thread just waits for an event to arrive on one of its queues to treat it, and depending on the event and the associated data, it will either exit or continue to receive events through queues.
- The main thread also manages ongoing threads execution, if one of them fails with an exception, it stops all the other threads, cleanup and exit.
Code presentation
I won't put the whole code wince there is a lot of it and it's quite specific to the application but I'll present some components in order to make things clearer.
- Thread class
I'm using a custom thread class in order to be able to detect if an exception occurred during the thread execution, this class also allows to stop the thread using an event, finally, this class also allows to retrieve a return code when joining a terminated thread
class ExceptionThread(threading.Thread):
def __init__(self, *args, **kwargs):
super(ExceptionThread, self).__init__(*args, **kwargs)
self.__stop_event = threading.Event()
def run(self):
self.exc = None
self.ret = None
try:
self.ret = self._target(*self._args, **self._kwargs)
except BaseException as e:
self.exc = e
self.stop()
def join(self, timeout=None):
threading.Thread.join(self, timeout)
if self.exc:
raise self.exc
return self.ret
def stop(self):
self.__stop_event.set()
def is_stopped(self):
return self.__stop_event.is_set()- Minimal example presenting the architecture
Here is a minimal pseudo-code example representing the architecture of the script into a Tester class
class Tester:
def __init__(self):
self.research_thread = None
self.ongoing_threads = {}
self.data_mutex = multiprocessing.Lock()
self.ongoing_threads_number = 0
self.exception_occured = False
self.event_timeout = 0.1
def run(self):
thread_name = f"[Thread-{threading.current_thread().name}]"
self.research_thread = ExceptionThread(name="research", target=self.research)
self.research_thread.start()
while True:
if self.research_thread.is_stopped():
try:
self.research_thread.join()
log(f"{thread_name} - Relaunch research thread")
self.research_thread = ExceptionThread(name="research", target=self.research)
self.research_thread.start()
except Exception as e:
log(f"{thread_name} - [ERROR] Exception in Thread-research : {traceback.format_exc()}")
self.exception_occured = True
log(f"{thread_name} - Join all ongoing_threads.")
with self.data_mutex:
for ongoing_thread in self.ongoing_threads.values():
ongoing_thread["thread"].stop()
ongoing_thread["thread"].join()
else:
with self.data_mutex:
for identifier, ongoing_thread in self.ongoing_threads.copy().items():
if ongoing_thread["thread"].is_stopped():
ongoing_thread_name = ongoing_thread["thread"].name
log(f"{thread_name} - Thread {ongoing_thread_name} stopped. Joining it.")
try:
ongoing_thread["thread"].join()
log(f"{thread_name} - Thread {ongoing_thread_name} joined.")
except Exception as e:
log(f"{thread_name} - [ERROR] Exception in Thread {ongoing_thread_name} : {traceback.format_exc()}")
self.exception_occured = True
self.research_thread.stop()
log(f"{thread_name} - Cleanup {ongoing_thread_name} data.")
for queue in self.ongoing_threads[identifier]["queues"].values():
queue.close()
queue.join_thread()
del self.ongoing_threads[identifier]
self.ongoing_threads_number -= 1
if self.exception_occured:
os._exit(1)
time.sleep(0.1)
def research(self):
thread_name = f"[Thread-{threading.current_thread().name}]"
log(f"{thread_name} - Start thread")
for data_set in self.data_sets:
identifier = self.has_opportunity(data_set)
if identifier:
# Subscribe websocket channels with callbacks
websocket_connector.subscribe_event_1(identifier, websocket_event_1_callback)
websocket_connector.subscribe_event_2(identifier, websocket_event_2_callback)
websocket_connector.subscribe_event_3(identifier, websocket_event_3_callback)
# Prepare associated data
data = {...}
# Create thread structure and launch thread
with self.data_mutex:
self.ongoing_threads[identifier] = {
"thread": ExceptionThread(name=f"ongoing_thead-{identifier}", target=self.ongoing_thread, args=(identifier,)),
"data": data,
"queues": {
"event_1": multiprocessing.Queue(),
"event_2": multiprocessing.Queue(),
"event_3": multiprocessing.Queue(),
},
}
self.ongoing_threads[identifier]["thread"].start()
self.ongoing_threads_number += 1
def ongoing_thread(self, identifier):
thread_name = f"[Thread-{threading.current_thread().name}]"
log(f"{thread_name} - Start thread")
started_at = datetime.datetime.now()
with self.data_mutex:
thread = self.ongoing_threads[identifier]["thread"]
queues = self.ongoing_threads[identifier]["queues"]
while not thread.is_stopped():
order_update = None
orderbook_update = None
mark_price_update = None
event_1_queue = queues["event_1"]
event_2_queue = queues["event_2"]
event_3_queue = queues["event_3"]
rd_set, _, _ = select.select([event_1_queue._reader, event_2_queue._reader, event_3_queue._reader], [], [], self.event_timeout)
# Timeout
if len(rd_set) == 0:
continue
if not event_1_queue.empty():
event_1 = event_1_queue.get()
if not event_2_queue.empty():
event_2 = event_2_queue.get()
if not event_3_queue.empty():
event_3 = event_3_queue.get()
# Treat events bellow
# Depending on the event and the associated data, the thread may continue or return
def websocket_event_1_callback(self, data):
with self.data_mutex:
for ongoing_thread in self.ongoing_threads.values():
ongoing_thread["queues"]["event_1"].put(data)
def websocket_event_2_callback(self, data):
with self.data_mutex:
for ongoing_thread in self.ongoing_threads.values():
ongoing_thread["queues"]["event_2"].put(data)
def websocket_event_3_callback(self, data):
identifier = data["identifier"]
with self.data_mutex:
for ongoing_thread in self.ongoing_threads.values():
if ongoing_thread["identifier"] == identifier:
ongoing_thread["queues"]["event_3"].put(data)
tester = Tester()
tester.run()The only thing which are not present here are the data_sets data, the has_opportunity function and the websocket_connector instance, however it's not necessary to understand the script logic.Debugging the crashes
Since the script crashes regularly, in order to understand what was happening, I recompiled python 3.10.10 from sources with debug support to be able to debug it with gdb.
- Compiling a debug python version
Output:$ wget -c https://www.python.org/ftp/python/3.10.10/Python-3.10.10.tar.xz
$ tar -Jxvf Python-3.10.10.tar.xz
$ mkdir Python-3.10.10/build
$ cd Python-3.10.10/build
$ ../configure --with-pydebug --prefix=~/python_installs/python-3.10.10-debug
$ make EXTRA_CFLAGS="-DPy_REF_DEBUG"
$ make install
$ mkdir ~/python_installs/python-3.10.10-debug/share/gdb/auto-load/usr/bin/
$ cp build/python-gdb.py ~/python_installs/python-3.10.10-debug/share/gdb/auto-load/usr/bin/python3.10-gdb.pyThe use of the file python3.10-gdb.py is just to be able to use gdb python extensions.
- Debugging the core dump files
In order to analyze coredumps I do as follow:
Output:$ gdb /home/debian/python_installs/python-3.10.10-debug/bin/python3.10d 1681297874_python3.10d_435902.coredump
GNU gdb (Debian 10.1-1.7) 10.1.90.20210103-git
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/debian/python_installs/python-3.10.10-debug-with-gdb/bin/python3.10d...
Illegal process-id: 1681233661_python3.10d_264681.coredump.
warning: Can't open file /dev/shm/oxui76 (deleted) during file-backed mapping note processing
warning: Can't open file /dev/shm/9XD0o6 (deleted) during file-backed mapping note processing
warning: Can't open file /dev/shm/TOnqY9 (deleted) during file-backed mapping note processing
warning: Can't open file /dev/shm/tg82K7 (deleted) during file-backed mapping note processing
warning: Can't open file /dev/shm/4iGkN7 (deleted) during file-backed mapping note processing
warning: Can't open file /dev/shm/dwBm98 (deleted) during file-backed mapping note processing
warning: Can't open file /dev/shm/RcFUs7 (deleted) during file-backed mapping note processing
warning: Can't open file /dev/shm/eGeFR8 (deleted) during file-backed mapping note processing
warning: Can't open file /dev/shm/HEFdx8 (deleted) during file-backed mapping note processing
warning: Can't open file /dev/shm/a5VDM6 (deleted) during file-backed mapping note processing
warning: Can't open file /dev/shm/JVvhwa (deleted) during file-backed mapping note processing
warning: Can't open file /dev/shm/SFFsE8 (deleted) during file-backed mapping note processing
warning: Can't open file /dev/shm/ttvUca (deleted) during file-backed mapping note processing
warning: Can't open file /dev/shm/gqm0P6 (deleted) during file-backed mapping note processing
warning: Can't open file /dev/shm/YXJLo6 (deleted) during file-backed mapping note processing
warning: Can't open file /dev/shm/BLVPsa (deleted) during file-backed mapping note processing
warning: Can't open file /dev/shm/LwGcm8 (deleted) during file-backed mapping note processing
warning: Can't open file /dev/shm/5hKxHa (deleted) during file-backed mapping note processing
warning: Can't open file /dev/shm/sddqv6 (deleted) during file-backed mapping note processing
warning: Can't open file /dev/shm/crImo8 (deleted) during file-backed mapping note processing
warning: Can't open file /dev/shm/t6RIY8 (deleted) during file-backed mapping note processing
warning: Can't open file /dev/shm/0lQrU8 (deleted) during file-backed mapping note processing
warning: Can't open file /dev/shm/F9zHF8 (deleted) during file-backed mapping note processing
warning: Can't open file /dev/shm/yUJQ29 (deleted) during file-backed mapping note processing
warning: Can't open file /dev/shm/5BV4G6 (deleted) during file-backed mapping note processing
warning: Can't open file /dev/shm/mSV595 (deleted) during file-backed mapping note processing
warning: Can't open file /dev/shm/e3ISba (deleted) during file-backed mapping note processing
warning: Can't open file /dev/shm/M8mZz8 (deleted) during file-backed mapping note processing
warning: Can't open file /dev/shm/ok9l79 (deleted) during file-backed mapping note processing
warning: Can't open file /dev/shm/iAbv89 (deleted) during file-backed mapping note processing
warning: Can't open file /dev/shm/Ui3Ok8 (deleted) during file-backed mapping note processing
warning: Can't open file /dev/shm/9tUh19 (deleted) during file-backed mapping note processing
warning: Can't open file /dev/shm/NTjE27 (deleted) during file-backed mapping note processing
warning: Can't open file /dev/shm/x11pV6 (deleted) during file-backed mapping note processing
warning: Can't open file /dev/shm/NC88j9 (deleted) during file-backed mapping note processing
warning: Can't open file /dev/shm/FnbDX6 (deleted) during file-backed mapping note processing
warning: Can't open file /dev/shm/Bl6lra (deleted) during file-backed mapping note processing
warning: Can't open file /dev/shm/HMsCca (deleted) during file-backed mapping note processing
warning: Can't open file /dev/shm/25Mku9 (deleted) during file-backed mapping note processing
warning: Can't open file /dev/shm/Zbnme6 (deleted) during file-backed mapping note processing
warning: Can't open file /dev/shm/ppmOF9 (deleted) during file-backed mapping note processing
warning: Can't open file /dev/shm/v70k26 (deleted) during file-backed mapping note processing
warning: Can't open file /dev/shm/qaGCF6 (deleted) during file-backed mapping note processing
warning: Can't open file /dev/shm/NsxeY9 (deleted) during file-backed mapping note processing
warning: Can't open file /dev/shm/9Nj249 (deleted) during file-backed mapping note processing
[New LWP 271007]
[New LWP 264691]
[New LWP 270824]
[New LWP 264716]
[New LWP 270840]
[New LWP 270835]
[New LWP 270950]
[New LWP 270951]
[New LWP 270843]
[New LWP 270953]
[New LWP 270995]
[New LWP 270949]
[New LWP 271004]
[New LWP 271045]
[New LWP 271006]
[New LWP 271068]
[New LWP 271046]
[New LWP 264681]
[New LWP 271079]
[New LWP 269294]
[New LWP 271096]
[New LWP 270823]
[New LWP 265931]
[New LWP 265134]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/home/debian/python_installs/python-3.10.10-debug/bin/python3.10d /home/debian/'.
Program terminated with signal SIGABRT, Aborted.
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
[Current thread is 1 (Thread 0x7f1258ff9700 (LWP 271007))]
(gdb) source ../Python-3.10.10/build/python-gdb.py
(gdb) bt
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1 0x00007f1298896537 in __GI_abort () at abort.c:79
#2 0x00007f12988ef768 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7f1298a0d3a5 "%s\n") at ../sysdeps/posix/libc_fatal.c:155
#3 0x00007f12988f6a5a in malloc_printerr (str=str@entry=0x7f1298a0fbf8 "malloc(): unsorted double linked list corrupted") at malloc.c:5347
#4 0x00007f12988f9d74 in _int_malloc (av=av@entry=0x7f1260000020, bytes=bytes@entry=4177) at malloc.c:3744
#5 0x00007f12988fb299 in __GI___libc_malloc (bytes=4177) at malloc.c:3066
#6 0x000055f40e696a5e in _PyMem_RawMalloc (ctx=<optimized out>, size=<optimized out>) at ../Objects/obmalloc.c:99
#7 0x000055f40e696950 in _PyMem_DebugRawAlloc (use_calloc=use_calloc@entry=0, ctx=0x55f40ea12f80 <_PyMem_Debug>, nbytes=4153) at ../Objects/obmalloc.c:2471
#8 0x000055f40e6969e6 in _PyMem_DebugRawMalloc (ctx=<optimized out>, nbytes=<optimized out>) at ../Objects/obmalloc.c:2504
#9 0x000055f40e697d49 in PyMem_RawMalloc (size=size@entry=4153) at ../Objects/obmalloc.c:572
#10 0x000055f40e699745 in _PyObject_Malloc (ctx=<optimized out>, nbytes=4153) at ../Objects/obmalloc.c:1966
#11 0x000055f40e696950 in _PyMem_DebugRawAlloc (use_calloc=use_calloc@entry=0, ctx=ctx@entry=0x55f40ea12fe0 <_PyMem_Debug+96>, nbytes=nbytes@entry=4129) at ../Objects/obmalloc.c:2471
#12 0x000055f40e6969e6 in _PyMem_DebugRawMalloc (ctx=ctx@entry=0x55f40ea12fe0 <_PyMem_Debug+96>, nbytes=nbytes@entry=4129) at ../Objects/obmalloc.c:2504
#13 0x000055f40e69736c in _PyMem_DebugMalloc (ctx=0x55f40ea12fe0 <_PyMem_Debug+96>, nbytes=4129) at ../Objects/obmalloc.c:2656
#14 0x000055f40e69874d in PyObject_Malloc (size=size@entry=4129) at ../Objects/obmalloc.c:685
#15 0x000055f40e64b1a6 in _PyBytes_FromSize (size=size@entry=4096, use_calloc=use_calloc@entry=0) at ../Objects/bytesobject.c:126
#16 0x000055f40e64ba10 in PyBytes_FromStringAndSize (str=str@entry=0x0, size=size@entry=4096) at ../Objects/bytesobject.c:159
#17 0x00007f1292d8ec4d in _pickle_Pickler___init___impl (self=self@entry=0x7f128608aa90, file=file@entry=<_io.BytesIO at remote 0x7f1285cf0830>, protocol=protocol@entry=None, fix_imports=1, buffer_callback=None) at /home/debian/python_installs/Python-3.10.10/Modules/_pickle.c:4777
#18 0x00007f1292d8ee02 in _pickle_Pickler___init__ (self=<ForkingPickler at remote 0x7f128608aa90>, self@entry=<error reading variable: value has been optimized out>, args=<error reading variable: value has been optimized out>,
kwargs=<error reading variable: value has been optimized out>) at /home/debian/python_installs/Python-3.10.10/Modules/clinic/_pickle.c.h:145
#19 0x000055f40e6a870a in wrap_init (self=<optimized out>, args=<optimized out>, wrapped=<optimized out>, kwds=<optimized out>) at ../Objects/typeobject.c:6954
#20 0x000055f40e8173dd in wrapperdescr_raw_call (kwds=<optimized out>, args=<optimized out>, self=<optimized out>, descr=<optimized out>) at ../Objects/descrobject.c:506
#21 wrapper_call (wp=<optimized out>, args=<optimized out>, kwds=<optimized out>) at ../Objects/descrobject.c:1385
#22 0x000055f40e6579e4 in _PyObject_Call (tstate=0x7f1278004fd0, callable=callable@entry=<method-wrapper '__init__' of ForkingPickler object at 0x7f128608aa90>, args=args@entry=(<_io.BytesIO at remote 0x7f1285cf0830>, None), kwargs=kwargs@entry=0x0) at ../Objects/call.c:305
#23 0x000055f40e657b6f in PyObject_Call (callable=callable@entry=<method-wrapper '__init__' of ForkingPickler object at 0x7f128608aa90>, args=args@entry=(<_io.BytesIO at remote 0x7f1285cf0830>, None), kwargs=kwargs@entry=0x0) at ../Objects/call.c:317
#24 0x000055f40e70e61f in do_call_core (tstate=tstate@entry=0x7f1278004fd0, trace_info=trace_info@entry=0x7f1258ff7f00, func=func@entry=<method-wrapper '__init__' of ForkingPickler object at 0x7f128608aa90>, callargs=callargs@entry=(<_io.BytesIO at remote 0x7f1285cf0830>, None),
kwdict=kwdict@entry=0x0) at ../Python/ceval.c:5943
#25 0x000055f40e71b727 in _PyEval_EvalFrameDefault (tstate=0x7f1278004fd0,
f=Frame 0x7f1286001550, for file /home/debian/python_installs/python-3.10.10-debug/lib/python3.10/multiprocessing/reduction.py, line 39, in __init__ (self=<ForkingPickler at remote 0x7f128608aa90>, args=(<_io.BytesIO at remote 0x7f1285cf0830>, None)),
throwflag=<optimized out>) at ../Python/ceval.c:4277
#26 0x000055f40e71d848 in _PyEval_EvalFrame (throwflag=0,
f=Frame 0x7f1286001550, for file /home/debian/python_installs/python-3.10.10-debug/lib/python3.10/multiprocessing/reduction.py, line 39, in __init__ (self=<ForkingPickler at remote 0x7f128608aa90>, args=(<_io.BytesIO at remote 0x7f1285cf0830>, None)), tstate=0x7f1278004fd0)
at ../Include/internal/pycore_ceval.h:46
#27 _PyEval_Vector (tstate=0x7f1278004fd0, con=0x7f1286d49710, locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at ../Python/ceval.c:5065
#28 0x000055f40e657be9 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at ../Objects/call.c:342
#29 0x000055f40e658416 in _PyObject_FastCallDictTstate (tstate=tstate@entry=0x7f1278004fd0, callable=callable@entry=<function at remote 0x7f1286d49700>, args=args@entry=0x7f1258ff8030, nargsf=nargsf@entry=3, kwargs=kwargs@entry=0x0) at ../Objects/call.c:142
#30 0x000055f40e658755 in _PyObject_Call_Prepend (tstate=tstate@entry=0x7f1278004fd0, callable=callable@entry=<function at remote 0x7f1286d49700>, obj=obj@entry=<ForkingPickler at remote 0x7f128608aa90>, args=args@entry=(<_io.BytesIO at remote 0x7f1285cf0830>, None),
kwargs=kwargs@entry=0x0) at ../Objects/call.c:431
#31 0x000055f40e6b619d in slot_tp_init (self=<ForkingPickler at remote 0x7f128608aa90>, args=(<_io.BytesIO at remote 0x7f1285cf0830>, None), kwds=0x0) at ../Objects/typeobject.c:7734
#32 0x000055f40e6b232c in type_call (type=0x55f4111214f0, args=(<_io.BytesIO at remote 0x7f1285cf0830>, None), kwds=0x0) at ../Objects/typeobject.c:1135
#33 0x000055f40e657f8e in _PyObject_MakeTpCall (tstate=tstate@entry=0x7f1278004fd0, callable=callable@entry=<type at remote 0x55f4111214f0>, args=args@entry=0x7f1286064748, nargs=<optimized out>, keywords=keywords@entry=0x0) at ../Objects/call.c:215
#34 0x000055f40e709ee1 in _PyObject_VectorcallTstate (tstate=0x7f1278004fd0, callable=<type at remote 0x55f4111214f0>, args=0x7f1286064748, nargsf=<optimized out>, kwnames=0x0) at ../Include/cpython/abstract.h:112
#35 0x000055f40e70e335 in PyObject_Vectorcall (kwnames=0x0, nargsf=9223372036854775810, args=0x7f1286064748, callable=<type at remote 0x55f4111214f0>) at ../Include/cpython/abstract.h:123
#36 call_function (tstate=tstate@entry=0x7f1278004fd0, trace_info=trace_info@entry=0x7f1258ff82f0, pp_stack=pp_stack@entry=0x7f1258ff82a0, oparg=oparg@entry=2, kwnames=kwnames@entry=0x0) at ../Python/ceval.c:5891
#37 0x000055f40e71b267 in _PyEval_EvalFrameDefault (tstate=0x7f1278004fd0,
f=Frame 0x7f12860645c0, for file /home/debian/python_installs/python-3.10.10-debug/lib/python3.10/multiprocessing/reduction.py, line 51, in dumps (cls=<type at remote 0x55f4111214f0>, obj={...}, protocol=None, buf=<_io.BytesIO at remote 0x7f1285cf0830>), throwflag=<optimized out>) at ../Python/ceval.c:4213
#38 0x000055f40e71d848 in _PyEval_EvalFrame (throwflag=0,
f=Frame 0x7f12860645c0, for file /home/debian/python_installs/python-3.10.10-debug/lib/python3.10/multiprocessing/reduction.py, line 51, in dumps (cls=<type at remote 0x55f4111214f0>, obj={...}, protocol=None, buf=<_io.BytesIO at remote 0x7f1285cf0830>), tstate=0x7f1278004fd0) at ../Include/internal/pycore_ceval.h:46
#39 _PyEval_Vector (tstate=0x7f1278004fd0, con=0x7f1286d49870, locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at ../Python/ceval.c:5065
#40 0x000055f40e657be9 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at ../Objects/call.c:342
#41 0x000055f40e80f6ec in _PyObject_VectorcallTstate (tstate=0x7f1278004fd0, callable=<function at remote 0x7f1286d49860>, args=0x7f124c0ed008, nargsf=2, kwnames=0x0) at ../Include/cpython/abstract.h:114
#42 0x000055f40e80ff50 in method_vectorcall (method=<optimized out>, args=0x7f124c0ed010, nargsf=<optimized out>, kwnames=0x0) at ../Objects/classobject.c:53
#43 0x000055f40e709df0 in _PyObject_VectorcallTstate (tstate=0x7f1278004fd0, callable=<method at remote 0x7f1285edbb30>, args=0x7f124c0ed010, nargsf=9223372036854775809, kwnames=0x0) at ../Include/cpython/abstract.h:114
#44 0x000055f40e70e335 in PyObject_Vectorcall (kwnames=0x0, nargsf=9223372036854775809, args=0x7f124c0ed010, callable=<method at remote 0x7f1285edbb30>) at ../Include/cpython/abstract.h:123
#45 call_function (tstate=tstate@entry=0x7f1278004fd0, trace_info=trace_info@entry=0x7f1258ff85f0, pp_stack=pp_stack@entry=0x7f1258ff8598, oparg=oparg@entry=1, kwnames=kwnames@entry=0x0) at ../Python/ceval.c:5891
#46 0x000055f40e71b1d6 in _PyEval_EvalFrameDefault (tstate=0x7f1278004fd0,
f=Frame 0x7f124c0ece10, for file /home/debian/python_installs/python-3.10.10-debug/lib/python3.10/multiprocessing/queues.py, line 244, in _feed (buffer=<collections.deque at remote 0x7f1285b373d0>, notempty=<Condition(_lock=<_thread.lock at remote 0x7f1285eb7c00>, acquire=<built-in method acquire of _thread.lock object at remote 0x7f1285eb7c00>, release=<built-in method release of _thread.lock object at remote 0x7f1285eb7c00>, _waiters=<collections.deque at remote 0x7f1285b37150>) at remote 0x7f1286b5be80>, send_bytes=<method at remote 0x7f1285ecd310>, writelock=<Lock(_semlock=<_multiprocessing.SemLock at remote 0x7f1285eb78d0>, acquire=<built-in method acquire of _multiprocessing.SemLock object at remote 0x7f1285eb78d0>, release=<built-in method release of _multiprocessing.SemLock object at remote 0x7f1285eb78d0>) at remote 0x7f1285eb7660>, reader_close=<method at remote 0x7f1286082f30>, writer_close=<method at remote 0x7f1286082a50>, ignore_epipe=False, onerror=<function at remote 0x7f12860859c0>, queue_sem=<...(truncated), throwflag=<optimized out>) at ../Python/ceval.c:4181
#47 0x000055f40e71d848 in _PyEval_EvalFrame (throwflag=0,
f=Frame 0x7f124c0ece10, for file /home/debian/python_installs/python-3.10.10-debug/lib/python3.10/multiprocessing/queues.py, line 244, in _feed (buffer=<collections.deque at remote 0x7f1285b373d0>, notempty=<Condition(_lock=<_thread.lock at remote 0x7f1285eb7c00>, acquire=<built-in method acquire of _thread.lock object at remote 0x7f1285eb7c00>, release=<built-in method release of _thread.lock object at remote 0x7f1285eb7c00>, _waiters=<collections.deque at remote 0x7f1285b37150>) at remote 0x7f1286b5be80>, send_bytes=<method at remote 0x7f1285ecd310>, writelock=<Lock(_semlock=<_multiprocessing.SemLock at remote 0x7f1285eb78d0>, acquire=<built-in method acquire of _multiprocessing.SemLock object at remote 0x7f1285eb78d0>, release=<built-in method release of _multiprocessing.SemLock object at remote 0x7f1285eb78d0>) at remote 0x7f1285eb7660>, reader_close=<method at remote 0x7f1286082f30>, writer_close=<method at remote 0x7f1286082a50>, ignore_epipe=False, onerror=<function at remote 0x7f12860859c0>, queue_sem=<...(truncated), tstate=0x7f1278004fd0) at ../Include/internal/pycore_ceval.h:46
#48 _PyEval_Vector (tstate=0x7f1278004fd0, con=0x7f1286085920, locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at ../Python/ceval.c:5065
--Type <RET> for more, q to quit, c to continue without paging--
#49 0x000055f40e657be9 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at ../Objects/call.c:342
#50 0x000055f40e657807 in PyVectorcall_Call (callable=callable@entry=<function at remote 0x7f1286085910>,
tuple=tuple@entry=(<collections.deque at remote 0x7f1285b373d0>, <Condition(_lock=<_thread.lock at remote 0x7f1285eb7c00>, acquire=<built-in method acquire of _thread.lock object at remote 0x7f1285eb7c00>, release=<built-in method release of _thread.lock object at remote 0x7f1285eb7c00>, _waiters=<collections.deque at remote 0x7f1285b37150>) at remote 0x7f1286b5be80>, <method at remote 0x7f1285ecd310>, <Lock(_semlock=<_multiprocessing.SemLock at remote 0x7f1285eb78d0>, acquire=<built-in method acquire of _multiprocessing.SemLock object at remote 0x7f1285eb78d0>, release=<built-in method release of _multiprocessing.SemLock object at remote 0x7f1285eb78d0>) at remote 0x7f1285eb7660>, <method at remote 0x7f1286082f30>, <method at remote 0x7f1286082a50>, False, <function at remote 0x7f12860859c0>, <BoundedSemaphore(_semlock=<_multiprocessing.SemLock at remote 0x7f1285eb77e0>, acquire=<built-in method acquire of _multiprocessing.SemLock object at remote 0x7f1285eb77e0>, release=<built-in method release of _multiprocessing.SemLock o...(truncated), kwargs=kwargs@entry={}) at ../Objects/call.c:255
#51 0x000055f40e657b06 in _PyObject_Call (tstate=0x7f1278004fd0, callable=callable@entry=<function at remote 0x7f1286085910>,
args=args@entry=(<collections.deque at remote 0x7f1285b373d0>, <Condition(_lock=<_thread.lock at remote 0x7f1285eb7c00>, acquire=<built-in method acquire of _thread.lock object at remote 0x7f1285eb7c00>, release=<built-in method release of _thread.lock object at remote 0x7f1285eb7c00>, _waiters=<collections.deque at remote 0x7f1285b37150>) at remote 0x7f1286b5be80>, <method at remote 0x7f1285ecd310>, <Lock(_semlock=<_multiprocessing.SemLock at remote 0x7f1285eb78d0>, acquire=<built-in method acquire of _multiprocessing.SemLock object at remote 0x7f1285eb78d0>, release=<built-in method release of _multiprocessing.SemLock object at remote 0x7f1285eb78d0>) at remote 0x7f1285eb7660>, <method at remote 0x7f1286082f30>, <method at remote 0x7f1286082a50>, False, <function at remote 0x7f12860859c0>, <BoundedSemaphore(_semlock=<_multiprocessing.SemLock at remote 0x7f1285eb77e0>, acquire=<built-in method acquire of _multiprocessing.SemLock object at remote 0x7f1285eb77e0>, release=<built-in method release of _multiprocessing.SemLock o...(truncated), kwargs=kwargs@entry={}) at ../Objects/call.c:290
#52 0x000055f40e657b6f in PyObject_Call (callable=callable@entry=<function at remote 0x7f1286085910>,
args=args@entry=(<collections.deque at remote 0x7f1285b373d0>, <Condition(_lock=<_thread.lock at remote 0x7f1285eb7c00>, acquire=<built-in method acquire of _thread.lock object at remote 0x7f1285eb7c00>, release=<built-in method release of _thread.lock object at remote 0x7f1285eb7c00>, _waiters=<collections.deque at remote 0x7f1285b37150>) at remote 0x7f1286b5be80>, <method at remote 0x7f1285ecd310>, <Lock(_semlock=<_multiprocessing.SemLock at remote 0x7f1285eb78d0>, acquire=<built-in method acquire of _multiprocessing.SemLock object at remote 0x7f1285eb78d0>, release=<built-in method release of _multiprocessing.SemLock object at remote 0x7f1285eb78d0>) at remote 0x7f1285eb7660>, <method at remote 0x7f1286082f30>, <method at remote 0x7f1286082a50>, False, <function at remote 0x7f12860859c0>, <BoundedSemaphore(_semlock=<_multiprocessing.SemLock at remote 0x7f1285eb77e0>, acquire=<built-in method acquire of _multiprocessing.SemLock object at remote 0x7f1285eb77e0>, release=<built-in method release of _multiprocessing.SemLock o...(truncated), kwargs=kwargs@entry={}) at ../Objects/call.c:317
#53 0x000055f40e70e61f in do_call_core (tstate=tstate@entry=0x7f1278004fd0, trace_info=trace_info@entry=0x7f1258ff8870, func=func@entry=<function at remote 0x7f1286085910>,
callargs=callargs@entry=(<collections.deque at remote 0x7f1285b373d0>, <Condition(_lock=<_thread.lock at remote 0x7f1285eb7c00>, acquire=<built-in method acquire of _thread.lock object at remote 0x7f1285eb7c00>, release=<built-in method release of _thread.lock object at remote 0x7f1285eb7c00>, _waiters=<collections.deque at remote 0x7f1285b37150>) at remote 0x7f1286b5be80>, <method at remote 0x7f1285ecd310>, <Lock(_semlock=<_multiprocessing.SemLock at remote 0x7f1285eb78d0>, acquire=<built-in method acquire of _multiprocessing.SemLock object at remote 0x7f1285eb78d0>, release=<built-in method release of _multiprocessing.SemLock object at remote 0x7f1285eb78d0>) at remote 0x7f1285eb7660>, <method at remote 0x7f1286082f30>, <method at remote 0x7f1286082a50>, False, <function at remote 0x7f12860859c0>, <BoundedSemaphore(_semlock=<_multiprocessing.SemLock at remote 0x7f1285eb77e0>, acquire=<built-in method acquire of _multiprocessing.SemLock object at remote 0x7f1285eb77e0>, release=<built-in method release of _multiprocessing.SemLock o...(truncated), kwdict=kwdict@entry={}) at ../Python/ceval.c:5943
#54 0x000055f40e71b727 in _PyEval_EvalFrameDefault (tstate=0x7f1278004fd0,
f=Frame 0x7f12802f3a90, for file /home/debian/python_installs/python-3.10.10-debug/lib/python3.10/threading.py, line 953, in run (self=<Thread(_target=<function at remote 0x7f1286085910>, _name='QueueFeederThread', _args=(<collections.deque at remote 0x7f1285b373d0>, <Condition(_lock=<_thread.lock at remote 0x7f1285eb7c00>, acquire=<built-in method acquire of _thread.lock object at remote 0x7f1285eb7c00>, release=<built-in method release of _thread.lock object at remote 0x7f1285eb7c00>, _waiters=<collections.deque at remote 0x7f1285b37150>) at remote 0x7f1286b5be80>, <method at remote 0x7f1285ecd310>, <Lock(_semlock=<_multiprocessing.SemLock at remote 0x7f1285eb78d0>, acquire=<built-in method acquire of _multiprocessing.SemLock object at remote 0x7f1285eb78d0>, release=<built-in method release of _multiprocessing.SemLock object at remote 0x7f1285eb78d0>) at remote 0x7f1285eb7660>, <method at remote 0x7f1286082f30>, <method at remote 0x7f1286082a50>, False, <function at remote 0x7f12860859c0>, <BoundedSemaphore(...(truncated), throwflag=<optimized out>) at ../Python/ceval.c:4277
#55 0x000055f40e71d848 in _PyEval_EvalFrame (throwflag=0,
f=Frame 0x7f12802f3a90, for file /home/debian/python_installs/python-3.10.10-debug/lib/python3.10/threading.py, line 953, in run (self=<Thread(_target=<function at remote 0x7f1286085910>, _name='QueueFeederThread', _args=(<collections.deque at remote 0x7f1285b373d0>, <Condition(_lock=<_thread.lock at remote 0x7f1285eb7c00>, acquire=<built-in method acquire of _thread.lock object at remote 0x7f1285eb7c00>, release=<built-in method release of _thread.lock object at remote 0x7f1285eb7c00>, _waiters=<collections.deque at remote 0x7f1285b37150>) at remote 0x7f1286b5be80>, <method at remote 0x7f1285ecd310>, <Lock(_semlock=<_multiprocessing.SemLock at remote 0x7f1285eb78d0>, acquire=<built-in method acquire of _multiprocessing.SemLock object at remote 0x7f1285eb78d0>, release=<built-in method release of _multiprocessing.SemLock object at remote 0x7f1285eb78d0>) at remote 0x7f1285eb7660>, <method at remote 0x7f1286082f30>, <method at remote 0x7f1286082a50>, False, <function at remote 0x7f12860859c0>, <BoundedSemaphore(...(truncated), tstate=0x7f1278004fd0) at ../Include/internal/pycore_ceval.h:46
#56 _PyEval_Vector (tstate=0x7f1278004fd0, con=0x7f1298453550, locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at ../Python/ceval.c:5065
#57 0x000055f40e657be9 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at ../Objects/call.c:342
#58 0x000055f40e709df0 in _PyObject_VectorcallTstate (tstate=0x7f1278004fd0, callable=<function at remote 0x7f1298453540>, args=0x7f1270022cd0, nargsf=9223372036854775809, kwnames=0x0) at ../Include/cpython/abstract.h:114
#59 0x000055f40e70e335 in PyObject_Vectorcall (kwnames=0x0, nargsf=9223372036854775809, args=0x7f1270022cd0, callable=<function at remote 0x7f1298453540>) at ../Include/cpython/abstract.h:123
#60 call_function (tstate=tstate@entry=0x7f1278004fd0, trace_info=trace_info@entry=0x7f1258ff8ab0, pp_stack=pp_stack@entry=0x7f1258ff8a58, oparg=oparg@entry=1, kwnames=kwnames@entry=0x0) at ../Python/ceval.c:5891
#61 0x000055f40e71b117 in _PyEval_EvalFrameDefault (tstate=0x7f1278004fd0,
f=Frame 0x7f1270022b60, for file /home/debian/python_installs/python-3.10.10-debug/lib/python3.10/threading.py, line 1016, in _bootstrap_inner (self=<Thread(_target=<function at remote 0x7f1286085910>, _name='QueueFeederThread', _args=(<collections.deque at remote 0x7f1285b373d0>, <Condition(_lock=<_thread.lock at remote 0x7f1285eb7c00>, acquire=<built-in method acquire of _thread.lock object at remote 0x7f1285eb7c00>, release=<built-in method release of _thread.lock object at remote 0x7f1285eb7c00>, _waiters=<collections.deque at remote 0x7f1285b37150>) at remote 0x7f1286b5be80>, <method at remote 0x7f1285ecd310>, <Lock(_semlock=<_multiprocessing.SemLock at remote 0x7f1285eb78d0>, acquire=<built-in method acquire of _multiprocessing.SemLock object at remote 0x7f1285eb78d0>, release=<built-in method release of _multiprocessing.SemLock object at remote 0x7f1285eb78d0>) at remote 0x7f1285eb7660>, <method at remote 0x7f1286082f30>, <method at remote 0x7f1286082a50>, False, <function at remote 0x7f12860859c0>, <Bou...(truncated), throwflag=<optimized out>) at ../Python/ceval.c:4198
#62 0x000055f40e71d848 in _PyEval_EvalFrame (throwflag=0,
f=Frame 0x7f1270022b60, for file /home/debian/python_installs/python-3.10.10-debug/lib/python3.10/threading.py, line 1016, in _bootstrap_inner (self=<Thread(_target=<function at remote 0x7f1286085910>, _name='QueueFeederThread', _args=(<collections.deque at remote 0x7f1285b373d0>, <Condition(_lock=<_thread.lock at remote 0x7f1285eb7c00>, acquire=<built-in method acquire of _thread.lock object at remote 0x7f1285eb7c00>, release=<built-in method release of _thread.lock object at remote 0x7f1285eb7c00>, _waiters=<collections.deque at remote 0x7f1285b37150>) at remote 0x7f1286b5be80>, <method at remote 0x7f1285ecd310>, <Lock(_semlock=<_multiprocessing.SemLock at remote 0x7f1285eb78d0>, acquire=<built-in method acquire of _multiprocessing.SemLock object at remote 0x7f1285eb78d0>, release=<built-in method release of _multiprocessing.SemLock object at remote 0x7f1285eb78d0>) at remote 0x7f1285eb7660>, <method at remote 0x7f1286082f30>, <method at remote 0x7f1286082a50>, False, <function at remote 0x7f12860859c0>, <Bou...(truncated), tstate=0x7f1278004fd0) at ../Include/internal/pycore_ceval.h:46
#63 _PyEval_Vector (tstate=0x7f1278004fd0, con=0x7f12984538c0, locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at ../Python/ceval.c:5065
#64 0x000055f40e657be9 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at ../Objects/call.c:342
#65 0x000055f40e709df0 in _PyObject_VectorcallTstate (tstate=0x7f1278004fd0, callable=<function at remote 0x7f12984538b0>, args=0x7f1280424420, nargsf=9223372036854775809, kwnames=0x0) at ../Include/cpython/abstract.h:114
#66 0x000055f40e70e335 in PyObject_Vectorcall (kwnames=0x0, nargsf=9223372036854775809, args=0x7f1280424420, callable=<function at remote 0x7f12984538b0>) at ../Include/cpython/abstract.h:123
#67 call_function (tstate=tstate@entry=0x7f1278004fd0, trace_info=trace_info@entry=0x7f1258ff8cf0, pp_stack=pp_stack@entry=0x7f1258ff8c98, oparg=oparg@entry=1, kwnames=kwnames@entry=0x0) at ../Python/ceval.c:5891
#68 0x000055f40e71b117 in _PyEval_EvalFrameDefault (tstate=0x7f1278004fd0,
f=Frame 0x7f12804242b0, for file /home/debian/python_installs/python-3.10.10-debug/lib/python3.10/threading.py, line 973, in _bootstrap (self=<Thread(_target=<function at remote 0x7f1286085910>, _name='QueueFeederThread', _args=(<collections.deque at remote 0x7f1285b373d0>, <Condition(_lock=<_thread.lock at remote 0x7f1285eb7c00>, acquire=<built-in method acquire of _thread.lock object at remote 0x7f1285eb7c00>, release=<built-in method release of _thread.lock object at remote 0x7f1285eb7c00>, _waiters=<collections.deque at remote 0x7f1285b37150>) at remote 0x7f1286b5be80>, <method at remote 0x7f1285ecd310>, <Lock(_semlock=<_multiprocessing.SemLock at remote 0x7f1285eb78d0>, acquire=<built-in method acquire of _multiprocessing.SemLock object at remote 0x7f1285eb78d0>, release=<built-in method release of _multiprocessing.SemLock object at remote 0x7f1285eb78d0>) at remote 0x7f1285eb7660>, <method at remote 0x7f1286082f30>, <method at remote 0x7f1286082a50>, False, <function at remote 0x7f12860859c0>, <BoundedSem...(truncated), throwflag=<optimized out>) at ../Python/ceval.c:4198
#69 0x000055f40e71d848 in _PyEval_EvalFrame (throwflag=0,
f=Frame 0x7f12804242b0, for file /home/debian/python_installs/python-3.10.10-debug/lib/python3.10/threading.py, line 973, in _bootstrap (self=<Thread(_target=<function at remote 0x7f1286085910>, _name='QueueFeederThread', _args=(<collections.deque at remote 0x7f1285b373d0>, <Condition(_lock=<_thread.lock at remote 0x7f1285eb7c00>, acquire=<built-in method acquire of _thread.lock object at remote 0x7f1285eb7c00>, release=<built-in method release of _thread.lock object at remote 0x7f1285eb7c00>, _waiters=<collections.deque at remote 0x7f1285b37150>) at remote 0x7f1286b5be80>, <method at remote 0x7f1285ecd310>, <Lock(_semlock=<_multiprocessing.SemLock at remote 0x7f1285eb78d0>, acquire=<built-in method acquire of _multiprocessing.SemLock object at remote 0x7f1285eb78d0>, release=<built-in method release of _multiprocessing.SemLock object at remote 0x7f1285eb78d0>) at remote 0x7f1285eb7660>, <method at remote 0x7f1286082f30>, <method at remote 0x7f1286082a50>, False, <function at remote 0x7f12860859c0>, <BoundedSem...(truncated), tstate=0x7f1278004fd0) at ../Include/internal/pycore_ceval.h:46
#70 _PyEval_Vector (tstate=0x7f1278004fd0, con=0x7f1298453600, locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at ../Python/ceval.c:5065
#71 0x000055f40e657be9 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at ../Objects/call.c:342
#72 0x000055f40e80ffcf in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=1, args=0x7f1258ff8de8, callable=<function at remote 0x7f12984535f0>, tstate=0x7f1278004fd0) at ../Include/cpython/abstract.h:114
#73 method_vectorcall (method=<optimized out>, args=0x7f1298724268, nargsf=<optimized out>, kwnames=<optimized out>) at ../Objects/classobject.c:61
#74 0x000055f40e657807 in PyVectorcall_Call (callable=callable@entry=<method at remote 0x7f1286083b30>, tuple=tuple@entry=(), kwargs=kwargs@entry=0x0) at ../Objects/call.c:255
#75 0x000055f40e657b06 in _PyObject_Call (tstate=0x7f1278004fd0, callable=<method at remote 0x7f1286083b30>, args=(), kwargs=0x0) at ../Objects/call.c:290
#76 0x000055f40e657b6f in PyObject_Call (callable=<optimized out>, args=<optimized out>, kwargs=<optimized out>) at ../Objects/call.c:317
#77 0x000055f40e7d29f1 in thread_run (boot_raw=boot_raw@entry=0x7f1285257740) at ../Modules/_threadmodule.c:1100
#78 0x000055f40e77077c in pythread_wrapper (arg=<optimized out>) at ../Python/thread_pthread.h:248
--Type <RET> for more, q to quit, c to continue without paging--
#79 0x00007f1298b9fea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#80 0x00007f1298970a2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95Error:(gdb) py-bt
Traceback (most recent call first):
<optimized out>
File "/home/debian/python_installs/python-3.10.10-debug/lib/python3.10/multiprocessing/reduction.py", line 39, in __init__
super().__init__(*args)
File "/home/debian/python_installs/python-3.10.10-debug/lib/python3.10/multiprocessing/reduction.py", line 51, in dumps
cls(buf, protocol).dump(obj)
File "/home/debian/python_installs/python-3.10.10-debug/lib/python3.10/multiprocessing/queues.py", line 244, in _feed
obj = _ForkingPickler.dumps(obj)
File "/home/debian/python_installs/python-3.10.10-debug/lib/python3.10/threading.py", line 953, in run
self._target(*self._args, **self._kwargs)
File "/home/debian/python_installs/python-3.10.10-debug/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
self.run()
File "/home/debian/python_installs/python-3.10.10-debug/lib/python3.10/threading.py", line 973, in _bootstrap
self._bootstrap_inner()Here are several backtrace I had from several coredumps
Error:(gdb) py-bt
Traceback (most recent call first):
File "/home/debian/python_installs/python-3.10.10-debug/lib/python3.10/ssl.py", line 1132, in read
return self._sslobj.read(len)
File "/home/debian/python_installs/python-3.10.10-debug/lib/python3.10/ssl.py", line 1259, in recv
return self.read(buflen)
File "/usr/local/lib/python3.10/site-packages/websocket/_socket.py", line 87, in _recv
return sock.recv(bufsize)
File "/usr/local/lib/python3.10/site-packages/websocket/_socket.py", line 108, in recv
bytes_ = _recv()
File "/usr/local/lib/python3.10/site-packages/websocket/_socket.py", line 131, in recv_line
c = recv(sock, 1)
File "/usr/local/lib/python3.10/site-packages/websocket/_http.py", line 312, in read_headers
line = recv_line(sock)
File "/usr/local/lib/python3.10/site-packages/websocket/_handshake.py", line 145, in _get_resp_headers
status, resp_headers, status_message = read_headers(sock)
File "/usr/local/lib/python3.10/site-packages/websocket/_handshake.py", line 57, in handshake
status, resp = _get_resp_headers(sock)
File "/usr/local/lib/python3.10/site-packages/websocket/_core.py", line 253, in connect
self.handshake_response = handshake(self.sock, url, *addrs, **options)
File "/home/debian/project/connector.py", line 2497, in __reconnect
websocket.connect(self.ws_url)
File "/home/debian/project/connector.py", line 2476, in __reconnect_timer
self.__reconnect(websocket)
File "/home/debian/python_installs/python-3.10.10-debug/lib/python3.10/threading.py", line 1378, in run
self.function(*self.args, **self.kwargs)
File "/home/debian/python_installs/python-3.10.10-debug/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
self.run()
File "/home/debian/python_installs/python-3.10.10-debug/lib/python3.10/threading.py", line 973, in _bootstrap
self._bootstrap_inner()Error:(gdb) py-bt
Traceback (most recent call first):
File "/home/debian/project/database.py", line 277, in __connect
pymysql.converters.conversions = pymysql.converters.encoders.copy()
File "/home/debian/project/database.py", line 215, in execute
self.__connect()
File "/home/debian/project/database.py", line 90, in get_columns
return [column[0] for column in self.__db.execute(f"SHOW columns FROM {self.__name}", fetch=True)]
File "/home/debian/project/database.py", line 76, in get_dataframe
columns = self.get_columns()
File "/home/debian/project/database.py", line 167, in get_table
return Table(self, table).get_dataframe(columns=columns, arg=arg, headers=headers)
File "/home/debian/project/debugged_script.py", line 319, in ongoing_threads_callback
df = self.database.getTable(table, arg=f"WHERE platform_id = {self.platform_id} AND user_id = {self.user_id}")
File "/home/debian/project/threading/Threads.py", line 14, in run
self.ret = self._target(*self._args, **self._kwargs)
File "/home/debian/python_installs/python-3.10.10-debug/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
self.run()
File "/home/debian/python_installs/python-3.10.10-debug/lib/python3.10/threading.py", line 973, in _bootstrap
self._bootstrap_inner()Error:(gdb) py-bt
Traceback (most recent call first):
<built-in method strftime of module object at remote 0x7f2c19d8e570>
File "/home/debian/project/logger.py", line 11, in get_timestamp
return datetime.datetime.now().strftime("%m-%d-%Y_%H-%M-%S.%f")
File "/home/debian/project/logger.py", line 18, in log
print("[{}] {}".format(get_timestamp(), message))
File "/home/debian/project/connector.py", line 2478, in __reconnect_timer
log(f"[{self}][RECONNECT] Reconnection failed.")
File "/home/debian/python_installs/python-3.10.10-debug/lib/python3.10/threading.py", line 1378, in run
self.function(*self.args, **self.kwargs)
File "/home/debian/python_installs/python-3.10.10-debug/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
self.run()
File "/home/debian/python_installs/python-3.10.10-debug/lib/python3.10/threading.py", line 973, in _bootstrap
self._bootstrap_inner()Since it seems to be memory related, I began to monitor the memory usage of the server and the process.
I have 8Gb available on the server and I'm always bellow 4Gb in global use
Even if the memory used by the process is slowly increasing, it's still acceptable (rarelly more than 200Mb), furthermore I have other python processes which consumes way more (For example I have one process which consumes up to 800Mb and runs for several days without never crashing)
Concluding
I don't know how to go further now, since the error doesn't seems to be consistent at all. This is the only script between many that crashes and even if it has a more complex architecture, it uses the same components.
I'm not really used to gdb with python so I don't fully understand the given backtrace and thus I don't understand where the error could come from. I do feel like this is related to a memory issue, but I can't go further into debugging and I'm beginning to lose faith.
That's why I'm coming here to see if someone would be able to help me to go through this issue.
If needed I can provide more gdb backtraces, make more runs with different python versions, etc...
Any help will be greatly appreciated.
Thanks in advance
