+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: General Coding Help (https://python-forum.io/forum-8.html)
+--- Thread: Socket connection and thread tracking... (/thread-13557.html)
Socket connection and thread tracking... - MuntyScruntfundle - Oct-20-2018
I've hit a bit of a brick wall, I'm sure there's a way around it, but my head is swimming in python syntax at the moment and I just can't see it!
My app is in two parts, a socket server that perversely sits on my clients, and a socket client that sits on my server.
When it's working ok the client logs onto all the servers, sends a code, the servers generate a data report and send it back. Lovely Jubly.
Except when a socket can't be opened. When the socket.connect(blah) fails for some reason the app stalls until the time out and things gradually start to fall apart.
So, I put all the data requests (socket connects) in new threads, so if one sits there the others don't get hung. This then needs some kind of marshalling as it's possible to go around the list and try to start a second thread on the same problem node/client.
So I wrote a class and some add/remove code to track open threads and make sure this didn't happen and now the app has such a large overhead I might as well me sending the data to a database, which is what I was trying to avoid in the first place!
I know I've asked a similar question before, and I have to admit I haven't checked responses yet, I thought I'd ramble while this was clear in my head.
If anyone out there has done anything similar, could you please let me know what you did in the end?
Outline:
Server/header raspberry pi
Clients raspberry pi
Number of clients/nodes 128
Data collection every 1 or 2 seconds. Could be increased with the server is working hard.
Data to be collected: core temperature, cpu load, memory stats, swap file stats, disk stats and some other bit of data which could be a one off call.
I'm going to write the database approach tonight, I'm very used to database work so this will be really easy, then try to gauge which will be the most heavy. Even with the database approach there will have to be some live communication to change timing, data required etc.
I don't need anything writing, I'm enjoying the learning, I just want to end up with the most sensible solution in place. While it's not mission critical I'd sooner do it right.
Many thanks.
RE: Socket connection and thread tracking... - Larz60+ - Oct-20-2018
You seem to have many threads on this same subject. This is a violation of forum rules.
If you want to get attention to a post, bump (re-post) with a simple post to same thread.
RE: Socket connection and thread tracking... - Skaperen - Oct-21-2018
i'm not sure which thread to answer. but you should try better to resolve the connection failure. if you are really trying to have your program do other stuff while the connecting is trying to be made, then yeah, a separate thread or process is needed so the connection is being tried in parallel, regardless whether or not it will fail.
it's not clear what you are doing. is it on thread for each of many servers the client connects to?
i wrote a script a while back that connects to several (11 of them if i recall) of those web sites that shows you IP address. it connects to them all using processes (instead of threads). but it handles connection failures gracefully because there are almost always a few that fail. it waits until all processes are done which means the one that took the longest time affects the timing of the script, whether it succeeds or fails. i think i put a timeout in there because a successful connection could be the worst case and wait forever. connection failures get timed out by the system, but an established connection could, in theory, wait for years ... or longer.
the reason i made this connection to many servers is because many of them could be down at the same time. many of the cloud scripts i run need to know what IP address they will be seen as coming from so they can make various requests properly, like setting up cloud security group rules. i would not want to depend on just one server since all of them have had some down time (even Google).
i also have other scripts doing multiprocessing. i generally don't use threads. so i really can't help with the specific difficulties of threads over processes.
maybe if you describe more of what you are doing, it could help. what operating system(s) are you using? Windows? OSX? Linux? BSD? what is your reason for choosing threads instead of processes (all those systems have both choices)?
please reply in this thread so everyone else can read both posts.