Python Forum
Socket connection and thread tracking...
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Socket connection and thread tracking...
#1
I've hit a bit of a brick wall, I'm sure there's a way around it, but my head is swimming in python syntax at the moment and I just can't see it!

My app is in two parts, a socket server that perversely sits on my clients, and a socket client that sits on my server.

When it's working ok the client logs onto all the servers, sends a code, the servers generate a data report and send it back. Lovely Jubly.

Except when a socket can't be opened. When the socket.connect(blah) fails for some reason the app stalls until the time out and things gradually start to fall apart.

So, I put all the data requests (socket connects) in new threads, so if one sits there the others don't get hung. This then needs some kind of marshalling as it's possible to go around the list and try to start a second thread on the same problem node/client.

So I wrote a class and some add/remove code to track open threads and make sure this didn't happen and now the app has such a large overhead I might as well me sending the data to a database, which is what I was trying to avoid in the first place!

I know I've asked a similar question before, and I have to admit I haven't checked responses yet, I thought I'd ramble while this was clear in my head.

If anyone out there has done anything similar, could you please let me know what you did in the end?

Outline:
Server/header raspberry pi
Clients raspberry pi
Number of clients/nodes 128
Data collection every 1 or 2 seconds. Could be increased with the server is working hard.
Data to be collected: core temperature, cpu load, memory stats, swap file stats, disk stats and some other bit of data which could be a one off call.

I'm going to write the database approach tonight, I'm very used to database work so this will be really easy, then try to gauge which will be the most heavy. Even with the database approach there will have to be some live communication to change timing, data required etc.

I don't need anything writing, I'm enjoying the learning, I just want to end up with the most sensible solution in place. While it's not mission critical I'd sooner do it right.

Many thanks.
Reply
#2
You seem to have many threads on this same subject. This is a violation of forum rules.
If you want to get attention to a post, bump (re-post) with a simple post to same thread.
Reply
#3
i'm not sure which thread to answer. but you should try better to resolve the connection failure. if you are really trying to have your program do other stuff while the connecting is trying to be made, then yeah, a separate thread or process is needed so the connection is being tried in parallel, regardless whether or not it will fail.

it's not clear what you are doing. is it on thread for each of many servers the client connects to?

i wrote a script a while back that connects to several (11 of them if i recall) of those web sites that shows you IP address. it connects to them all using processes (instead of threads). but it handles connection failures gracefully because there are almost always a few that fail. it waits until all processes are done which means the one that took the longest time affects the timing of the script, whether it succeeds or fails. i think i put a timeout in there because a successful connection could be the worst case and wait forever. connection failures get timed out by the system, but an established connection could, in theory, wait for years ... or longer.

the reason i made this connection to many servers is because many of them could be down at the same time. many of the cloud scripts i run need to know what IP address they will be seen as coming from so they can make various requests properly, like setting up cloud security group rules. i would not want to depend on just one server since all of them have had some down time (even Google).

i also have other scripts doing multiprocessing. i generally don't use threads. so i really can't help with the specific difficulties of threads over processes.

maybe if you describe more of what you are doing, it could help. what operating system(s) are you using? Windows? OSX? Linux? BSD? what is your reason for choosing threads instead of processes (all those systems have both choices)?

please reply in this thread so everyone else can read both posts.
Tradition is peer pressure from dead people

What do you call someone who speaks three languages? Trilingual. Two languages? Bilingual. One language? American.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  motion tracking script ig? k4ne 0 434 Dec-13-2023, 02:00 AM
Last Post: k4ne
  satellite tracking module barryjo 1 1,371 Mar-01-2022, 03:23 AM
Last Post: Larz60+
  Full Body Tracking and Recognition with OpenCV principemestizo 0 1,400 Oct-18-2021, 04:47 PM
Last Post: principemestizo
  Serial connection connection issue Joni_Engr 15 7,829 Aug-30-2021, 04:46 PM
Last Post: deanhystad
  Connect device using Visa TCP Socket connection d777py 1 3,338 Jan-08-2021, 05:08 PM
Last Post: d777py
  Error SQLite objects created in a thread can only be used in that same thread. binhduonggttn 3 15,386 Jan-31-2020, 11:08 AM
Last Post: DeaD_EyE
  Tracking 2 biggest objects in OpenCV Nup 0 1,665 Nov-12-2019, 11:08 AM
Last Post: Nup
  ATM Tracking Issue Zsapp01 1 1,973 Jun-04-2019, 07:20 PM
Last Post: Yoriz
  How to desing an app for tracking workload Cuz 4 2,760 Jan-08-2019, 10:45 AM
Last Post: Cuz

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020