Python Forum
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Handling Many Threads.
#1
Hi folks. I'm not quite sure how I'm going to do this yet, so if you think I'm barking up the wrong tree please say so.

I will soon have to gather a small amount of data from 120 nodes/servers, (micro pcs) back to a header node, also a micro pc. This needs to be done every 10 seconds.

I've thought of files, (see another post on file access exclusivity) I've thought of sockets, I've thought of serial, I've thought of single wire solutions.

Serial would be time consuming, but great fun to set up, single wire could be almost impossible, files will cause a huge amount of unneeded disk activity, so sockets do seem the best at first glance. But, it's the most sockets I'll have ever worked with, so advice please...

I envisage the header sending out a request every 10 seconds "Data Please!", the clients will do their calculations and send back "Here you go". That bit is simple, but how is the header going to deal with this flood of data?

So 120 packets of data are just hitting the server, so I create a thread as each packet arrives and process as needed, update variables, etc. I'm quite happy with this, but where do I control the bottleneck?

The server only has 4 cores and it's not lightning fast. Do I restrict the number of open connections to 4 and have the clients retry/retry, or do I allow 120 open connections and have data queue on the server, will that work?

I really need the overhead on the clients to be the minimum possible as they're doing loads off other stuff that I want to interrupt as little as possible.

Ideas? Observations? Criticism? Any feedback welcome.

Thank you.
Reply
#2
I highly suggest watching this video:
https://www.youtube.com/watch?v=MCs5OvhV9S4
It is one of David Beazley's best videos, and early on, he creates example code that handles 25,000 requests per second
Reply
#3
Holy crap, I'm going back to VB!

Thanks Larz.
Reply
#4
I'd suggest using Twisted for pretty much anything networking related.
It makes these things much simpler, and it takes care of all the weird edge cases and quirks that come up when dealing with raw sockets.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
Star python exception handling handling .... with traceback mg24 3 1,215 Nov-09-2022, 07:29 PM
Last Post: Gribouillis

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020