Python Forum

Hi All,

Forgive my ignorance on the realities of networking but...how can I support many users simultaneously with my (cloud-hosted) scraping app ? I have written a demo that supports only a single user:

After user input (via browser) the script does several cycles of scraping and crunching - which usually takes 5-10 mins. (Users will get the output by email). This is fine for a single user, but how could I support 100 or even 1000 users simultaneously? A separate script running for every single user?! :-O

Queue? - not really feasible because users wont wait for their report for very long
Multi-thread? - this will slow things down (each thread will take longer than it would as a single thread - right?)

...so I am left wondering if I have to have one script running for every single user ?!

Somewhere else I saw a reference to Twisted, but I am not sure how this fits.

There must be other web apps or cloud-based services that have to deliver serious real-time data crunching to users. How do they do that?

Thanks a lot for any help in advance - I really appreciate any advice.

When your processing takes 15 minutes, the user have to wait 15 minutes.
The program structure can be following:

• Manager process, which receives the tasks and sending them to a free worker process
  
• Worker processes which are started by Manager
  
• Asynchronous WebInterface which is doing the communication between Manager <> Client.
  
• E-Mail Client which sends finished work to the user, which can be accessed for a duration on the WebServer.
  

This are my thoughts. Maybe you can make it simpler, but I guess you'll end in a big message queue forward and backward.

Thanks for that. I appreciate it.

In order to support 1000 simultaneous 15-min requests, I am going to need 1000 workers - right?

Joe

Or faster machines. Or you optimize the function which is doing the work.