+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: General Coding Help (https://python-forum.io/forum-8.html)
+--- Thread: Need advice on pushing data to a server API (/thread-27952.html)
Need advice on pushing data to a server API - msoultan - Jun-28-2020
Hi,
I'm working on a project where I need to upload data from about 600 clients to a centralized server API, and I'm trying to figure out the best way to go about it. Initially what I was going to do was have the client send the data to the API, the API would return a checksum that it received the data, and then the API server would continue processing after responding to the client and commit the data to the database. Then the next time the client would go to upload data, it would first check to see if the previous set of data was committed successfully to the DB, and if so, push the next batch (if there was any data to send). This all seemed nice on paper until I tried to implement it and realized that once an API returns the data, it essentially ends processing (at least that's the case for Flask).
Now, if you're curious why I'm doing it this way, the reason is because I didn't want the client to sit and wait while the server committed the data, especially if there was some sort of issue or delay with the DB server or it was down, and that would mean the client would be sitting for a little bit. Time isn't *horribly* critical with my client, so I guess I could have it wait 5 seconds, but not much longer.
Since I hit the stumbling block above, I thought the other option would be for the client to spawn a thread and do the upload and commit verification in the thread so that it doesn't affect the main loop... but honestly, I'm really not sure if I need to do that. The reason is because I don't know how fast/slow the DB commit would be. It would be about 100k worth of data so it's not a ton of data by any means, so I'm guessing it should be sub-second, but that's merely a guess because I don't have much previous experience in this. If you're curious, I'll be using AWS for the API/DB.
Any thoughts and suggestions would be much appreciated on how others would approach this problem.
Thanks!
Mike