Python Forum
Need advice on pushing data to a server API
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Need advice on pushing data to a server API
#1
Hi,

I'm working on a project where I need to upload data from about 600 clients to a centralized server API, and I'm trying to figure out the best way to go about it. Initially what I was going to do was have the client send the data to the API, the API would return a checksum that it received the data, and then the API server would continue processing after responding to the client and commit the data to the database. Then the next time the client would go to upload data, it would first check to see if the previous set of data was committed successfully to the DB, and if so, push the next batch (if there was any data to send). This all seemed nice on paper until I tried to implement it and realized that once an API returns the data, it essentially ends processing (at least that's the case for Flask).

Now, if you're curious why I'm doing it this way, the reason is because I didn't want the client to sit and wait while the server committed the data, especially if there was some sort of issue or delay with the DB server or it was down, and that would mean the client would be sitting for a little bit. Time isn't *horribly* critical with my client, so I guess I could have it wait 5 seconds, but not much longer.

Since I hit the stumbling block above, I thought the other option would be for the client to spawn a thread and do the upload and commit verification in the thread so that it doesn't affect the main loop... but honestly, I'm really not sure if I need to do that. The reason is because I don't know how fast/slow the DB commit would be. It would be about 100k worth of data so it's not a ton of data by any means, so I'm guessing it should be sub-second, but that's merely a guess because I don't have much previous experience in this. If you're curious, I'll be using AWS for the API/DB.

Any thoughts and suggestions would be much appreciated on how others would approach this problem.

Thanks!

Mike
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  In SQL Server, mix data types. shiv11 0 854 Sep-21-2022, 12:50 PM
Last Post: shiv11
  How to modify python script to append data on file using sql server 2019? ahmedbarbary 1 1,177 Aug-03-2022, 06:03 AM
Last Post: Pedroski55
  How to take the tar backup files form remote server to local server sivareddy 0 1,871 Jul-14-2021, 01:32 PM
Last Post: sivareddy
  How to save Python Requests data sent to server? RedLeonard 5 4,799 Jul-05-2020, 10:33 AM
Last Post: RedLeonard
  Save Arduino data in mysql server via raspberrypi rithikvg 1 3,348 Jun-24-2020, 10:59 PM
Last Post: Larz60+
  Support required to data log in SQL server single table via raspbeery pi rithikvg 8 4,311 Jun-23-2020, 05:23 PM
Last Post: buran
  Get Python json data i n SQL Server Djin 1 1,980 May-13-2020, 05:56 AM
Last Post: ndc85430
  Moving large amount of data between MySql and Sql Server using Python ste80adr 4 3,325 Apr-24-2020, 01:24 PM
Last Post: Jeff900
  Beginner needing advice with data files JFI2019 2 2,158 Nov-06-2019, 04:56 PM
Last Post: JFI2019

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020