Jun-07-2020, 10:34 AM
Hey guys!
I'm fairly new to python, but I've created a script that tracks users attacks in an online game through the game's API. Due to the pace of the game, and the way the API is built I unfortunately have to make frequent calls - about 6 per minute, per user - to get the results I want with my script. The script takes the users API and userid as arguments, and continually runs a while-loop every 6/7 seconds. The users attacks are fed into a MySQL-database, and I do some other calculations with the JSON I recieve from the API. The JSON contains profile information, and the last 100 attacks in an unordered list.
The current iteration of the script works (yay!), but when my clan this weekend did a stress test of the script (20-30 users), it faltered during the most intensive times. Seeing as I want to open it up to the community, I need it to be able to handle that amount without breaking a sweat. As its my first real Python program the code is pretty "dirty" and probably lacks a lot (all?) best practices.
The issue I faced seems to be data leaking over between instances of the running script. I believe this is either because I didn't store the json-results as instances class-objects - or it's because the MySQL couldn't handle all of the I/O that I put it under. If theres something I missed I'd like to hear that too. I want to fix both of these with a new version of the program, and I'm pondering how to best build version 2 to be scaleable.
I'm thinking having 1 script running per user creates a lot of overhead with all libraries being loaded for each one. It's also caused me a headache with regards to allowing users to start/stop their instance of the script at will. Instead I think I want to write a single program with this workflow:
Am I onto something, or should I be doing something entirely different?
I'm not posting code as I'm not really looking for code in return. I'm hoping for some tips or guidelines on how I can make this work in a scaleable way :) I hope thats allright with you guys
Any tips is very appreciated!
Python v. 3.8.
I'm fairly new to python, but I've created a script that tracks users attacks in an online game through the game's API. Due to the pace of the game, and the way the API is built I unfortunately have to make frequent calls - about 6 per minute, per user - to get the results I want with my script. The script takes the users API and userid as arguments, and continually runs a while-loop every 6/7 seconds. The users attacks are fed into a MySQL-database, and I do some other calculations with the JSON I recieve from the API. The JSON contains profile information, and the last 100 attacks in an unordered list.
The current iteration of the script works (yay!), but when my clan this weekend did a stress test of the script (20-30 users), it faltered during the most intensive times. Seeing as I want to open it up to the community, I need it to be able to handle that amount without breaking a sweat. As its my first real Python program the code is pretty "dirty" and probably lacks a lot (all?) best practices.
The issue I faced seems to be data leaking over between instances of the running script. I believe this is either because I didn't store the json-results as instances class-objects - or it's because the MySQL couldn't handle all of the I/O that I put it under. If theres something I missed I'd like to hear that too. I want to fix both of these with a new version of the program, and I'm pondering how to best build version 2 to be scaleable.
I'm thinking having 1 script running per user creates a lot of overhead with all libraries being loaded for each one. It's also caused me a headache with regards to allowing users to start/stop their instance of the script at will. Instead I think I want to write a single program with this workflow:
- API/userid of active users are pulled from the MySQL. This can be done once every minute or something
- Asyncronous HTTP-requests (aiohttp?) to the API
- Handling errors from the API
- Profile information is stored in instanced class-objects.
- Attack information is stored in instanced class-objects
- Methods inside the classes handle calculations
- Insert/update new attacks and userdata into MySQL
- Add "inserted" into the class objects so I don't have to use "INSERT IGNORE" or tax the MySQL DB unessecarily
- Destroy attack-class-objects that are no longer in the JSON (i.e. old attacks) to avoid ever-increasing memory usage
Am I onto something, or should I be doing something entirely different?
I'm not posting code as I'm not really looking for code in return. I'm hoping for some tips or guidelines on how I can make this work in a scaleable way :) I hope thats allright with you guys
Any tips is very appreciated!
Python v. 3.8.