i am creating a script intended to fetch some large sets of data. this script is intended for public use. but it is very heavy on memory usage. part of this seems to be that underlying tools work with bulk data. that is they provide or expect all the data in one big list rather than work with a little at a time. i'm looking around for coding techniques to minimize memory use, such as sort() to sort in place instead of sorted() which makes a 2nd copy before the 1st is deleted. some argument combinations have used upwards of 12 GBs of memory. got any nice links?
If you're using 12GB of memory and don't have that much installed, then some sort of paging will have to be used.
This will always be inefficient no matter what you do. Only real solution is to add memory.
12GB is not large for today's computers.
I haven't upgraded for several years, and have 32GB installed. Next machine will have 128GB.
laptops tend to be smaller. i have found a few at 64GB, just one at 128GB (and reason to believe Linux can't run on it), and mostly 32GB.
my goal is to learn to be smarter in programming for the goal of keeping memory smaller, avoiding duplicates of data is a major way to achieve this. for exampled, the sorted() function make a copy of the list you are sorting. you will have 2 copies of the data for the instant in time until you del the original. the program could die there, or cause a lot of swapping of everything. using (list).sort() instead always has just the one copy.
i know this one, but i want to learn other techniques.
I think the first step could be to use a memory profiler.
e memory profiler sounds like an interesting tool. is it going to tell me about ways to organize my logic to make best use of Python? for the case of sorting would it tell me to sort in place? i'm looking to know of other things i should know any how python can use excesses of memory because of how you use it.