Memory Use in array

[DeaD_EyE]

Maybe it's a xyproblem.
Why do you need everything in memory?
Why does your query have no filter nor limit?
What do you want to do with the data?
Can you do this in a iterative way?

[fakka]

Maybe it's a xyproblem.
Why do you need everything in memory?
Why does your query have no filter nor limit?
What do you want to do with the data?
Can you do this in a iterative way?

Just how the product works. KeyValue keystore.
So the idea here is to fetch all the keys (this is just the address of the data set) so its the ~1.5GB of the 50GB real dataset.
You push that 1.5Gb of keys into an array.
Then splice/chunk that array into (n) smaller arrays.
You then process those array's in parallel using something like numpy.

This way to you (for example) do 5000 OPS/sec updates instead of 50.

The limiting factor here is the memory. Not understanding why the array blows out to many GB - when your only fetching the KEY and not the data.