Jun-28-2019, 08:13 PM
Looked for an answer on the forum but couldn't find one.
I use python pandas and xlswriter extensively and I like to load large dataframes into memory so I can run multiple operations on it and export to several different files (e.g. create multiple outputfiles typically CSV and excel that serve different purposes).
Currently at my company I have a remote desktop on a colo-archive with 1TB of RAM and I typically get close to maxing out that RAM. I like keeping all of the data in RAM to run multiple exports rather limiting the calc because it is so fast.
Let's say I am processing about 10TB per day in RAM, storing on 10TB of hard drive space, plus moving 100GB of data per day in and out of the colo-archive. CPU speed isn't a big deal currently as I am doing mostly pandas math.
Does anyone have any cost effective alternatives (e.g. refurb server, cloud, etc)? Everything I have looked at seems expensive.
I use python pandas and xlswriter extensively and I like to load large dataframes into memory so I can run multiple operations on it and export to several different files (e.g. create multiple outputfiles typically CSV and excel that serve different purposes).
Currently at my company I have a remote desktop on a colo-archive with 1TB of RAM and I typically get close to maxing out that RAM. I like keeping all of the data in RAM to run multiple exports rather limiting the calc because it is so fast.
Let's say I am processing about 10TB per day in RAM, storing on 10TB of hard drive space, plus moving 100GB of data per day in and out of the colo-archive. CPU speed isn't a big deal currently as I am doing mostly pandas math.
Does anyone have any cost effective alternatives (e.g. refurb server, cloud, etc)? Everything I have looked at seems expensive.