High RAM machines or cloud services

[trader144]

Looked for an answer on the forum but couldn't find one.

I use python pandas and xlswriter extensively and I like to load large dataframes into memory so I can run multiple operations on it and export to several different files (e.g. create multiple outputfiles typically CSV and excel that serve different purposes).

Currently at my company I have a remote desktop on a colo-archive with 1TB of RAM and I typically get close to maxing out that RAM. I like keeping all of the data in RAM to run multiple exports rather limiting the calc because it is so fast.

Let's say I am processing about 10TB per day in RAM, storing on 10TB of hard drive space, plus moving 100GB of data per day in and out of the colo-archive. CPU speed isn't a big deal currently as I am doing mostly pandas math.

Does anyone have any cost effective alternatives (e.g. refurb server, cloud, etc)? Everything I have looked at seems expensive.

[Larz60+]

Man what I could do with that much RAM!
Is that really RAM, or solid state drive?

[micseydel]

You may want to look into Spark. What you're trying to do sounds like vertical scaling, which is often considered expensive, compared to horizontal scaling.

If you want to consider horizontal scaling, checkout Spark.