May-17-2018, 12:18 PM
It is nice that is working, but remember that it is really limited, you will need to adapt it to your case.
The np.empty function creates a numpy array without initialising the memory. I do it in this way because in the loop I will write every element, so filling it with 0 will be just a waste of cycles.
In that line I am creating an array of N_years x N_lat x N_lon dimensions, and to do this I need to pass np.empty the tuple
About the NetCDF, yes, for sure you can store this amount of data. I personally used HDF5 to store several Gb of data without any issue and there are nice bindings in python.
If you need to perform this kind of analysis you might like to see pandas, that is designed to work with these kind of massive data arrays.
The np.empty function creates a numpy array without initialising the memory. I do it in this way because in the loop I will write every element, so filling it with 0 will be just a waste of cycles.
In that line I am creating an array of N_years x N_lat x N_lon dimensions, and to do this I need to pass np.empty the tuple
(n_year, n_lat, n_lon) Remember that Quantity has dimension (n_days, n_lat, n_lon), so I just need to use the 2nd and 3rd dimensions of that array... and that is what Quantity.shape[1:3] does (remember that in python slices, the upper value is not reached, so 1:4 means 1, 2, 3)About the NetCDF, yes, for sure you can store this amount of data. I personally used HDF5 to store several Gb of data without any issue and there are nice bindings in python.
If you need to perform this kind of analysis you might like to see pandas, that is designed to work with these kind of massive data arrays.