Selecting netcdf data files

[Jodman92]

Hi there,

I'm trying to adapt some code I have been given for running climate simulations to work with Python3.7 on windows 10 using Anaconda (code was written in linux).

 #select netcdf data file
fname_data = 'C:\Users\Jodman\Documents\data\' + model + '\' + exp_id + '\' + variable + '\' + variable + '_day_' + model + '_' + exp_id + '_r1i1p1_20051201-20991230.nc'

#load data file as cube
cube = iris.load(fname_data)

The code is trying to select different climate models and various variables as shown below:

 #define model and experiment of choice
model = 'HadGEM2-CC' #HadGEM2-CC or HadGEM2-ES
exp_id = 'rcp45' #rcp45 or rcp85
variable = 'ua' # ua, tas or zg
if(variable == 'tas'): unit = 'K' 
if(variable == 'ua'): unit = 'm/s' 
if(variable == 'zg'): unit = 'm'
sel_lev = 5000 #Pa #5000 or 1000

I'm getting SyntaxError: unexpected character after line continuation character.

The code in linux looked like this below.

#select netcdf data file
fname_data = '/nfs/jack/earataj/Jody/data/' + model + '/' + exp_id + '/' + variable + '/' + variable + '_day_' + model + '_' + exp_id + '_r1i1p1_20051201-20991230.nc' 

I've tried to adapt the windows script by changing the / to \ but have come to a dead end.

would appreciate any help!

[Larz60+]

could you please pose entire, unmodified error traceback
one thing, since you are using python 3.7, you can use f-string,so:

fname_data = 'C:\Users\Jodman\Documents\data\' + model + '\' + exp_id + '\' + variable + '\' + variable + '_day_' + model + '_' + exp_id + '_r1i1p1_20051201-20991230.nc'

can be better written:

fname_data = f'C:/Users/Jodman/Documents/data/{model}/{exp_id}/{variable}/{variable}{_day_}{model}_{exp_id}_r1i1p1_20051201-20991230.nc'

also use / in place of \ this will avoid collision with \t, \r, \n, etc.

[Jodman92]

Thanks for the reply!

Used the adapted code and now get this

File "<ipython-input-78-a21ccddadee0>", line 2
    fname_data = f'C:/Users/Jodman/Documents/data/{model}/{exp_id}/{variable}/{variable}{_day_}{model}_{exp_id}_r1i1p1_20051201-20991230.nc
                                                                                                                                           ^
SyntaxError: EOL while scanning string literal

---

Sorry for the double post, can't find the edit option.

This is actually the new error, missed the mark at the end.

NameError                                 Traceback (most recent call last)
<ipython-input-99-812a32deeb5e> in <module>()
      1 #select netcdf data file
----> 2 fname_data = f'C:/Users/Jodman/Documents/data/{model}/{exp_id}/{variable}/{variable}{_day_}{model}_{exp_id}_r1i1p1_20051201-20991230.nc'
      3 
      4 #load data file as cube
      5 cube = iris.load(fname_data)

NameError: name '_day_' is not defined

[Jodman92]

Managed to fix that error now appear to have a memory issue.

#these values can be changed to select a different Europe box
eurlon = iris.Constraint(longitude=lambda l: -10<l<40)  # the band of longitudes we want
eurlat = iris.Constraint(latitude=lambda l: 48<l<75)  # the band of longitudes we want

#average over longitude range selected above
dum = cube_djf[0].extract(eurlon).collapsed('longitude', iris.analysis.MEAN)

#compute weights for latitude area weighted mean
weights = cosine_latitude_weights(dum.extract(eurlat))

#average over latitude range selected above
cube_djf_eur = dum.extract(eurlat).collapsed('latitude', iris.analysis.MEAN, weights = weights)

cube_djf_zm = cube_djf.collapsed('longitude', iris.analysis.MEAN)

MemoryError: Failed to realise the lazy data as there was not enough memory available.
The data shape would have been (8580, 8, 20) with dtype('float32').
 Consider freeing up variables or indexing the data before trying again.

Any ideas, is 8gb of ram just too little for large datasets?

[Larz60+]

I run with 32 gig, so not sure. I expect that some large databases could easily go beyond 8 gigabytes.