I have 100 big nearly identical netcdf files. Each has 18 variables and ~22GB in size. The extents are not in the typical latlon rotation. The data in each file dates from 1850 to 2100.
I need to subset the data from 1970 to 2080 and crop the data to my study area's extent and extract individual variables. The size and rotation problem makes these tasks impossible in R, although I was able to rotate and crop a single layer from one of the files in R.
This would seem like a job for CDO (or NCO) but the program could not read the files, which I suspect was due to the rotation because the extent (-1.40625, 358.5938, 89.25846, 89.25846) is unusual (see below). The same goes for NCO.
How to handle these tasks either in R, CDO, NCO or Python? Specifically, how can I successfully rotate each ~22GB file such that they can be read by CDO or NCO.
>mynetcdfclass : SpatRaster dimensions : 64, 128, 602262 (nrow, ncol, nlyr)resolution : 2.8125, 2.789327 (x, y)extent : -1.40625, 358.5938, -89.25846, 89.25846 (xmin, xmax, ymin, ymax)coord. ref. : lon/lat WGS 84 (CRS84) (OGC:CRS84) sources : fwi_day_CanESM5_historical_r1i1p1f1_gn_full-outputs.nc:fwi (60225 layers) fwi_day_CanESM5_historical_r1i1p1f1_gn_full-outputs.nc:ffmc (60225 layers) fwi_day_CanESM5_historical_r1i1p1f1_gn_full-outputs.nc:dmc (60225 layers) ... and 15 more source(s)varnames : fwi ffmc dmc ...names : fwi_1, fwi_2, fwi_3, fwi_4, fwi_5, fwi_6, ... unit : 1, 1, 1, 1, 1, 1, ... ```