If you discover any possible issues with the CMIP data located at NCI, please report it here or by contacting NCI or climate help (contact info on wiki homepage). These will be added to the Known Issues and Errata page.
Some files/years seem to be missing from CMIP5 data in both the DRSv2 three and the paths returned by search_replica.py when searching for alternative file locations. The issues I ran into are:
1. CMCC-CESM , rcp85 , monthly , variable: tas , r1i1p1 :: data for the years/months 209601 to 210012 are missing
2. HadGEM2-CC , rcp85 , daily , variable: tasmax , r1i1p1 :: no data for 20051201 to 20991231
3. HadGEM2-ES , rcp85 , daily , variable: hfls , r1i1p1 :: no data for 20951201 to 21001231
Would it please be possible to add these missing data?
the CMCC-CESM data finishes at 2095 included, this seems to apply to all variables
I'm downloading these files
I'm downloading these files, this is done, but I've found out that the two missing files were listed in the older version, they were in fact the only two. I'll see if we can delete the older version, I've updated the db and delete it from there at least.
While I'm updating the database now, I won't make it copy it to the official version until I finish other downloads I'm doing, so it won't be there until Monday. I think the DRSv2 gets updated in the weekend too
It seems I ran again in the issue with some HadGEM2-CC data not being available (rcp85 , daily , variable: tasmax , r1i1p1, v20120531), there are no data for 20051201 to 20991231 in
if you use the search_replica.py script with the -a flag it will return all available versions not just the latest, so if there are two options for the latest you'll see both of them. DRSv2 will have to make a choice between latests too and sometimes is one which is broken. The dataset should be cleaned up for this to be avoided but it's not clear whose responsibility it is. In this case I'll delete this so DRSv2 won't pick it up anymore at its next update during the weekend.
I'm fixing the same issue for the variables: ta, tasmin, rsds, psl, rhs, ua, va, zg, hus and hur. I've deleted the incomplete versions from unofficial and from the db and I'm downloading the full ensembles from the new ceda server (download is very slow)
Thanks for your help with downloading the missing files.
re 1) But daily tasmax data from CMCC-CESM is available until including year 2100, see here /g/data1/ua6/DRSv2/CMIP5/CMCC-CESM/rcp85/day/atmos/r1i1p1/tasmax/v20120730
so I would expect that also monthly tas should be available for these years...?
All the available files have been downloaded , unfortunately although the daily files are available until year 2100, the monthly files stop consistently for all variables at 2095 included.
Looking further on the CCCMA web site, from whence we obtain the "CanadaExtremeIndices" data, I have located the previous dataset ("CMIP5_old") that contains the twelve (12) files that Laurence Garcia-Villada reported with errors.
I have checked the number of years in these new files and compared them to the old files, and they (new files) all contain at least 146 years, some 156, some 163.
From Claire:
Please note if using this data for publication that this is an older version of the CanadaExtremeIndices data, not the current version. The Bureau may decide to update to the more recent dataset at some stage.
i.e., with "_mon_" between "tnnETCCDI" and "CSIRO-Mk3-6-0"; and I have checked that (it had 144 years) and have re-downloaded a replacement file which has 156 years.
this is not maintained by any of us, these are links that Nadja created previously, I believe she cannot maintain them herself either. There should be a copy in /g/data1/w35/ which might have been updated more recently. Also currently ua6 which is the project the links would refer to is being transferred to a new disk by NCI and we cannot make any changes to this collection while this is happening (Claire Carouge sent an e-mail the other day about this).
Since soon the same transfer will happen for ua8 and I suggest you use the /g/data1/w35/CMIP5_Archive or try one of the other available tools as an alternative.
We'll have to delete the CMIP5_Archive from ua8 in the next few months, it will stay in w35.
Finally this are files from the CSIRO Mk model so they should be officially published by NCI (as it's the ACCESS model). Any model published by NCI is available under /g/data1/ua6/authoritative.
There's a CMIP5 induction on the NCI training site training.nci.org.au that has all this information
21 Comments
Markus Donat
Some files/years seem to be missing from CMIP5 data in both the DRSv2 three and the paths returned by search_replica.py when searching for alternative file locations. The issues I ran into are:
1. CMCC-CESM , rcp85 , monthly , variable: tas , r1i1p1 :: data for the years/months 209601 to 210012 are missing
2. HadGEM2-CC , rcp85 , daily , variable: tasmax , r1i1p1 :: no data for 20051201 to 20991231
3. HadGEM2-ES , rcp85 , daily , variable: hfls , r1i1p1 :: no data for 20951201 to 21001231
Would it please be possible to add these missing data?
Paola Petrelli
While I'm updating the database now, I won't make it copy it to the official version until I finish other downloads I'm doing, so it won't be there until Monday. I think the DRSv2 gets updated in the weekend too
Markus Donat
Hi,
It seems I ran again in the issue with some HadGEM2-CC data not being available (rcp85 , daily , variable: tasmax , r1i1p1, v20120531), there are no data for 20051201 to 20991231 in
/g/data1/ua6/DRSv2/CMIP5/HadGEM2-CC/rcp85/day/atmos/r1i1p1/tasmax/v20120531
or
/g/data1/ua6/unofficial-ESG-replica/tmp/tree/cmip-dn1.badc.rl.ac.uk/thredds/fileServer/esg_dataroot/cmip5/output1/MOHC/HadGEM2-CC/rcp85/day/atmos/day/r1i1p1/v20120531/tasmax
My script suggest I was accessing those files in Dec 2016, so it seem they may have got lost since then?
Thanks,
Markus
Paola Petrelli
Hi Markus,
ceda changed their server so the data downloaded in December is in
/g/data1/ua6/unofficial-ESG-replica/tmp/tree/esgf-data1.ceda.ac.uk/thredds/fileServer/esg_dataroot/cmip5/output1/MOHC/HadGEM2-CC/rcp85/day/atmos/day/r1i1p1/v20120531/tasmax
if you use the search_replica.py script with the -a flag it will return all available versions not just the latest, so if there are two options for the latest you'll see both of them. DRSv2 will have to make a choice between latests too and sometimes is one which is broken. The dataset should be cleaned up for this to be avoided but it's not clear whose responsibility it is. In this case I'll delete this so DRSv2 won't pick it up anymore at its next update during the weekend.
Paola
Paola Petrelli
I'm fixing the same issue for the variables: ta, tasmin, rsds, psl, rhs, ua, va, zg, hus and hur. I've deleted the incomplete versions from unofficial and from the db and I'm downloading the full ensembles from the new ceda server (download is very slow)
Markus Donat
Thank you Paola!
Markus Donat
Thanks for your help with downloading the missing files.
re 1) But daily tasmax data from CMCC-CESM is available until including year 2100, see here /g/data1/ua6/DRSv2/CMIP5/CMCC-CESM/rcp85/day/atmos/r1i1p1/tasmax/v20120730
so I would expect that also monthly tas should be available for these years...?
Paola Petrelli
All the available files have been downloaded , unfortunately although the daily files are available until year 2100, the monthly files stop consistently for all variables at 2095 included.
Paola Petrelli
Daily tas FGOALS-s2 r1i1p1 historical has two missing years: 1973, 1974.
There's no available version of this online, not even among replicas, so this cannot currently be fixed
Laurence Garcia-Villada
There are missing years in this file:
/g/data1/ua6/CanadaExtremeIndices/ftp.cccma.ec.gc.ca/data/climdex/CMIP5/historical/EC-EARTH/r12i1p1/tnxETCCDI_mon_EC-EARTH_historical_r12i1p1_185001-201212.nc
When I use the cdo -showyear command, the output only reveals these years:
[lgv561@raijin5 r12i1p1]$ cdo showyear tnxETCCDI_mon_EC-EARTH_historical_r12i1p1_185001-201212.nc
1850 1851 1852 1853 1854 1855 1856 1857 1858 1859 1860 1861 1850
All the other files in this model's realisation are fine.
Claire Trenham
Received and problem confirmed. Lawson Hanson will update this dataset to the newest version.
Internal reference: RT#94760
Laurence Garcia-Villada
Sorry for not being more thorough earlier, but I found these files are missing years as well.
I have not searched through every index/model just those I need.
/g/data1/ua6/CanadaExtremeIndices/ftp.cccma.ec.gc.ca/data/climdex/CMIP5/historical/CSIRO-Mk3-6-0/r4i1p1/tnnETCCDI_CSIRO-Mk3-6-0_historical_r4i1p1_185001-200512.nc
/g/data1/ua6/CanadaExtremeIndices/ftp.cccma.ec.gc.ca/data/climdex/CMIP5/historical/CSIRO-Mk3-6-0/r1i1p1/tn90pETCCDI_mon_CSIRO-Mk3-6-0_historical_r1i1p1_185001-200512.nc
/g/data1/ua6/CanadaExtremeIndices/ftp.cccma.ec.gc.ca/data/climdex/CMIP5/historical/CMCC-CM/r1i1p1/tx90pETCCDI_mon_CMCC-CM_historical_r1i1p1_185001-200512.nc
/g/data1/ua6/CanadaExtremeIndices/ftp.cccma.ec.gc.ca/data/climdex/CMIP5/historical/CNRM-CM5/r5i1p1/tn10pETCCDI_mon_CNRM-CM5_historical_r5i1p1_185001-200512.nc
/g/data1/ua6/CanadaExtremeIndices/ftp.cccma.ec.gc.ca/data/climdex/CMIP5/historical/GFDL-CM3/r1i1p1/tn90pETCCDI_mon_GFDL-CM3_historical_r1i1p1_186001-200512.nc
/g/data1/ua6/CanadaExtremeIndices/ftp.cccma.ec.gc.ca/data/climdex/CMIP5/historical/GFDL-CM3/r3i1p1/txnETCCDI_mon_GFDL-CM3_historical_r3i1p1_186001-200512.nc
/g/data1/ua6/CanadaExtremeIndices/ftp.cccma.ec.gc.ca/data/climdex/CMIP5/historical/GISS-E2-R/r6i1p1/tn90pETCCDI_mon_GISS-E2-R_historical_r6i1p1_185001-200512.nc
/g/data1/ua6/CanadaExtremeIndices/ftp.cccma.ec.gc.ca/data/climdex/CMIP5/historical/HadGEM2-ES/r4i1p1/tn90pETCCDI_mon_HadGEM2-ES_historical_r4i1p1_186001-200512.nc
/g/data1/ua6/CanadaExtremeIndices/ftp.cccma.ec.gc.ca/data/climdex/CMIP5/historical/MIROC-ESM-CHEM/r1i1p1/tx10pETCCDI_mon_MIROC-ESM-CHEM_historical_r1i1p1_185001-200512.nc
/g/data1/ua6/CanadaExtremeIndices/ftp.cccma.ec.gc.ca/data/climdex/CMIP5/historical/MIROC5/r2i1p1/dtrETCCDI_mon_MIROC5_historical_r2i1p1_185001-201212.nc
/g/data1/ua6/CanadaExtremeIndices/ftp.cccma.ec.gc.ca/data/climdex/CMIP5/historical/MPI-ESM-LR/r3i1p1/txnETCCDI_mon_MPI-ESM-LR_historical_r3i1p1_185001-200512.nc
Cheers,
Laurence
Laurence Garcia-Villada
Appologies, this one too
/g/data1/ua6/CanadaExtremeIndices/ftp.cccma.ec.gc.ca/data/climdex/CMIP5/historical/HadGEM2-ES/r3i1p1/rx5dayETCCDI_mon_HadGEM2-ES_historical_r3i1p1_186001-200512.nc
Claire Trenham
From Lawson Hanson:
Lawson Hanson
I just noticed that the file referenced as:
/g/data1/ua6/CanadaExtremeIndices/ftp.cccma.ec.gc.ca/data/climdex/CMIP5/historical/CSIRO-Mk3-6-0/r4i1p1/tnnETCCDI_CSIRO-Mk3-6-0_historical_r4i1p1_185001-200512.nc
was probably meant to be:
/g/data1/ua6/CanadaExtremeIndices/ftp.cccma.ec.gc.ca/data/climdex/CMIP5/historical/CSIRO-Mk3-6-0/r4i1p1/tnnETCCDI_mon_CSIRO-Mk3-6-0_historical_r4i1p1_185001-200512.nc
i.e., with "_mon_" between "tnnETCCDI" and "CSIRO-Mk3-6-0"; and I have checked that (it had 144 years) and have re-downloaded a replacement file which has 156 years.
Best regards,
Lawson
Laurence Garcia-Villada
Thank you Lawson
Laurence Garcia-Villada
Hi appologies for opening this up again but these files also have the same issue:
/g/data1/ua6/CanadaExtremeIndices/ftp.cccma.ec.gc.ca/data/climdex/CMIP5/historical/HadGEM2-CC/r1i1p1/tnnETCCDI_mon_HadGEM2-CC_historical_r1i1p1_186001-200512.nc
/g/data1/ua6/CanadaExtremeIndices/ftp.cccma.ec.gc.ca/data/climdex/CMIP5/historical/CESM1-BGC/r1i1p1/tn90pETCCDI_mon_CESM1-BGC_historical_r1i1p1_185001-200512.nc
/g/data1/ua6/CanadaExtremeIndices/ftp.cccma.ec.gc.ca/data/climdex/CMIP5/historical/IPSL-CM5A-MR/r1i1p1/dtrETCCDI_mon_IPSL-CM5A-MR_historical_r1i1p1_185001-200512.nc
Lawson Hanson
I have re-downloaded the three files specified, checked them; they appear to be OK now.
Best regards, Lawson
Claire Trenham
This version of the Canada Extreme Indices data scheduled to be replaced with the newer version (currently in ua6/processed/climdex) on 23/3/2017.
Laurence Garcia-Villada
Hi Claire and Lawson,
I've found missing CMIP5 'tos' files for the CSIRO-Mk3L-1-2 model:
/g/data1/ua8/tmp/CMIP5_Archive/historical/Omon/tos/CSIRO-Mk3L-1-2/r1i2p1/tos_Omon_CSIRO-Mk3L-1-2_historical_r1i2p1_185101-200512.nc
/g/data1/ua8/tmp/CMIP5_Archive/historical/Omon/tos/CSIRO-Mk3L-1-2/r2i2p1/tos_Omon_CSIRO-Mk3L-1-2_historical_r2i2p1_185101-200512.nc
/g/data1/ua8/tmp/CMIP5_Archive/historical/Omon/tos/CSIRO-Mk3L-1-2/r3i2p1/tos_Omon_CSIRO-Mk3L-1-2_historical_r3i2p1_185101-200512.nc
Could you please look into these
Cheers,
Laurence
Paola Petrelli
Hi Laurence,
this is not maintained by any of us, these are links that Nadja created previously, I believe she cannot maintain them herself either. There should be a copy in /g/data1/w35/ which might have been updated more recently. Also currently ua6 which is the project the links would refer to is being transferred to a new disk by NCI and we cannot make any changes to this collection while this is happening (Claire Carouge sent an e-mail the other day about this).
Since soon the same transfer will happen for ua8 and I suggest you use the /g/data1/w35/CMIP5_Archive or try one of the other available tools as an alternative.
We'll have to delete the CMIP5_Archive from ua8 in the next few months, it will stay in w35.
Finally this are files from the CSIRO Mk model so they should be officially published by NCI (as it's the ACCESS model). Any model published by NCI is available under /g/data1/ua6/authoritative.
There's a CMIP5 induction on the NCI training site training.nci.org.au that has all this information
Cheers, Paola