S1: Diagnosis and Metrics in Climate Models

Mean systematic errors of 500 hPa geopotential height fields (shading in dam) for winters (December–February) of the period 1962–2005 and various versions of the ECMWF model: (a) 29R2, (b) 30R1, (c) 31R1, (d) 32R1, (e) 32R3 and (f) 33R1. Also shown are mean fields (contours) obtained from a combination of ERA-40 (1962–2001) and operational ECMWF analysis data (2002–2005). Mean systematic errors significant at the 95% confidence level are hatched. Notice that negative values are contoured (as well as shaded). (From Jung et al. 2010; Q. J. R. Meteorol. Soc. 136: 1145–1160)

Principal investigators: Prof. Valerio Lucarini (Universität Hamburg), Prof. Thomas Jung (MARUM/AWI)


The hypothesis underlying the proposed Collaborative Research Centre is that a more consistent representation of parameterised processes and numerical formulations will result in better climate models. To test this hypothesis, therefore, it is critical to thoroughly evaluate the influence of enhanced model formulations, developed in the other project areas, on the realism of climate models. Here, we will focus on two next-generation climate models developed and used by the German climate research community, namely ICON-a/ICON-o and ECHAM6/FESOM.

Efforts towards developing a new generation of energetically consistent models require the availability of tools that can help to identify short-comings and quantify progress (e.g. Figure 1). Therefore, we will develop and apply new diagnostics and metrics that help us to understand the pathways of energy transport and energy conversion, analyse the energetic consistency of existing models and formulate measures taken up by the other subprojects to advance the energetic consistency and quality of parameterisations of climate models. Furthermore, we will develop effective metrics and diagnostics that can be used to quantify the realism of models and unravel the origin of model error. Metrics describing systematic model error in atmosphere, ocean and sea ice using some of the most advanced observational data sets for reference, will be elaborated. Furthermore, dedicated metrics will be developed that capture important weather and climate phenomena resulting from instabilities such as extratropical storms, Euro-Atlantic blocking, or monsoon dynamics. Finally, we will construct metrics able to analyse consistently a large class of extremes occurring across a wide range of spatial and temporal scales.

An important aspect will be the practical implementation of new tools that can be used for model evaluation. We will build on the newly developed Earth System Model Validation Tool (ESMValTool)—a community diagnostic and performance metrics tool for routine evaluation of climate models. By doing so, we can also benefit from other community efforts in developing evaluation tools. Moreover, we can add our new metrics and diagnostics to ESMValTool thereby increasing the applicability and international visibility—and hence the impact—of the research coming out of this subproject.

To aid effective evaluation of the model development activities, coming out of the other research areas, and to guide further efforts, we will design protocols for numerical experimentation, including common model setups, together with the PIs from subproject S2. These protocols will allow direct comparability between different modelling experiments, ensure availability of model output needed to compute all relevant metrics and diagnostics and provide a framework to evaluate model changes in the context of CMIP6-type experiments with ICON-a/ICON-o and ECHAM6/FESOM.

The results will be made available through a web-based tool—the Diagnostics and Metrics Explorer—for effective analysis. We will take advantage of direct collaboration with ongoing Horizon2020 projects (CRESCENDO and PRIMAVERA) dealing with evaluation of Earth System Models in the context of CMIP6.

Reports from the scientific front

Metrics and Diagnostics for model improvements

By Nikolay Koldunov

I am Nikolay Koldunov, Post Doc at MARUM and Alfred Wegener Institute. Since October I begin to work at Research Project S1: Diagnosis and Metrics in Climate Models. The main aim of the project is to integrate and synthesize work done in other parts of the TRR181. In particular we will provide metrics and diagnostics to help access the impact of model improvements suggested by TRR181 on quality of the climate models. One of the main challenges is to create model diagnostics that would not only quantify improvements, but also allow to clearly identify the cause of changes in model behavior. In this respect the proper protocol and experiment setup for numerical experiments is crucial and its development will be important part of my work. The resulting diagnostics will become available for the wider research community through the ESMValTool, that is going to be one of the main instruments of model analysis for the CMIP6 project.