|
Records |
Links |
|
Author |
Bellocchi, G.; Rivington, M.; Acutis, M. |
|
|
Title |
Protocol for model evaluation |
Type |
Report |
|
Year |
2014 |
Publication |
FACCE MACSUR Reports |
Abbreviated Journal |
|
|
|
Volume |
3 |
Issue |
|
Pages |
D-L2.2/D |
|
|
Keywords |
|
|
|
Abstract |
This deliverable focuses on the development of methods for model evaluation in order to have unambiguous indications derived from the use of several evaluation metrics. The information about model quality is aggregated into a single indicator using a fuzzy expert system that can be applied to a wide range of model estimates where suitable test data are available. This is a cross-cutting activity between CropM (C1.4) and LiveM (L2.2). No Label |
|
|
Address |
|
|
|
Corporate Author |
|
Thesis |
|
|
|
Publisher |
|
Place of Publication |
|
Editor |
|
|
|
Language |
|
Summary Language |
|
Original Title |
|
|
|
Series Editor |
|
Series Title |
|
Abbreviated Series Title |
|
|
|
Series Volume |
|
Series Issue |
|
Edition |
|
|
|
ISSN |
|
ISBN |
|
Medium |
|
|
|
Area |
|
Expedition |
|
Conference |
|
|
|
Notes |
|
Approved |
no |
|
|
Call Number |
MA @ admin @ |
Serial |
2229 |
|
Permanent link to this record |
|
|
|
|
Author |
Sanna, M.; Acutis, M.; Bellocchi, G. |
|
|
Title |
Interrelationship between evaluation metrics to assess agro-ecological models |
Type |
Report |
|
Year |
2014 |
Publication |
FACCE MACSUR Reports |
Abbreviated Journal |
|
|
|
Volume |
3 |
Issue |
|
Pages |
Sp3-5 |
|
|
Keywords |
|
|
|
Abstract |
When evaluating the performances of simulation models, the perception of the quality of the outputs may depend on the statistics used to compare simulated and observed data. In order to have a comprehensive understanding of model performance, the use of a variety of metrics is generally advocated. However, since they may be correlated, the use of two or more metrics may convey the same information, leading to redundancy. This study intends to investigate the interrelationship between evaluation metrics, with the aim of identifying the most useful set of indicators, for assessing simulation performance. Our focus is on agro-ecological modelling. Twenty-three performance indicators were selected to compare simulated and observed data of four agronomic and meteorological variables: above-ground biomass, leaf area index, hourly air relative humidity and daily solar radiation. Indicators were calculated on large data sets, collected to effectively apply correlation analysis techniques. For each variable, the interrelationship between each pair of indicators was evaluated, by computing the Spearman’s rank correlation coefficient. A definition of “stable correlation” was proposed, based on the test of heterogeneity, allowing to assess whether two or more correlation coefficients are equal. An optimal subset of indicators was identified, striking a balance between number of indicators, amount of provided information and information redundancy. They are: Index of Agreement, Squared Bias, Root Mean Squared Relative Error, Pattern Index, Persistence Model Efficiency and Spearman’s Correlation Coefficient. The present study was carried out in the context of CropM-LiveM cross-cutting activities of MACSUR knowledge hub. No Label |
|
|
Address |
|
|
|
Corporate Author |
|
Thesis |
|
|
|
Publisher |
|
Place of Publication |
|
Editor |
|
|
|
Language |
|
Summary Language |
|
Original Title |
|
|
|
Series Editor |
|
Series Title |
|
Abbreviated Series Title |
|
|
|
Series Volume |
|
Series Issue |
|
Edition |
|
|
|
ISSN |
|
ISBN |
|
Medium |
|
|
|
Area |
|
Expedition |
|
Conference |
|
|
|
Notes |
|
Approved |
no |
|
|
Call Number |
MA @ admin @ |
Serial |
2222 |
|
Permanent link to this record |
|
|
|
|
Author |
Fronzek, S.; Pirttioja, N.; Carter, T.R.; Bindi, M.; Hoffmann, H.; Palosuo, T.; Ruiz-Ramos, M.; Tao, F.; Trnka, M.; Acutis, M.; Asseng, S.; Baranowski, P.; Basso, B.; Bodin, P.; Buis, S.; Cammarano, D.; Deligios, P.; Destain, M.-F.; Dumont, B.; Ewert, F.; Ferrise, R.; François, L.; Gaiser, T.; Hlavinka, P.; Jacquemin, I.; Kersebaum, K.-C.; Kollas, C.; Krzyszczak, J.; Lorite, I.J.; Minet, J.; Minguez, M.I.; Montesino, M.; Moriondo, M.; Müller, C.; Nendel, C.; Öztürk, I.; Perego, A.; Rodríguez, A.; Ruane, A.C.; Ruget, F.; Sanna, M.; Semenov, M.A.; Slawinsky, C.; Stratonovitch, P.; Supit, I.; Waha, K.; Wang, E.; Wu, L.; Zhao, Z.; Rötter, R.P. |
|
|
Title |
Classifying multi-model wheat yield impact response surfaces showing sensitivity to temperature and precipitation change |
Type |
Report |
|
Year |
2017 |
Publication |
FACCE MACSUR Reports |
Abbreviated Journal |
|
|
|
Volume |
10 |
Issue |
|
Pages |
C4.3-D1 |
|
|
Keywords |
|
|
|
Abstract |
Crop growth simulation models can differ greatly in their treatment of key processes and hence in their response to environmental conditions. Here, we used an ensemble of 26 process-based wheat models applied at sites across a European transect to compare their sensitivity to changes in temperature (−2 to +9°C) and precipitation (−50 to +50%). Model results were analysed by plotting them as impact response surfaces (IRSs), classifying the IRS patterns of individual model simulations, describing these classes and analysing factors that may explain the major differences in model responses. The model ensemble was used to simulate yields of winter and spring wheat at sites in Finland, Germany and Spain. Results were plotted as IRSs that show changes in yields relative to the baseline with respect to temperature and precipitation. IRSs of 30-year means and selected extreme years were classified using two approaches describing their pattern. The expert diagnostic approach (EDA) combines two aspects of IRS patterns: location of the maximum yield (nine classes, Figure 1) and strength of the yield response with respect to climate (four classes), resulting in a total of 36 combined classes defined using criteria pre-specified by experts. The statistical diagnostic approach (SDA) groups IRSs by comparing their pattern and magnitude, without attempting to interpret these features. It applies a hierarchical clustering method, grouping response patterns using a distance metric that combines the spatial correlation and Euclidian distance between IRS pairs. The two approaches were used to investigate whether different patterns of yield response could be related to different properties of the crop models, specifically their genealogy, calibration and process description. Although no single model property across a large model ensemble was found to explain the integrated yield response to temperature and precipitation perturbations, the application of the EDA and SDA approaches revealed their capability to distinguish: (i) stronger yield responses to precipitation for winter wheat than spring wheat; (ii) differing strengths of response to climate changes for years with anomalous weather conditions compared to period-average conditions; (iii) the influence of site conditions on yield patterns; (iv) similarities in IRS patterns among models with related genealogy; (v) similarities in IRS patterns for models with simpler process descriptions of root growth and water uptake compared to those with more complex descriptions; and (vi) a closer correspondence of IRS patterns in models using partitioning schemes to represent yield formation than in those using a harvest index. Such results can inform future crop modelling studies that seek to exploit the diversity of multi-model ensembles, by distinguishing ensemble members that span a wide range of responses as well as those that display implausible behaviour or strong mutual similarities. The full manuscript of this study is currently under revision (Fronzek et al. 2017). |
|
|
Address |
|
|
|
Corporate Author |
|
Thesis |
|
|
|
Publisher |
|
Place of Publication |
|
Editor |
|
|
|
Language |
|
Summary Language |
|
Original Title |
|
|
|
Series Editor |
|
Series Title |
|
Abbreviated Series Title |
|
|
|
Series Volume |
|
Series Issue |
|
Edition |
|
|
|
ISSN |
|
ISBN |
|
Medium |
|
|
|
Area |
|
Expedition |
|
Conference |
|
|
|
Notes |
CropM |
Approved |
no |
|
|
Call Number |
MA @ admin @ |
Serial |
4956 |
|
Permanent link to this record |
|
|
|
|
Author |
Sándor, R.; Barcza, Z.; Acutis, M.; Doro, L.; Hidy, D.; Köchy, M.; Minet, J.; Lellei-Kovács, E.; Ma, S.; Perego, A.; Rolinski, S.; Ruget, F.; Sanna, M.; Seddaiu, G.; Wu, L.; Bellocchi, G. |
|
|
Title |
Multi-model simulation of soil temperature, soil water content and biomass in Euro-Mediterranean grasslands: Uncertainties and ensemble performance |
Type |
Journal Article |
|
Year |
2016 |
Publication |
European Journal of Agronomy |
Abbreviated Journal |
European Journal of Agronomy |
|
|
Volume |
|
Issue |
|
Pages |
|
|
|
Keywords |
Biomass; Grasslands; Modelling; Multi-model ensemble; Soil processes |
|
|
Abstract |
• We simulate biomass, soil water content (SWC) and temperature (ST) in grasslands. • We compare nine models to the multi-model median (MMM) at nine sites. • With model calibration, we obtain satisfactory estimates of ST, less of SWC and biomass. • We observe discrepancies across models in the simulation of grassland processes. • We improve performance with multi-model approach. This study presents results from a major grassland model intercomparison exercise, and highlights the main challenges faced in the implementation of a multi-model ensemble prediction system in grasslands. Nine, independently developed simulation models linking climate, soil, vegetation and management to grassland biogeochemical cycles and production were compared in a simulation of soil water content (SWC) and soil temperature (ST) in the topsoil, and of biomass production. The results were assessed against SWC and ST data from five observational grassland sites representing a range of conditions – Grillenburg in Germany, Laqueuille in France with both extensive and intensive management, Monte Bondone in Italy and Oensingen in Switzerland – and against yield measurements from the same sites and other experimental grassland sites in Europe and Israel. We present a comparison of model estimates from individual models to the multi-model ensemble (represented by multi-model median: MMM). With calibration (seven out of nine models), the performances were acceptable for weekly-aggregated ST (R² > 0.7 with individual models and >0.8–0.9 with MMM), but less satisfactory with SWC (R² < 0.6 with individual models and < ∼ 0.5 with MMM) and biomass (R² < ∼0.3 with both individual models and MMM). With individual models, maximum biases of about −5 °C for ST, −0.3 m3 m−3 for SWC and 360 g DM m−2 for yield, as well as negative modelling efficiencies and some high relative root mean square errors indicate low model performance, especially for biomass. We also found substantial discrepancies across different models, indicating considerable uncertainties regarding the simulation of grassland processes. The multi-model approach allowed for improved performance, but further progress is strongly needed in the way models represent processes in managed grassland systems. |
|
|
Address |
|
|
|
Corporate Author |
|
Thesis |
|
|
|
Publisher |
|
Place of Publication |
|
Editor |
|
|
|
Language |
Language |
Summary Language |
|
Original Title |
|
|
|
Series Editor |
|
Series Title |
|
Abbreviated Series Title |
|
|
|
Series Volume |
|
Series Issue |
|
Edition |
|
|
|
ISSN |
1161-0301 |
ISBN |
|
Medium |
|
|
|
Area |
LiveM |
Expedition |
|
Conference |
|
|
|
Notes |
|
Approved |
no |
|
|
Call Number |
MA @ admin @ |
Serial |
4768 |
|
Permanent link to this record |
|
|
|
|
Author |
Sanna, M.; Bellocchi, G.; Fumagalli, M.; Acutis, M. |
|
|
Title |
A new method for analysing the interrelationship between performance indicators with an application to agrometeorological models |
Type |
Journal Article |
|
Year |
2015 |
Publication |
Environmental Modelling & Software |
Abbreviated Journal |
Env. Model. Softw. |
|
|
Volume |
73 |
Issue |
|
Pages |
286-304 |
|
|
Keywords |
model evaluation; performance indicators; stable correlation; solar-radiation; simulation-model; environmental-models; statistical-methods; crop nitrogen; validation; rice; uncertainty; calibration; software |
|
|
Abstract |
The use of a variety of metrics is advocated to assess model performance but correlated metrics may convey the same information, thus leading to redundancy. Starting from this assumption, a method was developed for selecting, from among a collection of performance indicators, one or more subsets providing the same information as the entire set. The method, based on the definition of “stable correlation”, was applied to 23 performance indicators of agrometeorological models, calculated on large sets of simulated and observed data of four agronomic and meteorological variables: above-ground biomass, leaf area index, hourly air relative humidity and daily solar radiation. Two subsets were determined: {Squared Bias, Root Mean Squared Relative Error, Coefficient of Determination, Pattern Index, Modified Modelling Efficiency}, {Persistence Model Efficiency, Root Mean Squared Relative Error, Coefficient of Determination, Pattern Index}. The method needs corroboration but is statistically founded and can support the implementation of standardized evaluation tools. (C) 2015 Elsevier Ltd. All rights reserved. |
|
|
Address |
|
|
|
Corporate Author |
|
Thesis |
|
|
|
Publisher |
|
Place of Publication |
|
Editor |
|
|
|
Language |
English |
Summary Language |
|
Original Title |
|
|
|
Series Editor |
|
Series Title |
|
Abbreviated Series Title |
|
|
|
Series Volume |
|
Series Issue |
|
Edition |
|
|
|
ISSN |
1364-8152 |
ISBN |
|
Medium |
Article |
|
|
Area |
|
Expedition |
|
Conference |
|
|
|
Notes |
CropM LiveM, ftnotmacsur |
Approved |
no |
|
|
Call Number |
MA @ admin @ |
Serial |
4503 |
|
Permanent link to this record |