Wallach, D., Thorburn, P., Asseng, S., Challinor, A. J., Ewert, F., Jones, J. W., et al. (2016). Estimating model prediction error: Should you treat predictions as fixed or random. Env. Model. Softw., 84, 529–539.
Abstract: Crop models are important tools for impact assessment of climate change, as well as for exploring management options under current climate. It is essential to evaluate the uncertainty associated with predictions of these models. We compare two criteria of prediction error; MSEPfixed, which evaluates mean squared error of prediction for a model with fixed structure, parameters and inputs, and MSEPuncertain(X), which evaluates mean squared error averaged over the distributions of model structure, inputs and parameters. Comparison of model outputs with data can be used to estimate the former. The latter has a squared bias term, which can be estimated using hindcasts, and a model variance term, which can be estimated from a simulation experiment. The separate contributions to MSEPuncertain(X) can be estimated using a random effects ANOVA. It is argued that MSEPuncertain(X) is the more informative uncertainty criterion, because it is specific to each prediction situation.
|
Wallach, D., Thorburn, P., Asseng, S., Challinor, A. J., Ewert, F., Jones, J. W., et al. (2016). Overview paper on comprehensive framework for assessment of error and uncertainty in crop model predictions (Vol. 8).
Abstract: Crop models are important tools for impact assessment of climate change, as well as for exploring management options under current climate. It is essential to evaluate the uncertainty associated with predictions of these models. Several ways of quantifying prediction uncertainty have been explored in the literature, but there have been no studies of how the different approaches are related to one another, and how they are related to some overall measure of prediction uncertainty. Here we show that all the different approaches can be related to two different viewpoints about the model; either the model is treated as a fixed predictor with some average error, or the model can be treated as a random variable with uncertainty in one or more of model structure, model inputs and model parameters. We discuss the differences, and show how mean squared error of prediction can be estimated in both cases. The results can be used to put uncertainty estimates into a more general framework and to relate different uncertainty estimates to one another and to overall prediction uncertainty. This should lead to a better understanding of crop model prediction uncertainty and the underlying causes of that uncertainty. This study was published as (Wallach et al. 2016)
|
Hoffmann, H., Zhao, G., Van Bussel, L., Enders, A., Specka, X., Sosa, C., et al. (2014). Effects of climate input data aggregation on modelling regional crop yields. FACCE MACSUR Mid-term Scientific Conference, 3(S) Sassari, Italy.
Abstract: Crop models can be sensitive to climate input data aggregation and this response may differ among models. This should be considered when applying field-scale models for assessment of climate change impacts on larger spatial scales or when coupling models across scales. In order to evaluate these effects systematically, an ensemble of ten crop models was run with climate input data on different spatial aggregations ranging from 1, 10, 25, 50 and 100 km horizontal resolution for the state of North Rhine-Westphalia, Germany. Models were minimally calibrated to typical sowing and harvest dates, and crop yields observed in the region, subsequently simulating potential, water-limited and nitrogen-limited production of winter wheat and silage maize for 1982-2011. Outputs were analysed for 19 variables (yield, evapotranspiration, soil organic carbon, etc.). In this study the sensitivity of the individual models and the model ensemble in response to input data aggregation is assessed for crop yield. Results show that the mean yield of the region calculated from climate time series of 1 km horizontal resolution changes only little when using climate input data of higher aggregation levels for most models. However, yield frequency distributions change with aggregation, resembling observed data better with increasing resolution. With few exceptions, these results apply to the two crops and three production situations (potential, water-, nitrogen-limited) and across models including the model ensemble, regardless of differences among models in simulated yield levels and spatial yield patterns. Results of this study improve the confidence of using crop models at varying scales.
|
Martre, P., Wallach, D., Asseng, S., Ewert, F., Jones, J. W., Rötter, R. P., et al. (2015). Multimodel ensembles of wheat growth: many models are better than one. Glob. Chang. Biol., 21(2), 911–925.
Abstract: Crop models of crop growth are increasingly used to quantify the impact of global changes due to climate or crop management. Therefore, accuracy of simulation results is a major concern. Studies with ensembles of crop models can give valuable information about model accuracy and uncertainty, but such studies are difficult to organize and have only recently begun. We report on the largest ensemble study to date, of 27 wheat models tested in four contrasting locations for their accuracy in simulating multiple crop growth and yield variables. The relative error averaged over models was 24-38% for the different end-of-season variables including grain yield (GY) and grain protein concentration (GPC). There was little relation between error of a model for GY or GPC and error for in-season variables. Thus, most models did not arrive at accurate simulations of GY and GPC by accurately simulating preceding growth dynamics. Ensemble simulations, taking either the mean (e-mean) or median (e-median) of simulated values, gave better estimates than any individual model when all variables were considered. Compared to individual models, e-median ranked first in simulating measured GY and third in GPC. The error of e-mean and e-median declined with an increasing number of ensemble members, with little decrease beyond 10 models. We conclude that multimodel ensembles can be used to create new estimators with improved accuracy and consistency in simulating growth dynamics. We argue that these results are applicable to other crop species, and hypothesize that they apply more generally to ecological system models.
|
Rivington, M., & Wallach, D. (2015). Information to support input data quality and model improvement (Vol. 6).
Abstract: Data quality is a key factor in determining the quality of model estimates and hence a models’ overall utility. Good models run with poor quality explanatory variables and parameters will produce meaningless estimates. Many models are now well developed and have been shown to perform well where and when good quality data is available. Hence a major limitation now to further use of models in new locations and applications is likely to be the availability of good quality data. Improvements in the quality of data may be seen as the starting point of further model improvement, in that better data itself will lead to more accurate model estimates (i.e. through better calibration), and it will facilitate reduction of model residual error by enabling refinements to model equations. This report sets out why data quality is important as well as the basis for additional investment in improving data quality. No Label
|
