The need for diagnostic assessment of bootstrap predictive models

Glen Barnett and Ben Zehnwirth

Contents:

• The need for diagnostic assessment of bootstrap predictive models
• A basic bootstrap introduction
• Diagnostic displays for a bootstrapped chain ladder
• Assessing bootstrap predictive distributions
• Some other considerations
• Conclusions
• References
• Appendices

The following set of pages are available as a PDF document here.

Conclusions

The use of the bootstrap does not remove the need to check assumptions relating to the appropriateness of the model. Indeed, it is clear that there's a critical need to check the assumptions.

The bootstrap cannot get around the facts that chain-ladder type models have no simple descriptors of features in the data. Note further for triangles ABC and LR-High there is so much remaining structure in the residuals -the bootstrap cannot get around this.

If you do fit a quasi-Poisson GLM, it's important to check the one-step-ahead prediction errors in order to see how it performs as a predictive model - the residuals against fitted values don't show you the problems.

In any case, it should be looked at before bootstrapping a model, and once a bootstrap has been done, you should also validate at least the last year - examine whether the actual values from the last calendar year could plausibly have come from the predictive distribution standing a year earlier.

If it is the predictive behaviour that is of interest, prediction errors are appropriate tools to use in standard diagnostics, and they can be analyzed in the same way as residuals are for models where prediction is within the range of the data.

Checking the model when bootstrapping is achieved in much the same way as it is for any other model - via diagnostics - but they must be diagnostics selected with a clear understanding of the problem, the model and the way in which the bootstrap works.

Continue with: References.