Model validation: what’s in it for you?

Model validation is expensive and yields little

In 2018, news had it that some US Aegon subsidiaries were using asset management calculation models that contained ‘numerous errors’ and ‘did not work as promised’. The errors weren’t surprising in hindsight, as all the modeling work had been done by one junior analyst and not checked by anyone. This resulted in a $ 97 million fine imposed by the US SEC for misleading investors. So it turns out that the lack of model validation can be quite expensive!

You may think this is exceptional? Uncontrolled models are never implemented in your practice? Our experience in insurance practice is different.

Often enough there is nothing more than ‘4-eyes’ checking: calculations are checked, but not sufficiently independently tested, so that thinking errors can remain in the design of a model. Data sources outside the model are not always properly controlled. Model changes are only marginally checked.

And also think of the ‘key-person risk’, where your organization relies heavily on that one actuarial specialist – possibly even an externally hired specialist. Is he or she flawless? And what about that ‘externally validated tool’ from the specialist: are the assumptions and parameters used really suitable for your portfolio? You don’t really know without validation.

What can be the yield of model validation? Estimate the maximum consequences of a ‘stupid mistake’: an error that could possibly persist for years without anyone signalling the outcome as illogical. Including fines and reputation risk, this can be quite expensive. And your company was managed on the basis of incorrect data during that time . This analysis gives you insight into the budget that you are willing to allocate for better model management.

Model validation is far too difficult

You can take a big approach by drawing up a general validation policy and setting up a new department for it. But it actually includes some essential steps that belong to the ‘primary care’ tasks, such as:

• rules for model management,
• model documentation,
• standard for statistical quality and arithmetic correctness,
• standard for calibration of parameters,
• rules for the use of external models and data.

During the model validation, a checklist is made along all these components and an independent, critical person assesses whether all criteria have been demonstrably met. The ‘difficult’ thing is that the requirements for a good model are made visible, and that in most cases the quality of management and documentation must be improved in order to ‘pass’ the model validation. But once that quality improvement has been made, validation becomes increasingly easier.

In our experience, validations always reveal errors. In addition, it provides valuable information for second line functions, in particular Actuarial and Risk management. The result is a transparent picture of the reliability of the model and outcomes, on which they must advise the management.

Model validation – let’s get started!

It goes without saying that Arcturus is ready to validate the most important actuarial models: premium setting, reinsurance, technical provisions, solvency capital and ORSA.

We propose a pragmatic approach, starting with a risk inventory. Then the board selects the priorities, such as an initial validation for new models, and a regular validation frequency of once every three years.

Arcturus has extensive experience in model validation and can easily adapt a ‘standard’ validation policy to suit your organization. Validation can then be coordinated on a project basis per model. We can help the first line to organize the model and documentation, and we have enough experts to send an independent validator along afterwards. 

Validation projects are also easy to plan in the less busy periods of your team.