Are models more complex than they need to be?

This was one of the issues addressed by participants in an InsuranceERM roundtable discussion, sponsored by Aon Benfield's ReMetrica. Also covered in part one of our report are some of the benefits of using capital models, involving and educating the board, and technology issues

Cundy: Where does the value of a capital model lie?

Shaw: The real value of a capital model, from my perspective, is getting all the information into one place so that you can actually then improve understanding and measuring risk across the whole business and capture the overall risk profiles. And then, when you bring it together in a big IT modelling system, aka in a capital model, the more transparent you can keep that the better.

Hitchcox: A key risk five years ago was a hurricane or a stock market crash.  Now key risk number one is loss of diversification benefit. If you had one A4 side of all your key risk amounts, that is the biggest number: the total diversification credit. Finally we have got people looking at it. What has brought it right up the agenda is the risk that the regulator takes it away from us -- calculating it wrongly is the other risk.

Toller: We spend so much time trying to get the numbers correct that we tend to forget that one of the largest risks is actually model risk. It is the risk of what we do not know very much about.

Who understands the model

Cundy: Does the board understand the model?

Tait: I think they are getting better at understanding. How do I know? Because they ask better questions. So it is obvious that they are moving along the journey.

Could we ask a different question? Does anyone here think that they really, really understand their model? I would put my hand up and say I am not sure - absolutely every bit of the model? No.

Skinner: But, equally, they do not need to.

Tait: No.

Skinner: This comes back to where the expectations are. I think most boards, nowadays, have been exposed to capital models for up to eight years or more. That has given them a fairly good appreciation of how the bits fit together. They probably have enough information and enough understanding of it to grasp where the key limitations are and how to use them.

Cundy: Are there areas that the board tends to find difficult to grasp?

Shaw: Diversification: that is one of the biggest areas to grasp, both the assumptions underlying it and magnitude of the quantum. I also think that the whole move to Solvency II balance sheet has made it unnecessarily complex. We have now got everything running on a standard business-plan GAAP-basis and then on the Solvency II basis as well. That dislocation between the business plan and then the bits moving round the Solvency II balance sheet make it difficult to explain that in the same sentence.

"Diversification is an area that is very difficult. It is judgemental and we just do not have the data to support it." Henrietta Tait, LV=

Tait: Understanding what the key assumptions are and how much difference they can make is fundamental to board understanding and board acceptance of the notion of a model. There is a lot of judgement in there.

Diversification is an area that is very difficult. It is judgemental and we just do not have the data to support it. Also, it is fundamental they understand what goes in, rather than the complexity in the actual process of the model itself.

Skinner: We have a structured training programme as well as making sure key assumptions get discussed regularly. One of the techniques we use is risk drivers and scenarios - discussing what we think could happen as a story, not a statistic.

Ross: It is much easier to get engagement from senior management looking at whether output is reasonable.  And then using that to challenge the assumptions, rather than attempting to get buy‑in on high, medium, low correlations or standard deviations or one-in-200s. It is a question of: does the story stack up?  If it does not, then we will go back and work on the assumptions and make a note of them.

I guess you need a top-down and a bottom-up validation process. So you get comfort at a high level because conceptually it stacks up, but you have also been through a more detailed process to challenge the ground‑up assumptions going in.

Hitchcox: In reality, you just do not have time to start at the beginning and work your way into a meeting. You have an hour, maybe an hour and a half, you have the important people around the table and you have to get straight into the business of the capital numbers.

Shaw: At the moment, there has been too much bottom‑up focus and now we have to almost reposition ourselves to do a top‑down filter of appropriate information for real use. With use-test requirements, everybody is listening now. We have got to change what we are providing, because I think it is all still too detailed and cumbersome.

The right level of complexity

Cundy: Are models becoming too complex?

Noronha: As we have introduced more and more technology, it is very tempting to put more and more complexity into a capital model and you get pushed by users to put more and more complexity in. And I always push back, because complexity does not actually help them answer the questions they are looking for.

Tait: There are some areas where the model will become more complex and possibly needs to be, but you need the discipline to only do that in areas that are actually going to be value‑adding and to know when complexity does not become value‑adding.

"We can actually ask far more interesting questions within the current level of complexity." Justin Skinner, QBE European Operations

Skinner: But there is huge opportunity from technology. Utopia would be where the technology enables you to drill down into the model interactively within a meeting.

Shaw: Yes, utopia would be a scenario where all decision-makers will actually just be able to log in on their terminal – whether they are looking at a subset of the model or overall model – and ask, 'what happens if I were to make this decision?' But they should also understand it is but a tool.

Skinner: I think we can actually ask far more interesting questions within the current level of complexity. There is a good part of the model that you want to make more complex but, if possible, there is probably a bit of the model you could simplify. The general principle should be: let us stop it expanding forever.

Skelding: I have just done a quick run‑down of what I can remember as being the major differences between a model I built in 2006 and a more recent one I have been using in 2014, given all the improvements and the massive investment in terms of model speed and storage. What changed in that time is probably more about recognition patterns - Solvency II, one year, ultimate, etc. The 2014 version dumps out a lot more data so rather than sampling on, maybe, 100 different variables within the model, it is a multiple of that. And then there is a more sophisticated ESG [economic scenario generator] as well.

The 2014 model is certainly run more often, with more in-depth analysis of the calibration but I think those are the key differences to the modelling process. It does not seem very much given the improvements or the additional speed. I am not sure where the industry will go beyond this but I think other limitations will come into play before the technology.

Don't ditch the compass

Shaw: I have heard decision makers say in relation to capital models, 'Why should I put down my trusty compass and use your GPS?' I do not think we are there yet, in terms of actually providing a replacement for the compass. Capital models still have a long way to go.

Ross: I use the same analogy with some of the underwriters. I say, "Do not put your compass down. Do not get rid of your Ordnance Survey (OS) map. But start to become familiar with the GPS as well."  And at some point in the future, who knows, maybe we will all decide to put down the OS map and never use the compass, but for now I would be slightly wary because we will end up going down that narrow road, through the river and across the field.

Burke: If I were setting out on a long journey these days, I think I would choose a GPS.

Ross: I can now run 100,000 simulations in under two hours.  That is perfectly adequate for all the questions that could reasonably be lobbed at me that I could use a model to help answer.  I would much rather spend my time with the business: getting the business more comfortable with the idea of looking at this information when it makes those decisions, right now, than I would spending it on some extra kit to run in half an hour rather than two hours.

Skinner: The capital models themselves are probably as sophisticated as they need to be, but there is a lot of work that can be done on how we capture that initial insurance data and how we can use that.

Up in the cloud

Cundy: Has anyone had any experience of trying to put their capital models in the cloud?

Shaw: Cat models, yes, but our main concern is data security and similar issues.

Jootun: We do not have a cloud but the upside is granularity of data and getting all the versions consistent.

"There is little point progressing the model faster than the business can utilise it. You have to remain cost effective." Martin Burke, MSIUL

Toller: It feels like a very incremental benefit. It may be useful and it might improve efficiency, but I don't believe it will create major change in approach.

Cundy: Is it something that would facilitate Justin's idea of utopia, with everyone on their iPads?  That is certainly what RMS is pushing with their system. The idea is that you could have the board playing with catastrophe models on their iPads.

Skinner: I think we are five or ten years off that. At the moment the costs and benefits don't stack up.

Burke: There is little point progressing the model faster than the business can utilise it. You have to remain cost effective.

Ross: I do not think it is a decision that we would take at the level of a capital model – the cloud is more the domain of the wider IT infrastructure.

Cundy: So the feeling around the table is that, if the model runs faster, it is not necessarily more useful?

Ross: It depends how fast it is. But I think you hit a point at which decisions do not have to be made that quickly. If it took a week, maybe it would be a problem, but not if it takes a small number of hours. Most decisions do not have to be made that quickly.

Tait: I think there are areas of modelling that are improved by better run times. It depends very much on the type of models you are looking at. Asset-liability models, notoriously, take a very long time, when you have a lot of interaction and you are trying to programme-in management actions. So with another step-change in run-time improvement, you will be able to eliminate some of that model risk.

Jootun: What decisions are really made within an insurance company that require that sort of timescale?  For most insurance companies, what you are going to do in terms of investment strategy, which is normally the level at which they are used, tend to be decisions that are made over a slightly longer timescale.

The second part of this roundtable can be read here