How we check our own estimates

A loss estimate is a promise about the future, and promises can be checked. A backtest looks back over the last year or so, counts the days the portfolio lost more than the estimate said it should, and compares that count to the number you would expect.

At 95% confidence you expect the market to break through the line on about one day in twenty. Over 250 trading days that is roughly 12 or 13 breaches. If it happened 14 times, the model is doing its job. If it happened 30 times, the line was too optimistic and the estimate is not worth much.

The traffic light

Sentinel runs a standard statistical test on that count, named after the economist Paul Kupiec, and turns the result into a traffic light so you do not need to read the statistics.

• Green: the number of breaches is about what the confidence level promised. The estimate has been holding up.
• Amber: more breaches than expected, but not so many that chance is ruled out. Watch it, and lean on the stress tests.
• Red: far more breaches than the estimate allows. The model has been underestimating your risk, and you should treat the number as a floor, not a forecast.

Why we show it at all

A risk number shown without its record asks to be taken on trust. Leaving the record out is easier: a red light is an admission. We show it because a number you cannot check is not information, it is decoration. When our estimate has been wrong, the thing to do is say so on the same page as the estimate, rather than quietly recalibrate and hope you did not notice.

Red does not mean the app is broken. It usually means the market has changed faster than the last year of history can describe, which is exactly when you want to know.