
The last time I checked the weather forecast, I only wanted to know what the weather would be like and not why. Why is for meteorologists, what is for regular people.
Thus, accurate forecasts have a real place. As a regular human being, all I care about is whether I need to pack an umbrella or not.
On the other hand, if I need deeper understanding, of a specific business process (or if I’m intellectually curious), I will need something more, and I will need a reliable model for that. One that can explain why it’s raining today or why a particular number has moved in the regulatory finding, or why a particular team is destined to lose today instead of winning in the football league.
Statisticians often get credit when their prediction is right. That’s fair — but only as long as their prediction doesn’t need deeper scrutiny. If, on the other hand, their models need to be explainable, then an inaccurate prediction that’s explainable is often better than the opposite.
Where un-explainability goes awry
To be fair, a model that’s always — and I mean always — accurate goes a long way. But models in the real world are almost never like that. And often it only needs a small handful of inaccurate predictions to ruin trust in a whole model — especially when those inaccurate predictions were of high stakes.
Consider, for example, Robodebt. As the BBC reported, the Australian government ran an automated scheme that sent letters to citizens asking them to pay back certain welfare benefits. Likely part of a larger scheme, it ran from 2016 until 2019 until it was declared illegal. Citizens, many among the country’s poorest, received letters to pay back thousands of Australian dollars, which in many caused anxiety and made them feel like criminals. There were even three confirmed suicides which were at least in part attributable to the recipients feeling pressured by these letters.
There are indeed situations where citizens need to pay back welfare benefits — the model had a reason to exist. However, where there are inaccuracies, they’re likely to affect those that needed those benefits the most. In other words, such a model’s predictions have an outsized effect. The Australian government at that time had seemingly underestimated these devastating effects on some of its most vulnerable citizens.
A similar scandal actually brought the Dutch government to fall in 2021, which was also reported by the BBC. Thousands of families had been wrongly accused of welfare fraud and paybacks of tens of thousands of Euros were demanded of them. Already under this financial pressure, many affected people lost their jobs and couldn’t find new ones, as they were seen as fraudsters.
This came when a year prior the Dutch tax administration had, a year prior, admitted that “11,000 people were subjected to extra scrutiny simply because they had dual nationality.” Not only were people’s very livelihoods and wellbeing affected; it also had a discriminatory effect.
Had these models been explainable from the start, these mistakes likely would have been much less likely to happen: If a statistician clearly saw a label “low income” or “dual nationality” coupled with a high parameter to an outcome like “must pay back welfare,” they’d frown and start fixing the model — or throwing it away entirely. But because many prediction models are very complex machine learning models, which are not designed for humans to look into and interpret in an understandable way, no one could know the real outcomes until they happened.
This is not just a public administration problem. Likely, it happens all the time with private-company models; however, the consequences tend to be less disastrous. Academics agree that explainability is sorely needed here.
Academia agrees
A keen reader might interject at this point that Shapley values exist to open up such complex machine learning models. This is true, since Shapley values tell you which variables have most influence on the outcome and by how much. However, this does not make a model explainable: Such post-hoc explanations “are often not reliable, and can be misleading.”
This statement comes from a paper by Cynthia Rudin at Duke University, which is literally titled “Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead.” She argues (in my own words) that post-hoc explanations are often just over-simplified versions of these black-box models, which thus fail to capture the nuance of exactly those cases where inaccuracy is very costly — especially when the only proper explanation is “the model said so,” like in the two government welfare scandals from above.
Furthermore, these post-hoc explanations often don’t cover those rare but important cases that are outliers in the database. In a government welfare scheme, there are usually millions of recipients, yet only a few tens of thousands are dependent on them because of their vulnerability. Often, explanations can’t grasp such small slithers of the original dataset; sometimes they actually can’t because the datapoints are so rare that the model wasn’t even trained with them.
Besides, black-box models being inherently complicated, they could lead to complicated decisions being made. And you know what humans are really bad at? Taking and then executing complicated decisions.
Better, then, to use a simpler model that might be less accurate at times but is inherently explainable. For example, the type of model that explicitly holds the variables from the original dataset — including variables like “low income” or “dual nationality” — with a simple regression score. A sixth-grader can interpret it, a statistician can validate or invalidate it, and a government can take proper action on that basis.
Keeping a balance
That’s not to say that one should always rely on the simplest explainable model. Black-box models have their merit: they’re usually easy to train, they often uncover relationships between variables that humans don’t, and they tend to be highly accurate. So long as the edge cases, where inaccuracies abound, are not too costly, black-box models can be a fine choice. And — so long as nobody demands an explanation even with the right prediction.
In reserve management for insurances, for example, regulatory scrutiny is high. Insurers need to keep extra money in the bank, just in case something happens to many policy holders at once, demanding many payouts. How much exactly the insurers need in the bank is determined by actuaries and statisticians; it’s also subject to strict rules by the government.
A modeling team inside an insurance can’t just submit a report along the lines of “our model says our reserves will be just fine to cover all the payouts in any case, and we believe it because it’s been right before.” A regulator won’t swallow that. They need to explain exactly why this is the case, which scenarios they modeled, and which variables would change their verdict. For this, one needs explainable models — even if they’re less accurate overall.
Another example closer to home: Many bookmakers for sports bets put out their predictions in advance. It’s okay to say “Bayern Munich will win because the model says so.” However, if that team surprisingly loses, they’ll be unable to explain what they did. Dear soccer fans, even Bayern Munich loses a match once in a while — and a bookmaker who is unexplainably wrong even once can ruin their whole career that way.
It’s thus more sustainable in the long run to make explainable predictions, such as “Bayern Munich will win this match 4-1 because striker Harry Kane is in top shape.” Not only does this make intuitive sense to human brains; it also makes bookmakers, sports betters, and the likes more adaptable. Because if Harry Kane calls in sick that day, they can change their bets immediately (if the rules allow it of course) and save their hard-earned cash.
Rule of thumb: Keep it dumb
To resume the point of the article: Really accurate models are tempting, because they’re easy enough to set up, fun to train, and — well — they predict the right things 99% of the time. The thing is that when they’re wrong, they’re stumped for an explanation. And even when they’re right, it might not be enough because some contexts always warrant an explanation.
It’s a bit like cake and a healthy sandwich: Both give energy and fill the stomach, but cake is (for many people) the more tempting option. As per the usual in a tempting world, the more boring-looking option is often the better and more sustainable one.
Explainable models may be more tedious to set up. And — since a sixth-grader can interpret them, they don’t make you look very smart. (This is honestly a big deal in my experience.) However, they survive scrutiny and do a better and fairer job when its predictions happen to be wrong. Arguably, much damage could be avoided if statisticians, governments, and institutions of many kinds invested in these kinds of dumber models.
But it’s not always critical, and a generous piece of cake is part of life also. It’s, as always, a balance: The next time it rains when the forecast said it was going to be sunny, I’ll “invest” three Euros in yet another umbrella and call it a day. But whenever my model says, “Bayern wins,” I check closely. After all, soccer fans are not to be argued with!
Meanwhile, at Wangari
It’s the big week — I’m speaking at Machine Learning Week Europe! I’m looking forward to my two sessions:
A workshop on how to create reliable, high-quality sports predictions — applied to this season’s Bundesliga. I co-authored the book Soccer Analytics with Machine Learning (O’Reilly, 2026); this is the practical version of it.
A table discussion titled How to get AI projects production-ready. I’d run a course on that topic last summer; it’s just getting timelier now.
You’ll hear about my experience and takeaways here, and of course on LinkedIn.
And a housekeeping note: Everybody’s inboxes are sufficiently filled, and I’m also concentrating on writing my second book (you’ll hear more about it from about mid-2027). I’ve thus decided to scale back this newsletter’s frequency: You’ll hear from me every two weeks from now onwards, instead of weekly. This also allows me to further concentrate on quality over quantity, while still keeping you up-to-date with the latest AI/data trends that I see.
So you can expect me in your inbox again on Tuesday, December 1st.
Reads of the Week
Since we’ve mentioned weather forecasts: So, which weather forecast is the best? by ActuallyWeather describes exactly the conundrum we face, where consumer weather apps look like all is deterministic (aside from maybe rain probabilities, which have made it into the mainstream). Professional apps skew the other direction, with raw sensor data and uncertainty everywhere — more than even experts can take in and digest in less than 10 minutes. ActuallyWeather thus proposes a weather app that not only contains the prediction but also the actual past weather, so the user can draw informed conclusions themselves.
Agus Sudjianto has a very interesting piece with MRM Lessons from Banking (MRM = model risk management). The one line that stuck with me is that assessing a model’s risk, and indirectly its quality, is an ongoing challenge: “Models and their environments keep changing after approval, so approval is not the finish line.” This is an excellent read for anyone working with serious models at their job.
A very eye-opening read on why predictions have become the norm: In Prediction Addiction, Jason Lewis writes that prediction models shift the responsibility of an action away from a human and towards a model. That feels safer and easier, but when the model is wrong, there’s no-one left to blame. Predictive power seems to have plateaued, yet governments and other institutions are doubling down on models, because it feels safer. The result might be a sort of paralysis: only acting when the model says so, not when your own brain says so and you might risk being wrong.


