My Twitter stream is full of links to articles and blog posts with titles like, “The 7 must-haves for business success,” or “The 5 things that make a successful entrepreneur,” or “10 reasons why beer drinkers make better businessmen.”
OK, maybe I exaggerated a bit with that last one but you get the picture.
This piece about selection bias by Michael Blastland on the BBC website was a useful antidote. He quotes the work of Jerker Denrell who insists that, to really understand success, we have to understand failure too. His HBR article Selection Bias and the Perils of Benchmarking is well worth reading in full. (You can get it for free if you are a CIPD member.)
Characteristics like risk-taking, confidence, willingness to act on hunches, persistence in the face of setbacks and the ability to persuade others are, he says, typical of successful entrepreneurs. Problem is, they are typical of those who fail in business too. In other words, confidence, risk taking and all the other good things may indicate that someone is more likely to start a company but it doesn’t tell us much about the likelihood of that company’s success. It becomes a truism; successful business founders have the characteristics of people who start businesses. We might as well, says Denrell, point to the fact that all successful business leaders brush their teeth.
He tells another great story about how the US Air Force in the Second World War recorded where its planes got hit most often and therefore proposed to reinforce their armour in those places. That is, until statistician Abraham Wald spotted the selection bias, pointing out that they were only looking at the survivors. The planes that got hit in these places were the ones that made it back. Wouldn’t it be better, therefore, to reinforce the planes in the areas where they hadn’t been hit?
The “inescapable logic of statistics”, says Denrell, means that managers must study failure as much as they study success:
No managers should accept a theory about business unless they can be confident that the theory’s advocates are working off an unbiased data set.
The trouble is, businesses are made up of biased data sets. Studying failures presents something of a problem for management research. We tend to airbrush failure out of corporate history. We recruit and promote the people who do well in our assessment processes and assume that they do well because we have selected for the right things. However, if we were being truly scientific we would also take on those who didn’t do well, so that we could test our processes and our definitions of good performance.
Imagine the conversation:
“We’ve got three Head of Business Unit roles coming up and three internal executives at the right level. Our High Potential Leaders’ Programme showed that Stella was a star, Mick was mediocre and Dave was a disaster. But, instead of recruiting externally, in the interests of scientific research, I appointed Stellar Stella, Mediocre Mick and Disastrous Dave to each of the roles. This way, we can test whether our HiPo programme really does identify the best leaders.”
It’s just not going to happen is it?
Selection bias is, to an extent, inbuilt in most management research because to study managers and their behaviour is to study group of people already selected by previous definitions of merit. Focusing only on the most successful restricts the sample still further. Proper science demands that we set up control groups and test our theories to breaking point by trying to disprove them. That’s never going to happen in a corporate setting. It’s too much of a risk.
It’s all too easy to look for the attributes of the successful and claim that it is these attributes that made them successful. If they are characteristics we share, then we are even more inclined to draw such conclusions. We don’t know whether the failures share those attributes too. We don’t even know whether the failures had other attributes which might, over time, have made them successful, because we have long since dumped them by the wayside. This inevitable failure to study failure means that management can never be truly evidence based.