Research on data provided by 122 companies in the advertising, digital, publishing, and software sectors (industries characterized by uncertainty over outcomes) suggests that data driven decision-making could be counter-productive under conditions of uncertainty. Heuristics and gut feelings often offered a better tradeoff in terms of decision-making speed and accuracy; the inclusion of analysis in the decision-making process did not bring about any meaningful improvement in accuracy while significantly reducing speed.
Data-driven decision making is often viewed as the gold standard in modern management. And this is for good reason. The explosion of available data and rapid advances in data science enable managers to know substantially more about their business. This knowledge, if used well, should bring about better decision-making on about every aspect of the business.
This is perhaps why most companies are in a horse race in building analytical capabilities to make the most of this unprecedented abundance of data. For example, a recent survey of Fortune 1000 companies shows that 91.9% of firms report increasing investment in data initiatives.
While the potential of big data is irrefutable, is it the panacea for all decision-making situations? Put differently, could a strong emphasis on data and analysis backfire under some circumstances? We explored this in our recent research.
Our intuition was that data-driven decision making could be counterproductive under extreme uncertainty. In such cases, it will be highly challenging and sometimes impossible to collect reliable data. This could explain why 12 publishers were unable to see the potential of Harry Potter and the Philosopher Stone before Bloomsbury Publishing accepted to publish an initial print run of 500 copies. The book was so innovative that there was by definition no prior data available to accurately assess its potential.
To test our intuition, we collected data from 122 companies in creative industries (in advertising, digital, publishing, and software sectors) about their latest innovation projects. We chose creative industries due to high levels of uncertainty about customer reactions and an infinite variety of potential new products and product modifications. For the same reason, we focused on innovation screening decisions — the decision to select what innovation projects to pursue for development. These decisions are characterized by high uncertainty; managers often lack sufficient past data that would enable them to predict customer reactions accurately, market potential, feasibility, and risks. Even if they had such data, it would often be extremely difficult and sometimes even misleading to extrapolate.
We asked managers in these companies to think about their most recent innovation project for which they needed to make a screening decision and included questions to understand how they made this decision. Specifically, the questions addressed the extent to which they relied on analysis (i.e., choosing the option that proved best based upon analyzing the data), instinct (i.e., choosing the option following their instincts), and a range of well-known heuristics (i.e., practical strategies to make decisions faster and more frugally). These heuristics included “tallying” (choosing the option with highest number of favorable points), “experience” (choosing the option most experienced person in team wanted) and “majority” (choosing the option most people wanted) amongst several others. Next, we asked managers to indicate whether they think they got the decision right (perceived decision-making accuracy) and how fast they were in making the decision (perceived decision-making speed).
The results first showed, to our surprise, despite the huge interest in big data, that the managers in our sample did not rely on analysis any more than on their instincts or some of the simple heuristics. The most commonly used heuristic, more than both analysis and instinct, was tallying.
We also find that relying on analysis is not necessarily the ideal way to choose between innovation projects. While the decisions based on data analysis brought about a good level of decision-making accuracy, the process was slow. Managers who relied on their instincts together with some simple heuristics made decisions that were just as accurate but were undertaken much more quickly. That is, heuristics and gut feelings offered a better tradeoff in terms of decision-making speed and accuracy; the inclusion of analysis in the decision-making process did not bring about any meaningful improvement in accuracy while significantly reducing speed.
One note of caution for managers who consider embarking on gut-based innovation decisions: the effectiveness of their intuition might rely on prior experience. Prior research suggests that the effectiveness of intuition compared to analysis is contingent upon domain knowledge; experts in a domain are more likely to make better gut decisions. Managers with limited domain expertise might therefore be better off by refraining from extensive reliance on intuition. Our results suggest relying mainly on heuristic also presents a viable alternative.
The next time when you face a managerial decision that is ambiguous, bear in mind that data might not be the only basis for a choice. Following your instincts, together with some simple heuristics, can lead to quicker and potentially as accurate decisions especially for those with the requisite expertise.