Warning: A non-numeric value encountered in /home/msightan/public_html/wp-content/themes/Divi/functions.php on line 5837

Why Do Analytical Pilots Fail?​

It is said that some astounding proportion of BI and analytics efforts fail. Depending on the context, that number appears to range from 50% to 80%. Certainly, numerous debriefs have been conducted on what worked and what did not work; many have opined on the top reasons for failure. So, why does this continue to happen?​

Take analytical pilots. (Here we refer to pilots whose main concern involves analyzing the data and not simply implementing a tool–the latter deserves a separate discussion.) Pilots are particularly important, because the resulting decisions shape the course of what to come. At the risk of stating the obvious, organizations conduct them to do something with little to no precedent, and pilots are a financially prudent way to see if it works before investing in a larger-scale capability; starting small does provide an opportunity to work out the kinks, while also allowing organizations to plan better.

However, the fundamental reasons for the start-small approach deserve more careful thought in analytics. Specifically, we should ask whether the organization is ready for the consequences of the analysis results, and recognize that the transformation expected from the positive conclusion does not happen naturally. A pilot intended to prove the value of analytics is especially tricky, as the very need to prove may indicate that someone in the organization has not yet bought into the idea of the consequences–applying analysis results to make changes, and changes are uncomfortable. There is merit to convincing the unconvinced, but the degree to which the entire organization becomes convinced is a huge factor in whether there is a realistic future for analytics beyond the pilot.

That is, the organization must have the collective desire to be data-driven and have the next steps already defined, ready to accept change. The main goal of the analysis should be simply to prove the sufficiency of the business impact, with everything else already in place or ready to be executed immediately upon the completion of the analysis. Unfortunately, many non-financial planning and decisions are put on hold until the results of the analysis are available; the situation is exasperated with the unconvinced or the marginally convinced. We have seen pilot analysis executed, only to be followed by lack of priority, a long time to define the next steps, and finally the demise.

But a data-driven culture does not deprioritize the conversion of data-driven efforts into business results. First, if for some reason a well-selected and well-executed analytical pilot end up with less-than-favorable results, it has others in the wing waiting to benefit from the learning. Second and more importantly, for a pilot to be effective, those who will consume its results must be willing, able, and empowered to do so immediately–accept and action on those results to change themselves. The real challenge with analytics is that, without the resulting operational or strategic change–i.e., non-analytical change–it has no business value. And pilots cannot succeed with no business value. A data-driven culture is all about establishing an ecosystem of consumption of analytics throughout the organization and less about acquiring tools and data scientists. Having experience and capabilities in analytics is not a prerequisite.

I am not ruling out the possibility that there exist organizations for whom analytics makes no sense whatsoever, but I have yet to come across one in my nearly two decades of looking at analytics and analytical practices. I have, however, seen plenty that were not ready to consume. I am willing to bet that some analytics is sufficiently positively impactful to well more than the 20-50% of the opportunities as suggested by the failure rate. I am also willing to bet that a substantial portion, if not the majority, of the failures never came close to implementing the non-analytical changes needed to understand the business value.

​Are you going to be content with continuing the trend of failure, or are you going to challenge your business to transform?

So You Want to Do Predictive Analytics?

​It is not uncommon to hear business leaders say how predictive analytics is important and strategic. However, is predictive analytics really the Holy Grail of analytical maturity?

We can start by clarifying what predictive analytics is and where it resides in relation to the business objectives for leveraging analytics. We can slice the analytics space along the following three dimensions:

  • Predictive vs. Explanatory: Is the primary objective to quantify the likelihood of a particular outcome, or explain a particular phenomenon or behavior?
  • Exploratory vs. Confirmatory: Is the primary objective to discover something new that can help you form a hypothesis, or to confirm the hypothesis you have already formed?
  • Strategic vs. Tactical: Is the goal to inform business strategy decisions or to inform and execute on a specific set of actions?


We can table the discussion on methodology—from the business perspective, the specific quantitative methodology, statistical or otherwise, is secondary. Predictive methodologies can be applied while the objectives of the analysis remain explanatory, and in practice this is rather common. We should also acknowledge that some combinations of the above do not really exist, at least theoretically, and the distinctions can get a little blurry sometimes.

The point is that predictive analytics is just one class of just one dimension that defines the business objectives for analytics. The concern is that a blind focus on the “predictive” could be boxing organizations into analytical activities that do not necessarily address the most impactful business needs.

Going back to the strategic importance of predictive analytics, I believe it is important to make the following distinction: that having access to the predictive analytics capability is certainly strategic to the business, but what predictive analytics accomplishes is almost always tactical. The results of predictive analytics (scores, alerts, etc.) are most commonly used to automate certain aspects of decision making, such as recommending the next movie to watch, rank-ordering or prioritizing customers to target, or making decisions on a large volume of credit card applications very rapidly.

Businesses must start with the business objectives, then leverage the right analytical approach for the business objectives in order to realize full potential of data-driven decision making. While predictive analytics capabilities indeed often indicate a level of analytical maturity, it is only one part of analytics maturity. Setting any specific type of analytics as the Holy Grail cheats the organization of the best impact analytics could have. And it would be a shame if business leaders became disillusioned with analytics because that specific type of analytics did not produce the aggregate business impact they were expecting.