Editor’s note: Since 2007 Dresner Advisory Services has conducted a series of comprehensive annual Wisdom of Crowds® studies to reveal end-users’ real-world perspectives on the business intelligence (BI) market including drivers, how companies use BI, vendor rankings and buyer guides along with analysis of trends. In 2014 the company conducted its inaugural Wisdom of Crowds® Advanced and Predictive Analytics market study as collaboration with fellow industry expert Neil Raden. I spoke with chief research officer Howard Dresner about this growing analytics phenomenon in BI.
How do you define “advanced and predictive analytics?”
Howard Dresner: Advanced and predictive analytics is the ability to analyze current and historical facts to make predictions about future or unknown events or business aspects. It includes statistics, modeling, machine learning and data mining to analyze facts to make the predictions.
How widespread is the use of advanced and predictive analytics (A&PA) in companies today?
Howard Dresner: Predictive analytics are becoming key to business users who want to be able to look ahead as well as behind for analysis. It’s still a nascent market at this point, but users are growing daily in terms of their analytical sophistication and moving increasingly toward predictive analytics.
The majority (90 percent) of study respondents stated they view predictive analytics as at least important, and 54 percent view it as “very important” to “critical” for their organizations. Marketing and BICC (Business Intelligence Competency Center) functions have the strongest opinion that it is of critical importance.
However, despite the high awareness or sentiment of importance of advanced and predictive analytics, adoption is far from universal. A&PA is deployed today in less than 30 percent of the study participants’ organizations. Another 30 percent (especially in the financial services, insurance and tech industries) stated they plan to deploy it in the next 18 months.
So although predictive analytics should be part of a balanced BI portfolio, most organizations are still in the early stages.
Why is that?
Howard Dresner: It’s partly due to the fact that it’s difficult to maximize value without plenty of good historical data. Besides implementation challenges, there are challenges with businesses understanding how to deliver A&PA value. Greater adoption will occur as it becomes less complex and difficult to create models. In fact, respondents ranked support/guidance in preparing data analytical models as highly important.
In addition, some users are not yet comfortable and trusting of being directed by technology.
A further challenge is the state of data available for analysis.
What are the challenges associated with the data?
Howard Dresner: As explained in www.advancedanalyticsreport.com, our study revealed that only 30 percent of participants’ organizations have a common view of the data available in their enterprise. Most are using semantics and parochial views to support specific positions.
A significant number reported conflicting data; 22 percent said consistent data is available only at a departmental level and functional views of data conflict and cause confusion.
Do BI vendors recognize this problem and provide tools or functionality addressing it?
Howard Dresner: Almost 40 percent of the respondents stated they view deletion of duplicates as the most critical component of data preparation plans. Clearly healthcare has more concerns about data preparation than other industries due to the sheer complexity of health information as well as concerns about privacy and regulations.
And data preparation also has strong support among the software vendors we surveyed. However, only 70 percent today support detection of duplicates or outliers. Of the six components of data preparation we tracked, most vendors indicate they plan improvement in the next 12-18 months.
That said, despite the fact that the majority of software vendors indicating they consider advanced and predictive analytics to be either “critically” or “very” important, more than one-fourth of them place a relatively low priority upon it, suggesting it’s still a developing market.
Besides data prep, what other functionalities are users demanding of vendors?
Howard Dresner: We tracked 12 functionalities in this first A&PA study. Respondents indicated high demand for basic functionalities such as descriptive statistics and hierarchical clustering and less demand for advanced capabilities such as neural networks. This is again indicative of a developing market.
Users, especially in larger organizations, also attach a significant degree of criticality to fast cycle times for analysis.
What did the study reveal about analytics architecture?
Howard Dresner: We tracked respondents’ opinions of the importance of In-Memory, In-Database, In-Hadoop analytics and support for PMML and MPP architecture.
Respondents view In-Memory analytics as critical, along with In-Database analytics. Both are important for scalability and both are most important in the areas of technology and financial services. Our data found that North American organizations prefer In-Database analytics while Europe, Middle East and Africa hold more to In-Memory analytics.
How did you evaluate the vendor rankings for A&PA capabilities that are included in your report on this study?
Howard Dresner: Vendors were ranked according to their offered features, data preparation support, usability functionalities (such as fast cycle time for analysis and automatic creation of models from data), scalability and integration. The report includes a detailed Buyer’s Guide with vendor capability rankings for the top 18 vendors.
So, as you mentioned earlier, your study reveals that predictive analytics may actually be more talked about than practiced at this point, even though most organizations perceive it as very important.
Howard Dresner: As I mentioned earlier, our study found a substantial adoption gap in advanced and predictive analytics between the messaging of vendors, media and analysts versus the actual people who generate analytical output in organizations. SAS has certainly been in mainstream use by statisticians, but that’s far from the business end users. The market isn’t mature enough yet for line-of-business analysts to use it every day.
However, the study also found that the trajectory of analytics among those already producing and using BI in their organizations will likely gravitate to more complex modeling shortly through training, packages from third parties and more useful tools.
I think for BI to completely succeed in the future, it must be pervasive, meaning it will be operational and task specific. That will require just-in-time and in-context analysis. Both require deep understanding of the business processes impacted by the BI decisions. It takes time for organizations to understand this this and be prepared to adapt to such impacts.
Survey demographics: The 679 survey respondents represent a broad span of geographies. North America (United States, Canada and Puerto Rico) represents 53.31 percent of all respondents. The second largest group is the EMEA with 29.31 percent of respondents. Small (1-100) organizations represent about 35 percent of respondents; midsized (101-1,000) account for about 30 percent; the largest of organizations (1,000 and greater) reflect 35 percent of the survey sample.
Howard Dresner is president, founder and chief research officer at Dresner Advisory Services, LLC, an independent advisory firm. He is one of the foremost thought leaders in Business Intelligence and related areas having coined the term “Business Intelligence” in 1989. He has published two books on the subject, The Performance Management Revolution — Business Results through Insight and Action, and Profiles in Performance — Business Intelligence Journeys and the Roadmap for Change. He hosts a weekly tweet chat (#BIWisdom) on Twitter each Friday. Prior to Dresner Advisory Services, Howard served as chief strategy officer at Hyperion Solutions and was a research fellow at Gartner, where he led its Business Intelligence research practice for 13 years.
Kathleen Goolsby is managing editor of SandHill.com.