Editor’s note: End user data preparation (EUDP) is a “self-service” capability for end users to model, prepare and combine/manipulate data prior to analysis for business intelligence insights. Users can leverage EUDP for independent use or to complement traditional IT-driven data quality/ETL processes. As a significant amount of relevant data now lives outside of organizations and the importance of users’ ability to gain new insights by exploring those data sources increases, EUDP is becoming an issue in hybrid data environments and in agile BI processes. This interview of Howard Dresner explores the issues in end user data preparation and presents findings in a Dresner Advisory Services survey and market study of EUDP.
Q: When in a business intelligence project life cycle are people using end user data preparation?
Howard Dresner: It’s used for any interactive forms of BI, including data discovery / data exploration and data mining and advanced analytics. It can be a key factor in faster analysis and faster time to insights. Organizations can gain new value by integrating disparate data, and EUDP enables users to quickly explore various data sources to gain new insights. Our market study found that end users are really excited to combine data and build models themselves. Sales and marketing especially are key users, and they like to leverage external databases in their market and customer intelligence analyses.
Q: If users manipulate data, how trustworthy and valuable is the end result?
Howard Dresner: In a more traditional “governed” environment, IT employs various tools to prepare data for users in a data warehouse. BI tools are given to the user population, but control of the data access and models is centrally governed. There needs to be an effective balance between data governance (for security, etc.) and end user data preparation. EUDP is an important step in the ongoing trend of user empowerment and self-service in business intelligence.
The best practice process in EUDP is to enable users to find value, then verify and justify the value, and then integrate the valued insight into information for corporate strategy and objectives.
Q: Are the insights and value from user-manipulated data mostly strategic or tactical?
Howard Dresner: It’s both. But it’s important to note that our study found more than half of the surveyed companies consider end user data preparation either critical or very important. And they rank EUDP in the top half of initiatives they consider strategic to business intelligence – in fact, they rank it above data mining, advanced algorithms, SaaS/cloud and collaborative support for group-based analysis.
Of course use of EUDP functionalities varies by industry (with manufacturing being the greatest users); but overall, 66 percent of respondents said they constantly or frequently make use of end user data preparation, and 21 percent said they require EUDP occasionally.
Q: So EUDP is a critical and strategic capability in many organizations’ business intelligence. But the capabilities and EUDP expectations are evolving quickly. So how effective are the current tools and approaches to end user data preparation?
Howard Dresner: Only 14 percent of the survey respondents reported their company’s current approach to EUDP is highly effective. This is across the board, no matter the size of the organization.
Q: What is causing the ineffectiveness? Are vendors not yet supporting the functionalities that users require?
Howard Dresner: Some of the challenge relates to user education and data literacy. Before creating/building data models, users need to have an understanding of the underlying data, its relationship to other data and how to correctly combine and prepare it for analysis.
Beyond that, most users are not using the proper EUDP tools to prepare their data. Instead they’re using spreadsheets – which are great general purpose tools, but not especially well suited to preparing data for analysis.
Q: Did your study examine how frequently users access databases outside their organization?
Howard Dresner: One-third of respondents enrich their end-user data models with third-party data either constantly or frequently, and more than 60 percent use third-party data at least occasionally.
Q: Does use of third-party data vary much according to type of user?
Howard Dresner: Our 2015 report on end user data preparation segments all the findings according to size of organization, departmental functions, vertical industries and geographies. Industries that are sensitive to customer attitudes and look at loyalty/churn metrics are the most frequent users of third-party data enrichment. By function, sales and marketing are the biggest users, and they depend on information regarding customer demographics, mapping and social media. Executive management also is a big user, as they are interested in views of competitive performance.
Both large and small organizations employ third-party data constantly or frequently, whereas midsized enterprises use it less frequently.
Q: You mentioned earlier that users find value in their manipulated data, verify and justify it, and then integrate it into other corporate information. Do EUDP features provide an automated functionality for integration?
Howard Dresner: EUDP integration features are in high demand. We asked study participants to rank the importance/priority of various integration features, and 60 percent reported that even the least-desired feature – the ability to infer metadata by introspecting data elements – is either critical or very important to their business.
Providing greater support by incorporating more automation in integration features in EUDP is a clear opportunity for vendors.
Q: Is end user data preparation still an early-stage market in business intelligence solutions?
Howard Dresner: Yes. The fact that respondents reported they have very high interest in the full range of EUDP usability features reflects market immaturity and unmet demand for greater self-service and user autonomy. For example, more than 70 percent ranked visual user interface and non-technical/non programming user access as critical or very important to their business. And 60 percent ranked visual highlighting of data relationships, along with a variety of automation features, as critical or very important.
Almost no respondents indicated that any of the many usability, integration, manipulation, output and deployment EUDP features we sampled are not important to their business. Again, this indicates broad opportunity for product and service providers.
Q: Are EUDP features and capabilities separate tools, or are they integrated into business intelligence solutions?
Howard Dresner: Both. But we asked the survey respondents their preference. Only 16 percent said they prefer to use end user data preparation as a standalone tool. Nearly 60 percent prefer that it be part of their selected BI solution, and 26 percent prefer that it be included with data quality or integration tools.
Click here to access the 2015 End User Data Preparation Market Study report. The report covers key issues including perceptions and intentions surrounding end user data preparation, end-user requirements and feature sets (usability, integration, manipulation features, output options and deployment options), degree of alignment between user requirements and vendor capabilities, vendor support and vendor ratings.
Howard Dresner is president, founder and chief research officer at Dresner Advisory Services, LLC, an independent advisory firm. He is one of the foremost thought leaders in Business Intelligence and Performance Management, having coined the term “Business Intelligence” in 1989. He has published two books on the subject, The Performance Management Revolution — Business Results through Insight and Action, and Profiles in Performance — Business Intelligence Journeys and the Roadmap for Change. He hosts a weekly tweet chat (#BIWisdom) on Twitter each Friday. Prior to Dresner Advisory Services, Howard served as chief strategy officer at Hyperion Solutions and was a research fellow at Gartner, where he led its Business Intelligence research practice for 13 years.
Kathleen Goolsby is managing editor of SandHill.com.