A decade after the Internet made possible whole new ways of doing business, Big Data is emerging as the next tool to accelerate productivity growth and improve effectiveness. The ability to collect, analyze and mine vast amounts of data at a reasonable cost gives businesses and government agencies the potential to generate rich real-time insights to guide their operations and strategies. Indeed, we believe that the effective use of Big Data will become a key basis of competition for organizations in every sector.
Big Data will also provide a whole new set of opportunities — and challenges — for the software and IT services industries. Based on recent research conducted jointly by the McKinsey Global Institute (MGI) and McKinsey & Company’s Business Technology Office, we believe that the increasing use of Big Data is clearly a disruptive trend. It will not only change how end-user organizations operate but will also challenge the status quo in software and services. Companies will need to master the emerging Big Data technologies and techniques to build strategies that take advantage of them.
The changes are already apparent. A new crop of startups is building specialized products, often for vertical markets to serve the Big Data market. Incumbent suppliers of database and business intelligence products are investing in ways to bring Big Data applications to their customers. Companies in the IT services and data markets are positioning themselves to address the Big Data opportunity.
Before we discuss the implications for technology product and service vendors, it is useful to understand Big Data’s potential for end-user customers and some of the obstacles they will have to overcome to use it. In our research we found that the Big Data market will span all industry sectors — even now, the average company with 1,000 or more employees has more data on hand than the Library of Congress. Many industry sectors have more than one petabyte (1,000 terabytes) of stored data. These enormous data sets are used to create transparency (to increase operating efficiency); enable experimentation to improve performance and speed product development; create new possibilities for customer segmentation; augment or replace human decision making; and to innovate new business processes and services.
Major Big Data opportunities
We studied the potential for Big Data in five areas: the U.S. healthcare and retail sectors, public sector administration in Europe, and manufacturing and personal location data globally. Among our findings:
- In the U.S. healthcare sector, Big Data can be used by providers of patient care (e.g., for remote monitoring, providing outcomes-based medicine), by payors and by pharmaceutical and medical products companies. Altogether, we estimate Big Data can help create $300 billion in annual value through these applications. Two-thirds of that could be realized in cost savings (about eight percent of total U.S. health care spending).
- Retail has been a leading user of large data sets, but still can extract considerable additional value from Big Data. Applications range from location-based micro-segmentation to sentiment analysis (tracking consumer attitudes via social media) to optimizing operations. Overall, we estimate that Big Data can improve often-challenging retail margins by up to 60 percent and boost productivity by 0.5 percent annually.
- For manufacturing, we estimate that Big Data could help reduce product development time by 50 percent and cut capital investment by 7 percent.
- The use of personal location data gathered from cell phones and other electronic devices can generate $100 billion in value for service providers and $700 billion in value to consumers; the biggest benefit would be smart routing for vehicles, which could save $500 billion annually by 2020. This is equivalent to saving drivers 20 billion hours on the road and about $150 billion in fuel.
- Governments in Europe could save more than €100 billion (~$149 billion) in annual operational efficiency improvements alone by using Big Data; this does not include Big Data tools to reduce fraud and errors and boost tax collection. Similar gains could be realized in governments elsewhere.
These are just some of the potential application areas for software and services vendors to target with Big Data offerings. Other markets with considerable potential include financial services, transportation, infrastructure and logistics, and professional services.
Customer pain points
Before the full benefits of Big Data can be realized, however, technology vendors need to help customers address obstacles and “pain points.”
The most obvious is learning to manage and analyze larger data sets. Also, many of the biggest benefits arise from the ability to share large data sets from different sources, which often involves integrating data from multiple legacy systems. At the same time, many companies will also need to access information from third parties and integrate it with their own, requiring a set of capabilities that few companies have today.
As the use of Big Data becomes a basis of competition, ensuring data security will become even more important and challenging. Companies will need strong capabilities for managing intellectual property and for fulfilling regulatory compliance requirements efficiently. Many companies will face new privacy concerns among consumers and other stakeholders.
One looming challenge – which will be felt both by end users and by software and service vendors – is finding Big Data talent. According to our estimates, in the next seven years, the United States could need 140,000 to 190,000 more workers with the deep analytical training (in statistics or machine learning) than will be in the workforce, given current educational and training trends. In addition, under current trends, the U.S. workforce will have 1.5 million too few people with the managerial and quantitative skills to make decisions based on Big Data inputs in 2018.
Software and services opportunities
Software and services suppliers have multiple opportunities to help their customers take advantage of Big Data. Incumbent database and information management companies are expanding their offerings to help their customers manage the growing deluge of data. But startup companies are also entering this space, focusing on technologies such as Hadoop and NoSQL databases that have been developed specifically to solve Big Data problems.
Similarly, suppliers of business intelligence tools are expanding their toolkits to include products for massive data sets while new competitors emerge to target industry verticals or business functions with Big Data analytics. We see the same kinds of industry dynamics in the market for data integration, where incumbent Extract, Transform and Load (ETL) providers find themselves competing with younger, entrepreneurial companies.
The Big Data trend is also creating opportunities in services. Companies that offer IT implementation, applications development, and management services are positioning themselves to support Big Data deployments. At the same time, providers of cloud services are also logical candidates for meeting Big Data needs. The looming talent gap that we have described opens up the possibility for both kinds of service providers to furnish the Big Data expertise that end-user organizations may not be able to maintain in-house.
Finally, Big Data is blurring categories between technology and information suppliers. Providers of data, which traditionally have been considered part of the media industry, are adding analytics, encroaching on markets that had been considered the domain of software and services players. We also expect software and services players to add data feeds to enhance their value propositions, and some have already begun to do so.
As managing Big Data becomes a critical competency for end-user organizations, the software and IT services industries have tremendous potential to meet growing customer needs. There will be more new companies and more pressure on incumbents to innovate. For these companies, Big Data will create big opportunities.
Brad Brown is a director in McKinsey’s New York Office; Michael Chui is a senior fellow with the McKinsey Global Institute (MGI), based in the San Francisco office, where James Manyika is a director and a director of MGI. They are co-authors of “Big Data: The next frontier for innovation, competition and productivity,” which is available here.