Once upon a time . . . actually just two months ago . . . #BIWisdom tweetchat tribe members were facing off in response to the question of whether the EDW (electronic data warehouse) is dead.
But then at the end of November, along came Amazon Web Services with its preview-launch of Redshift, a fully managed, petabyte-scale data warehouse service in the cloud.
So is the sense of talking about the valid future of the EDW diminished by the new reality of Redshift? I don’t think so, for the controversy exposes the criteria necessary for Redshift’s success.
At one of my Friday #BIWisdom tweetchats in October, the tribe tweeted their opinions on the problems with BI data warehouses in the realities of a Big Data world:
- An EDW is thought of as a system, not as infrastructure.
- Big Data enables one-off analysis, which is the antithesis of the motivations for an EDW.
- EDW and Big Data overlap. Their precepts and methods are highly complementary.
- Big Data hype may be confusing the EDW role. EDW is needed for some mundane tasks, and it’s still important for polystructured data.
- Hadoop, with its flexibility in parallel programming, is better than an EDW in exploring and handling raw unstructured data. But not all have the ability yet to go the full Hadoop route; the skill gap is still huge.
- EDW functionality is on the decline, but not the number of EDWs. They’re as important as ever; but at this point, most systems linked to an EDW can’t integrate with Big Data.
One participant asked, “How can organizations afford to continue paying for every drop of data in an EDW, $ per TB, when data volumes are only growing?” Another agreed: “Speed and agility are huge factors. Customers want their data now and EDW changes are too slow and costly.”
The tweetchat group concluded that the EDW not only is not dead but it’s alive and kicking. It’s just getting back to the basics and going through a state of redefinition, tweeted some. Others tweeted that the EDW process will radically change. It will shift to be line-of-business driven, iterative and more dynamic, not designed upfront and not IT owned, predicted another.
First glimmerings of Amazon Redshift show AWS may be on the right track. AWS claims that Redshift enables analyzing virtually any data set, has a low on-demand pricing structure and scales linearly up to more than a petabyte. And Amazon’s expertise in data warehousing is evident with its highly successful book business based on one of the world’s largest and most complex data warehouses in existence.
Even so, the #BIWisdom crowd tweeted that trust issues will exist for some time: “Will organizations trust in a cloud solution for critical analytics?” A participant tweeted that “The problem lies not in the provider but rather in the cloud itself. That’s why private, on-premises clouds appeared.”
Bottom line: We need to resolve the security issues and get beyond it being the stumbling block to advancement for cloud business intelligence solutions. AWS states that Redshift supports most current security standards. Will Amazon Redshift be powerful enough to change the negative perception of placing enterprise data in the cloud?
In addition, it’s not clear how a massive cloud warehouse solution like Redshift will be able to address customer diversity. Quoting one of the #BIWisdom members, “To be successful, a cloud data warehouse solution will need to be similar to a Swiss Army Knife, having multi-purpose tools that address varying requirements.”
Although perhaps limited, I think Redshift is a visionary move that will be followed by others. What do you think?
Howard Dresner is president, founder and chief research officer at Dresner Advisory Services, LLC, an independent advisory firm. He is one of the foremost thought leaders in Business Intelligence and Performance Management, having coined the term “Business Intelligence” in 1989. He has published two books on the subject, The Performance Management Revolution — Business Results through Insight and Action, and Profiles in Performance — Business Intelligence Journeys and the Roadmap for Change. He hosts a weekly tweet chat (#BIWisdom) on Twitter each Friday. Prior to Dresner Advisory Services, Howard served as chief strategy officer at Hyperion Solutions and was a research fellow at Gartner, where he led its Business Intelligence research practice for 13 years.