Skip to main content

How to Eradicate Data Silos and Keep Up with the API Explosion

By August 21, 2012Article

It’s undeniable that we’re awash in data. With more data being created than ever before, companies must find new ways to identify, organize, move and consume relevant information or they will be left behind. Enterprises are constantly struggling to get rid of data silos and effectively connect business applications like their ERP, CRM, HR, marketing, customer service and financial systems in order to reduce data friction. However, in order to take advantage of the explosion of new Software-as-a-Service (SaaS) choices and keep up with the ever-growing set of disparate application programming interfaces (APIs), companies clearly need better, longer-lasting approaches to integrating data.
I propose that the most important technique we can embrace in our quest to eradicate data silos is the containerization of data. It is crucial that we establish some kind of standardization in how we move data so it can easily be plugged into or pulled out of any application for business use. We have an opportunity similar to that of the shipping industry in the 1950s. Back then, intermodal, standardized containers were adopted to tear down the barriers to moving goods from boats to trains and/or trucks, which revolutionalized global trade. I believe we’re at a similar inflection point when it comes to how we move enterprise data between the growing universe of business applications.
If we do data containerization right, we can make data more manageable and transportable across enterprise applications, despite today’s broader variety of data types (unstructured vs. flat files), bigger volumes of information (Hadoop clusters vs. data warehouses), and faster velocity of data (real time vs. batch). Think about what’s driving the vast majority of our data explosion — it’s not humans, it’s machines.
Machine data from RFID sensors, weblogs, etc. is producing exabytes of helpful customer and supply chain information that is all but impossible to move around with traditional techniques because of its real-time nature and lack of structure. At the same time, most enterprises’ legacy data from relational databases and on-premise systems still offers tremendous insight about historical business trends and needs to be consumed alongside new forms of social media and machine data.
Here’s what I believe are the top three things driving a need for the containerization of data:
1) Agility requires adaptability.
If you want to run an agile business, you must expect that you’ll be changing your SaaS systems much more frequently than you ever changed on-premise systems. Back in the day, couples buying their first homes would invest in sturdy furniture sets to last a lifetime; but today most people plan to redecorate and purchase new furniture several times over the years. People’s tastes change all the time, and this is true for enterprise apps as well; so you should expect to replace business apps as often as you replace your household gadgets.
Say you want to switch your HR system from PeopleSoft to Workday or your CRM from Seibel to Salesforce. Depending on what approach you take, these kinds of changes can be migraine-inducing. With all the wonderful SaaS choices in the market today, it’s important to leave yourself the flexibility of easily moving between vendors to best support your business. Every enterprise can benefit from instituting both a data “Bill of Rights” and a data containerization approach. Otherwise, leaving a SaaS vendor could mean saying farewell to tons of critical data about your customers or employees.
2) But not everything will change.
Even as more and more CIOs are embracing the cloud, we’re finding that some enterprise applications just may not get to the cloud as fast as we’d all like them to. At a recent advisory board meeting with several of the finest minds in IT, it was very clear that finance and supply chain apps are the least likely to change in the near future due to security and reliability concerns. Again, a containerization approach to this data can help you manage a hybrid ecosystem, so you can easily access data that will be on premise for a long time (like financial accruals or inventory levels), right alongside your cloud data (like consumer demand forecasts from social media insights).
3) We’re all on the move.
The rise of enterprise mobility has been discussed at length. When it comes to business data, it’s no longer enough to give your employees Web access to their apps. Whether they’re on sales calls, providing on-site customer service or trying to stay on top of things during their kid’s soccer game, your employees want mobile access across their smartphones, tablets and e-readers. Data containerization is key to making that possible because it can help enterprises securely move important data to and from IT-approved enterprise mobile apps.
Three key technical considerations
Once enterprises decide to embark on this journey, three key technical considerations of containerizing data are:
1) Modern Web technologies.
From my perspective, REpresentational State Transfer (REST) has created a new realm of possibilities for data sharing and movement. REST is more flexible and simple than other approaches commonly used in the enterprise because it lets you connect with site-specific APIs and publish data anywhere. REST’s API-centric style works like a Web server and assigns a unique identifier with a path and name for every object or resource in your enterprise so that it’s all easily callable. This kind of loosely coupled approach is critical for the massive scalability that you will need in order to deal with Big Data.
2) Reusability.
Because change has become the only constant in today’s business world, we need data containers to be future proofed. The most efficient way to do this is to make them adaptable to new data types and applications while also leveraging reusable technologies and components that save developers time. Things like language-neutral connectors and templates for data can speed up your integration work and also ensure that data containers will work for any new application or data source that you want to connect down the road.
3) Open development environments.
No two IT ecosystems are the same, and your enterprise’s application portfolio of 2012 will most likely be different from your portfolio of 2013. Given this reality, companies should employ an open approach to data containerization. This can provide access to a wide variety of data connectors for database connectivity, business productivity, business intelligence, analytics, social media and more. It also puts the power in your own hands, so your company can build its own containers for any custom application or niche data source that’s important to you.
What all of this comes down to is taking advantage of your choices. The hundreds of SaaS and cloud applications out there are of no use to you if you’re not equipped to swap things in and out of your enterprise. Some of the most important decisions a CIO or CFO makes are which applications to leverage for optimal business insight and productivity. You need to quickly get rid of apps that aren’t working for your company and embrace new tools to support things like global expansion, emerging marketing trends, etc. For example, SnapLogic customer Fusion-io recently implemented a data containerization approach that allowed the company to move from one ERP system to another in just a few weeks – not usually an easy task for a public company with compliance requirements.
Some think that the explosion of Application Programming Interfaces (APIs) is the answer to free-flowing data in this era of change. But while APIs play a crucial role, they’ve been around since 2000 and they’re still not a solution on their own. The 5,000 APIs out there have been published in a wide variety of formats — from REST to SOAP to XML; and most SaaS vendors update their APIs three to four times a year, which really makes it challenging for developers and traditional integration platforms to keep up. It’s simply not realistic to expect that our industry will standardize all these APIs or that we’ll magically find structure in unstructured data.
Because the move towards SaaS environments actually makes obsolete the architectures of traditional integration platforms many companies are still using, switching applications in and out is unfortunately a huge challenge for them. Without data containerization, these companies are basically trading their on-premise shackles for cloud handcuffs. A new generation of standardized data containers is needed to finally disperse the gridlock of information and allow enterprises to easily embrace any application option that benefits their businesses.
Gaurav Dhillon is the CEO of SnapLogic, which offers the market-leading cloud integration product for the enterprise, and counts some of the leading companies in the Global 2000 as customers. Previously, Gaurav co-founded and served as CEO of Informatica, which he led from a startup idea to a public company and a leader in the data integration vision he pioneered for enterprise software.

Copy link
Powered by Social Snap