Skip to main content

Big Data and Insight as a Service

By May 8, 2012Article

We hear a lot about Big Data these days, and rightly so. We are generating large data sets at increasing rates from a variety of sources. The Web (desktop, social and mobile), sensors, and most recently cloud-based applications are all contributing to the massive flow of data. It is estimated that this year alone we will generate 2.7 thousand exabytes of data. In a recent speech, Eric Schmidt, Google’s executive chairman, reported that we are generating five exabytes of data and information every two days, and the pace is increasing. But just because we can capture a lot of data doesn’t mean that all the data will be useful.

As a recent McKinsey report indicated (see figure below), different data carries different value and this value may be industry specific.

The value of data is established through its analysis and the effective actions that can be generated as a result. Analyses that lead to impactful actions create insight. The cloud can be used effectively for the generation and delivery of insights from data. This leads to a new class of analytic applications that I call “Insight as a Service.” Vinod Khosla recently wrote that reducing, filtering and processing data streams to deliver relevant information or action is extremely crucial to our ability to deal with the data we generate.

Definition of Insight as a Service

Analytics as a Service and Data as a Service are specialized classes of SaaS applications that have arisen from the increasing acceptance of cloud computing.

Insight as a Service describes cloud-based, action-oriented, analytic-driven applications and solutions that operate on Big Data. This definition differentiates Insight as a Service, which I associate with insight and action, from other types of cloud-based solutions. Other solutions enable users to generate reports or perform analytics. They do not link the conclusions derived to a specific action or set of actions.

For example, a cloud-based solution analyzes data to create a model that predicts customer attrition. It then rates the customer base to establish their propensity to churn. This is Analytics as a Service. Insight as a Service targets the next layer of the analysis. It provides a cloud-based solution that establishes each customer’s attrition score, then automatically identifies the customers to focus on, recommends the attrition-prevention actions to apply to each, and finally, determines the portion of the marketing budget allocated to each set of related actions.

To provide insights, these applications incorporate a variety of data sources. These include proprietary corporate data, syndicated and open source data, and data generated by other SaaS applications.

SaaS applications have also emerged as important generators of Big Data. SaaS applications generate four types of data:

  1. Application usage data: how many times an application was accessed, by whom, what activities were performed during the session, how long the session lasted, etc.
  2. Application performance data: how well the application is performing, what is the response time, what is the uptime, how well the APIs are performing.
  3. Customer performance data: how well a company is doing based on their DSOs, outstanding AR, product delivery times, etc.
  4. Customer feedback data: corporate or consumer customer suggestions, complaints, referrals, etc.

The data generated by SaaS applications remains unexplored.

Two problems in analyzing Big Data

Today we face two problems analyzing Big Data including the data generated by SaaS applications:

  1. Our ability to analyze data is increasing more slowly than the types and quantities of data to be analyzed. We lack adequate numbers of data analysts. Business users don’t have time to build analytics and derive insights. By 2018, the United States alone will face a shortage of 140-190K of people with deep analytical skills. The shortage of managers with enough know-how to make effective decisions utilizing the analysis of big data is even higher, at 1.5M (endnote 1).
  2. Analytics are good for pattern extraction. However, they can’t result in insights that drive performance-enhancing actions. Deriving insights that result in effective actions is difficult and time-consuming and is often outside the purview of analytic solutions.


Today, the few insights we can derive are collected manually and mostly with the help of specialized personnel, called Connectors. Connectors translate a business problem to a data problem. The data problem can subsequently be addressed by data scientists. The Connectors then form the extracted patterns into insights. A business user can manually formulate the appropriate actions from the Connector’s insights.

Let’s review an example most of us are familiar with. A cable TV company wants to allocate a budget to reduce attrition of premium movie channel customers. To develop an effective budget, they execute a series of steps. They develop predictive models that score the customers by the likelihood that they will subscribe to the movie channels. They may even score the probability that the customer will upgrade their service by adding premium sports channels.

The Connector works with the cable company to understand the characteristics of the attrition problem. He sources the data that is available to pinpoint the problem. He defines the problem in terms the data scientist can understand. A predictive model is then developed by the data scientists.

Once developed, the Connector steps in again to work with the company to determine which segments of customers are worth paying attention to (insight), what is their expected Lifetime Value (insight), what percent of the marketing budget to allocate to each such segment (insight) and what action(s) to take with this budget.

Finally, using these insights, the cable company can select the most effective allocation of the budget. They could offer a package of premium movie channels for free for a period of time to some users. They could upgrade another customer’s broadband Internet service. Both actions have a cost to the company. The insight helps determine the best use of marketing dollars for different customers and provides the most effective use of capital for the company.

Connectors are hard to find. Today most of them are independent consultants. However, their extensive and continuous use by corporations is not feasible, mostly for financial reasons; they are too expensive and in too high demand. Even companies like IBM that have large analytics groups cannot meet their clients’ demand for Connectors.

Insights are hard to derive and connectors are hard to find. The cloud provides some important ingredients to address these two problems:

  1. An efficient and cost-effective environment for collecting the SaaS application data and for distributing and managing other data types.
  2. An environment enabling collaboration among teams of data scientists, analysts, connectors and domain experts in utilizing, refining and applying the insights and associated actions.
  3. Elastic computing and storage resources. Insight-generation solutions are even more computationally and storage intensive than simple analytic solutions.
  4. A medium for rapid insight and associated action dissemination as well as solution deployment and incremental improvement.

The components of Insight-as-a-Service applications

As shown in the figure below, an Insight-as-a-Service application consists of five basic components:

1. Insights generator. This component incorporates one or more models: predictive, clustering, deviation-detection and optimization as appropriate for the task being addressed, e.g., marketing budget optimization. These analytic models may be created within the application under a different analytic framework, e.g., Revolution Analytics, or even crowdsourced, and then imported in the application. Jbara’s application utilizes internally developed deviation-detection analytics to provide insights around customer satisfaction.

2. An explicit description of the business process automated by a specific class of SaaS applications. For example, Host Analytics’ application incorporates descriptions of the corporate financial budgeting, planning and account consolidation processes it automates. Jbara’s application incorporates an explicit description of the customer acquisition and retention business process.

3. Vertical industry knowledge. This is necessary for applications that are automating industry-specific business processes such as logistics for the chemical industry, e.g., Odyssey Solutions, or the customer acquisition process in B2C ecommerce.

4. An action generator. This component generates actions the user can take based on the insights that have resulted from the analysis of data. Jbara’s application offers actions regarding reducing customer acquisition costs based on the results of its deviation-detection analytics, e.g., how much a company’s customer acquisition costs vary from best-in-class costs.

5. User interface. The interface must present the insights and actions in a way that is comprehensible by the business user. Companies like 9Lenses, Host Analytics and Jbara have determined which of the identified insights must be presented graphically and which must be presented textually, often in a narrative form. Finally, the user interface of these applications must also allow for the evaluation of what-if scenarios to determine the impact of different actions based on a particular insight. For example, the application recommends 10 percent of the marketing budget be allocated to reducing the churn of customers with Lifetime Value that is two standard deviations above the norm. The user may then want to assess the impact to churn if only five percent of the budget is allocated to this action.

Insight as a Service is a distinct layer of the cloud stack and is positioned above Software as a Service. Several companies such as Acteea, 9Lenses, JBara, Totango, Startup Genome Compass, as well as a few that are still in stealth, have developed such solutions. Companies like 8thbridge, Dachis Group, and Host Analytics have created Insight-as-a-Service offerings as a complement to their existing SaaS solutions.

Insight-as-a-Service application types

Since I wrote my initial article introducing Insight as a Service, I have identified two types of IaaS applications. The first type operates on data that is primarily managed by SaaS, or behind the firewall, transactional applications (ERP, HCM, CRM), and may be supplemented with syndicated or open source data.

A good example of the first type of Insight-as-a-Service application is offered by Host Analytics. This application uses forecasting techniques to automatically analyze the product, employee-related and other financial data stored in an organization’s ERP and HCM systems. The forecasts provide the CFO with insights about budget deviations and actions on how to address them. It also provides plans for attaining specific financial objectives based on predictions using corporate financial data.

This application can benchmark the budgeting and planning data of two or more corporate customers from the same industry. Two hospital chains could be compared to each other and also to industry-specific best practices maintained by the application. The end result is a formulation of actions on how to improve a corporation’s financial performance. As Craig Schiff recently wrote, benchmarking will become an important component of corporate performance management.

Finally, Host Analytics’ application enables its business users to collaborate and more easily generate insights and the performance-enhancing actions.

Another good example is 8thbridge’s social commerce application. The engine behind 8thBridge combines open source data about Facebook engagement/use, Twitter use, and Klout data with data its corporate customers have about their consumer customers. A customized Social Commerce IQ index is then generated for each customer.

The second type of Insight-as-a-Service application operates on the data generated by SaaS applications.

A good example of the second type of Insight-as-a-Service application is that developed by Jbara. This application analyzes customer performance and customer feedback data and presents actions aimed at improving all identified issues. For example, by analyzing data from Marketo’s SaaS marketing automation application, JBara can provide insights around the potential causes of specific customer satisfaction issues. It can also calculate a customer’s probability to churn over a period of time. It then recommends actions aimed at countering each identified issue.

As with the first type of Insight-as-a-Service applications, this second type can also be used for corporate benchmarking. For example, the applications can identify whether a company is best in class at in responding to customer complaints, whether its Customer Acquisition Cost is best in class and how its customer churn rate compares to that of its competitors.


Insight as a Service is starting to be recognized as a distinct layer of the cloud’s stack (following Infrastructure as a Service, Platform as a Service and Software as a Service) as well as an important extension to analytic applications. Insight-as-a-Service applications are now being developed by new and exciting pure-play vendors. Existing SaaS application vendors view such solutions as a way to clearly differentiate their offerings and help their customers derive more value from their applications.

Data diversity is becoming more complex and the pace of data-generation is increasing. As corporations begin to understand the importance of data-driven decision making, Insight-as-a-Service applications will become as essential and as ubiquitous as SaaS applications.

Endnote 1: McKinsey Global Institute full report, “Big Data: The next frontier for innovation, competition, and productivity,” available free of charge online at

Evangelos Simoudis is a senior managing director at Trident Capital where he focuses on investments in SaaS, Internet and data. Prior to Trident, he spent five years as a partner at Apax Partners, a private equity and venture capital firm where he invested and served on the boards of early and mid-stage IT companies. At Apax he also chaired the firm’s IT advisory board. Prior to entering venture capital, Evangelos had more than 20 years’ experience in high-technology industries, in executive roles spanning operations, marketing, sales and engineering. Follow him on Twitter @esimoudis.

Some portions of this article are comprised from three blogs originally published on the Trident Capital blog.

Copy link
Powered by Social Snap