Over the last decade we have witnessed the power of search to revolutionize online shopping and mobile experiences. Recognizing that almost no data is irrelevant, retailers like Amazon have begun assembling millions of bits of information about consumers — the proliferating data “signals” that people and systems now generate — to drive recommendation engines and provide more relevant advertising.
Nearly as well known is the way that mobile application developers are combining location data gleaned from WiFi and GPS signals with social media check-in data, keywords we search on, and websites we visit to provide location-aware recommendations and advice. Just arrived in a new city around dinnertime? Up pops a list of highly recommended sushi restaurants. Recently browsing online shoe retailers? Up pops an ad for a sale at your favorite brick-and-mortar retailer just two blocks from your hotel.
New realms
What isn’t nearly as well known, however, is that many companies, healthcare and government organizations worldwide are now starting to realize that enterprise search can enable new business insight and drive completely new capabilities across their organizations — all with incredible speed. Software providers are increasingly embedding search into their applications to enrich them and provide more value for customers. Search is cropping up everywhere, in ways you may not have realized.
For example, a telecom company is using enterprise search to correlate mobile traffic analysis (e.g., a sudden spike in calls in a certain area) with social media analysis (e.g., tweets about a flood or earthquake) in order to more rapidly detect and respond to events that may impact the network. In this way, the company is able to anticipate potential outages and provision additional resources before an outage occurs.
In cybersecurity, enterprise search is enabling security administrators to combine data on known vulnerabilities and current security activity with network activity, application transactions and employee roles and behavior patterns. This enrichment process is enabling them to find previously hidden links that create a more complete picture of where threats may have occurred and where they may be developing.
In the medical field, enterprise search holds the promise of enabling doctors to enhance diagnoses and preventive care by enriching patient history with extended family history, local diet and exercise habits, geography, current weather conditions, seasonal weather patterns, traffic congestion, occupations and unemployment rates, social and political events — a million signals that can be indexed and analyzed in order to detect links and provide insight so doctors can tailor health recommendations and suggest issues patients need to be aware of. And what is learned from each diagnosis can be used to further enrich the data to improve future diagnoses.
How to turn millions of pieces into insights
To determine how, when and where enterprise search can be used by data-driven applications to turn millions of pieces of disparate data into useful insights, increased revenues or more beneficial outcomes, you should answer a series of critical questions.
1. What are the chief data challenges you face in meeting the goals defined by our core mission? For example, sometimes it’s a simple technology decision. Do you store things in relational databases or take advantage of more flexible technologies like indexes or key/value stores? The harder challenge is what data is available and whether you can obtain that data while respecting the privacy of your users.
2. What questions would you like answered to help meet these challenges? I recently had a discussion with a firm that considered optimizing power usage in their buildings by monitoring employee key card activity to understand when lights could be run at a lower capacity. It was discovered, however, that this was in gross violation of the company’s employee privacy policy.
3. Are there specific datasets that you could combine with your current data to help answer those questions? User behavior is often overlooked in most enterprise applications. Applying tricks from ecommerce such as click scoring, tracking users’ patterns through an application and when a user reaches an end result signaled by leaving the site or logging out can provide a much more productive end-user experience in CRM or ERP systems.
4. What technology do we need to combine and query these datasets? This should include the ability to perform sophisticated keyword searches against a highly scalable, distributed analytics infrastructure that functions in real or near real time.
Enterprise search has become a critical tool for unlocking the full value of Big Data, first by making the analysis of that data accessible to the enterprise, and second, by providing a means to enrich the data and improve the analysis. Companies that move quickly to take advantage of this opportunity will soon enjoy all the promised benefits of Big Data: more business insight, greater agility, faster time to value and increased competiveness.
Will Hayes, chief product officer at LucidWorks, has 15 years of product, marketing and business development experience. Most recently he was head of technical business development for Splunk and defined its market category and key product feature sets. He created and led the company’s global partner program, building the ecosystem to more than 50 consultants, developers, resellers, system integrators, service providers and technology partners. Earlier in his career, he served as a software engineer at Genentech.