Building blocks for your Social Data Integration solution

In the previous two posts, I have presented a basic approach for enterprises to embark on the Social Media integration journey and took a stab at how to view the wide array of use cases through the lens of data domains and departmental functions. Ideally, a long-term solution - what we might lengthily call a Enterprise Social Data Integration Platform - should support all four categories of use cases and also all (as many as possible!) departments in the enterprise.

Venturing into system building territory, a basic functional flow will help in slotting the basic building blocks of such a solution. In essence, such a flow comprises:

• Setting up listening and publishing posts on various social media: This is a continuous process of identifying hot spots of relevant activity in the social world and setting up a process to listen and participate in the most effective manner. It is interesting to note the trend of such connectivity needing to go beyond the top-3 or even top-10 social networks to very niche, highly-specialized discussion forums and industry blogs. Do you have a well-defined and maintained list of the top social media hot-spots for your company or even industry?
• Collecting and optionally staging raw data: As the raw data comes in from various sources, it needs to be collected and fed into further analysis steps to start extracting value from it. Each source comes with its own standards and formats for the data as well as myriad other considerations like security credentials, API rate limits, automated agent limitations, etc. There are emerging standards like Activity Streams and OpenSocial but there is no broad mainstream convergence on any of these yet. So for the foreseeable future, the solution needs to work with all the low-level complexities of social data.
• Cleanse and prepare the raw data for analysis: Once the raw data is collected, it is imperative to improve the signal-to-noise ratio before doing heavy-duty analysis on the data. Traditional Data Quality methods will have to be adapted to look at Entity extraction based relevance scores and removing data sets falling below a threshold value of relevance. 
• Generate Insights out of the raw data: Depending on the use case, specific text analytics/algorithms and business analytics routines will have to be applied to the cleansed data. Basic routines would include Semantic Analysis, Entity Extraction, Sentiment Analysis and Influence Analysis which would apply to individual "records" (an "activity" in the social world). These routines add annotations, if you will, to these activity records. These records and their annotations can then be summarized and sliced/diced depending on the end business use case. For some use cases, complex event processing will be required to quickly correlate events in discrete systems to find emerging patterns.
• Most importantly, Evoke Actions: Of course, all the analysis in the world is useless, if you do not act on it (which does include getting to the conclusion that no action is required!). In the social world, action can manifest in traditional enterprise systems and channels in various ways like sharing content, designing more refined customer experiences, building communities and collaborating both internally and externally.
• Enterprise Data Integration: This will be required to be integrated either as input or as a target for insights/enriched data. As you might recall from the data domains discussion, the most valuable use cases are in the intersection of enterprise and social data. As an example, for people related information, identity matching between CRM records and social handles provides very powerful capabilities to understanding them better. As is evident from this example, seamless integration with enterprise systems and social media is required for such mashup analyses.

Much of this flow applies to all use cases. Of course, you should close the loop eventually by feeding back inputs from each downstream stage to the listening stage!

The basic building blocks of such a solution can then be derived from this functional flow:

There is an interesting component - Information Lifecycle Management (ILM) - which is often an afterthought in such solutions, but shouldn't be! ILM becomes vitally important in Big Data scenarios (of which the social data is definitely one). ILM helps in policy-based automated management of data lifecycle - creation, validation, integration, archival, deletion.

Do you see any important pieces missing in this list? What are the key issues / success factors that you see with these components?

In subsequent posts, we will look at some interesting issues in some of these components.


_______________________________________________________________________________
Ram Subramanyam Gopalan - Product Management at Informatica
My LinkedIn profile | Follow me on Twitter
Views expressed here are personal and do not necessarily represent those of Informatica.
_______________________________________________________________________________

Value is in the intersection of social and enterprise data

Continuing on from the previous post, where a phased approach to social media was talked about, I wanted to share my thoughts on the business use cases that enterprises can realize through this journey.

Before jumping into that, I do want to set the context for these posts. I believe every participant in any market needs to have their own perspective on it. So with social media integration - customers, industry analysts, business consultants, systems integrators, enterprises and solution providers all have a point of view on it. As you might have noticed, this is an attempt at a deliberate first principles based thought process which I believe is key for any enterprise software vendor to succeed in any segment especially one which is this dynamic.

That out of the way, let's take a look at the spread of cross-functional use cases across these stages. This list of use cases is mainly attributed to the March 2010 post by Jeremiah Owyang and Ray Wang and a follow-up from Ray in August 2011 (thanks to both!).

This is not exhaustive by any means but a good starting point for enterprises. These use cases span the continuum of two key data domains:

1. Enterprise data (Transaction, Structured)
2. Soicial data (Interaction, Less Structured)

The expectation is that the value of integrating these two domains increases as the overlap increases and is bolstered by collaboration (internal and external). The value of using social data is incremental to what is already achieved using Traditional BI.

There are additional synergies that can be gained by sharing the underlying analytics infrastructure across multiple functions.

Stretching this thought a bit further, in order for enterprises to prioritize their data integration efforts for these use cases, there are four categories that seem to emerge:

1. Traditional BI - Only focuses on internal enterprise data
2. "Inside-Out" - Starts with enterprise data and enriches it with social data
3. "Outside-In" - Starts with an event/trend of interest on the social side and then prods action from the enterprise side
4. "Outside-only" - Focuses on only social data to glean insights

As one would expect, the majority of high-value use cases are those that move from (1) to (2) and (3) - the social dimension gives new insights into enterprise trends. That said, it would be ideal for enterprises to leverage the same solution to cater to all four categories of use cases.

Interestingly, there is another (super)-dimension to this data domain discussion which is the structured to unstructured continuum that exists in both the social and enterprise data domains. Ideally any solution that you work with should be able to handle this whole spectrum of data - tall order indeed but a worthy objective.

This kind of framework does help us (vendors) to know our strengths and opportunities better. Hopefully this will add a few dimensions to the enterprises' thought process on what categories of use cases they want to go after.

It would be interesting to hear your insights and thoughts on this. Do leave your comments here or ping me on Twitter (@ramsgopa).

Now that we have given some thought to the business side of things, I will venture into the solution components (Informatica as a software vendor do need to build a solution finally!) in my next post.


_______________________________________________________________________________
Ram Subramanyam Gopalan - Product Management at Informatica
My LinkedIn profile | Follow me on Twitter
Views expressed here are personal and do not necessarily represent those of Informatica.
_______________________________________________________________________________

Social Data Integration journey - Where are you on it?

There are several "Stages of Evolution" thoughts around how enterprises can go about ingesting social data into their business DNA. Adding to these, from my experience creating CRM and data integration products, here is an outside-in approach for enterprises towards social media.

Companies must first map the customer journey through various social media as they interact with the company. The following depicts such a typical journey (which, in most part, applies to both B2B and B2C businesses):

Note that some of these "Events / Triggers" are actually a cumulative experience for your customers.

Once this map is created, companies can decide on how to start their own journey. A phased approach is prudent especially in the social media arena where missteps can get amplified quickly. The following illustrates this:

Importantly, this is an additive approach in that companies will have to continue to listen and monitor on a broad set of social media while participating and innovating on the most effective among these social media streams.

How are you approaching this massive opportunity? Where is your company on this journey?

In subsequent posts, I'll share my thoughts on how enterprises can derive the most of their investments in social media.


_______________________________________________________________________________
Ram Subramanyam Gopalan - Product Management at Informatica
My LinkedIn profile | Follow me on Twitter
Views expressed here are personal and do not necessarily represent those of Informatica.
_______________________________________________________________________________

Big Data + Big Money = Big Change

The dust has barely settled on last week’s Hadoop World and the blogosphere is still abuzz with the implications of various announcements, analyst reports and miscellaneous provocations that are inevitable in events like this.   No better time than this blogger to jump into the fray! 

My takeaway is the evidence has never been stronger that Big Data is real, and several related announcements and data points bear this out.

2012 Predictions Season already starting - Big Data front and center

2012 predictions season seems to be starting early this year, and not surprisingly Big Data is showing up on everybody's lists. The top 10 list from Nucleus Research caught my eye, not because of the obligatory Big Data listing (it showed up as #5), but because of how relevant Big Data is to several other trends that were listed.

As frequent readers of this blog will soon recognize as a common theme here - the real value in Big Data is in the opportunities it creates, far beyond solving its "big mess" management problems. In other words, maximizing the return on data.

Without going into the details of Nucleus' report (free download can be found here) specific areas that seem particularly ripe for Big Data related opportunity include: