Critical path capabilities on your social integration journey

Combining the Social Integration Journey and the basic building blocks of your solution, let us look at what could be the capabilities that figure in the critical path of this journey.

The objective is to help you decide on where to invest your efforts if you are building out a solution on your own for your enterprise or if you are looking at buying a solution, what would be the critical capabilities depending on where you are, on your social journey.

Aggregating the functionality across the basic building blocks, the key capabilities are:

• Wide Social Data Source Coverage: For Listening and Monitoring, it is essential to "cast the net wide". I would go as far as to say that you should in fact include search engine results as a key component of discovering the hot spots of relevant activity on social media! You should look for support for both API-based collection as well as Web content extraction (which has definitely become way more involved than what used to be brute-force scraping techniques). Remember that the APIs are still evolving fairly rapidly and the solution should be able to evolve at the same pace too. You might also need historical data for certain use cases.
• High Data Volumes: As a corollary of the wide coverage, you will also need the ability to handle large raw data sets. You might also have to handle real-time streaming sources (which are being recommended by the social networks more) for large data sets. Aggregators like Gnip and DataSift also provide streaming for large result sets.
• Data Quality/Cleansing: To improve the Signal-to-noise ratio in the raw data set, you should be able to apply tough data cleansing/filtering rules. These could be in the form of entity recognition and matching thresholds. This could also involve use case based relevance rules for e.g. if you are looking to build the network profile of a customer, you might not be interested in the details / sentiment of their activity stream. You should be able to leverage a library of DQ rules if possible.
• Text Analytics: You will need powerful semantic and sentiment analysis capabilities to infer key signals from all the data flowing through the system. If you operate in multiple geographies, you will need the ability to do this analysis across multiple languages.
• Enterprise Data Access: A lot of value lies untapped in the intersection of the social and enterprise data domains. You should be able to seamlessly work with CRM, ERP, PLM and MDM system data as you add the social dimension to the data. 
• Collaboration: As you move further on your social journey, it is important to facilitate collaboration both among employees and between employees and customers. At the minimum, your solution should be able to interface with existing / established collaboration systems so that end-users do not need to switch between multiple screens to share/consume data.
• Publishing: Content is king in social media. Community building is the queen probably! You need well-integrated Content Publishing capabilities or at least the ability to reference/identify content items in your overall solution for end-to-end analysis of results. You will also need community platforms where you can engage and innovate with highly influential customers and influencers. 

Here is my take on the critical path of capabilities:

What do you think are the capabilities on the critical path of your social enterprise journey?

Do not depend on Facebook (alone) for customer insights

Diving deeper into one of the key areas where all enterprises can find value from integrating social media, customer profile enrichment with the social dimension can help several business functions.

Even though I have been thinking that companies need to maintain their own customer profiles, I got emboldened to take a stronger stance, by this white paper (Regn./PDF) from @gxsoftware on a related subject of avoiding over-dependence on external social networks like Facebook to gain deeper insights on their customers.

Though my headline has Facebook, this applies to all of external social media sources.

Factors to consider

Here are the factors you should consider in your customer profile building strategy:

1. Your data needs may not be easily satisfied: Though the social networks do try and expose as much data as possible through APIs and Aggregators, you might not get the data in the most efficient manner for your specific needs. There are myriad restrictions on APIs (data volumes, number of calls at various levels like IP Address, App, API and User) which need to be negotiated through to get the data. Also, not every type of activity on the social network might be of interest to you.
2. Privacy policies unclear and likely to toughen up: As Ray Wang lays out demands in his recent post, users are likely to press for better privacy controls on their favorite destinations like Facebook. In fact, there are startups offering fine grained user-driven privacy settings e.g. Diaspora. So you should realistically expect unpleasant surprises on non-availability of data that you assumed will be available forever!
3. Data Ownership is also unclear: Even if you had the data available, the social media that hosts the data and/or the end-users may assert uncomfortable-sounding rights on the data and its permitted usage/storage etc. This may lead to expensive workarounds/solutions to adhere to these rules if you haven't taken care to design it upfront.
4. Data access may become pricey: Finally, even as you work through these issues, you might end up staring at steep fees to access the data. After all, Facebook and the like will monetize the data in as many avenues as they can (advertising, data access, etc). 

The WP lists a few other factors like losing competitive advantage since everyone can get the same data access - these are valid factors too but they are almost a given in the social world.

What can you do?

As you might have realized, many of the factors are beyond the enterprise's control. So the enterprise should have a plan of action to mitigate the downsides. Some of the elements in that plan could be:

1. Start Listening now: Even as the privacy and data ownership rules continue to toughen up (and they should), you should be collecting as much data as possible (while of course still adhering to the T&C's of the social media!) by engaging with customers and listening to their activity streams. Start your social media integration journey now.
2. Look at a policy-driven enterprise social platform: As you start listening, remember the value of an enterprise-wide platform leveraged by multiple business functions for multiple use cases. One of the first issues you will have to solve is to gain connectivity to the social media (see factor#1). In a subsequent post, I will describe the challenges in this area and what to look for in a possible solution. The platform should ideally have policy-driven (and automated) models pertaining to data ownership, access rights and storage restrictions e.g. you should be able to dynamically model a source as having members-only access with data storage restrictions on certain fields in the incoming data. The platform should be able to enforce the data access and data archival based on these settings. At scale, poilcy-driven automation is key.
3. Create your own Enterprise Social Graph: We shouldn't expect others to do the heavy lifting for our business needs. You should evaluate how you can mashup enterprise and social dimensions for your key entities like Customers, Products and Suppliers. If you already have a MDM-based Customer/Product Hub, it would provide an ideal platform for this. What you might end up with is a logical graph which captures the dense relationships between and among these entities. Ideally, all your business cases should be able to query this graph to get the required information. This is truly Big Data territory - a salivating possibility for another post!
4. Invest in community building: Now that your customers see enough value in engaging with you across multiple channels (mainly social media since it is ubiquitous), you should be able to invite them to participate in your own communities or at least on forums where you have access and maybe even ownership of all the data. The data access terms are now more favorable to you.
5. Leverage this graph in CEM and PLM: The Enterprise Social Graph can now be even more relevant and enriched with very detailed customer profile information. The rich profiles that this graph provides for your customers and products is ripe for leveraging in improving the customer experience and your products.

What is your take on how the data access situation will play out in the medium-term? Where are you storing / planning to store the social dimension of your customers and products?


_______________________________________________________________________________
Ram Subramanyam Gopalan - Product Management at Informatica
My LinkedIn profile | Follow me on Twitter
Views expressed here are personal and do not necessarily represent those of Informatica.
_______________________________________________________________________________

Building blocks for your Social Data Integration solution

In the previous two posts, I have presented a basic approach for enterprises to embark on the Social Media integration journey and took a stab at how to view the wide array of use cases through the lens of data domains and departmental functions. Ideally, a long-term solution - what we might lengthily call a Enterprise Social Data Integration Platform - should support all four categories of use cases and also all (as many as possible!) departments in the enterprise.

Venturing into system building territory, a basic functional flow will help in slotting the basic building blocks of such a solution. In essence, such a flow comprises:

• Setting up listening and publishing posts on various social media: This is a continuous process of identifying hot spots of relevant activity in the social world and setting up a process to listen and participate in the most effective manner. It is interesting to note the trend of such connectivity needing to go beyond the top-3 or even top-10 social networks to very niche, highly-specialized discussion forums and industry blogs. Do you have a well-defined and maintained list of the top social media hot-spots for your company or even industry?
• Collecting and optionally staging raw data: As the raw data comes in from various sources, it needs to be collected and fed into further analysis steps to start extracting value from it. Each source comes with its own standards and formats for the data as well as myriad other considerations like security credentials, API rate limits, automated agent limitations, etc. There are emerging standards like Activity Streams and OpenSocial but there is no broad mainstream convergence on any of these yet. So for the foreseeable future, the solution needs to work with all the low-level complexities of social data.
• Cleanse and prepare the raw data for analysis: Once the raw data is collected, it is imperative to improve the signal-to-noise ratio before doing heavy-duty analysis on the data. Traditional Data Quality methods will have to be adapted to look at Entity extraction based relevance scores and removing data sets falling below a threshold value of relevance. 
• Generate Insights out of the raw data: Depending on the use case, specific text analytics/algorithms and business analytics routines will have to be applied to the cleansed data. Basic routines would include Semantic Analysis, Entity Extraction, Sentiment Analysis and Influence Analysis which would apply to individual "records" (an "activity" in the social world). These routines add annotations, if you will, to these activity records. These records and their annotations can then be summarized and sliced/diced depending on the end business use case. For some use cases, complex event processing will be required to quickly correlate events in discrete systems to find emerging patterns.
• Most importantly, Evoke Actions: Of course, all the analysis in the world is useless, if you do not act on it (which does include getting to the conclusion that no action is required!). In the social world, action can manifest in traditional enterprise systems and channels in various ways like sharing content, designing more refined customer experiences, building communities and collaborating both internally and externally.
• Enterprise Data Integration: This will be required to be integrated either as input or as a target for insights/enriched data. As you might recall from the data domains discussion, the most valuable use cases are in the intersection of enterprise and social data. As an example, for people related information, identity matching between CRM records and social handles provides very powerful capabilities to understanding them better. As is evident from this example, seamless integration with enterprise systems and social media is required for such mashup analyses.

Much of this flow applies to all use cases. Of course, you should close the loop eventually by feeding back inputs from each downstream stage to the listening stage!

The basic building blocks of such a solution can then be derived from this functional flow:

There is an interesting component - Information Lifecycle Management (ILM) - which is often an afterthought in such solutions, but shouldn't be! ILM becomes vitally important in Big Data scenarios (of which the social data is definitely one). ILM helps in policy-based automated management of data lifecycle - creation, validation, integration, archival, deletion.

Do you see any important pieces missing in this list? What are the key issues / success factors that you see with these components?

In subsequent posts, we will look at some interesting issues in some of these components.


_______________________________________________________________________________
Ram Subramanyam Gopalan - Product Management at Informatica
My LinkedIn profile | Follow me on Twitter
Views expressed here are personal and do not necessarily represent those of Informatica.
_______________________________________________________________________________