How social media and big data will unleash what we know

ZDnet reporting:
...In the shorter term, the ability to analyze and mine data in scale within social networks is enabling a range of intriguing and useful applications that can plug into social media networks and make use of the knowledge inside them. Doing it well, however, has proven to be more than non-trivial, such as making making analytic sense of content types that are very large and opaque, such as high definition video, or sensing the connections between the thousands of unstructured natural language messages. Each of these require technologies that can handle the enormous scale, complexity, speed, and computation requirements in a way that remains cost effective inside a rapidly-rising exponential window. From this you can begin to see the challenge of traditional approaches to large data, which tend to break down fairly soon under large geometric growth.
...
Though social networks may soon contain the visible sum of humanity’s communication and interaction, the challenges of deriving what is increasingly called social business intelligence are two-fold. First, big data sets itself apart from previous approaches because it applies new ways of thinking about the capture, storage, and processing of truly vast amounts of data, precisely the kind that emanates from today’s social media ecosystems. This includes the supporting technology, often starting with emerging tech such as data mining grids or MapReduce infrastructures (see my exploration of one example, Hadoop, here) as well as software architecture that is often surprisingly non-deterministic and non-linear in design. For a quick example, see this discussion of LinkedIn’s challenges and counter intuitive solutions to data scale in social networks. In practice, this means that there is a distinct generational and technical divide between how most organizations are dealing with data today, and the very different things they’ll need to do in the future.
Computerworld: As ‘big data’ grows, IT job roles, technology must change.
The second issue goes back to the old adage that “you manage to what you measure.” In the big data world of social media, this means that ones analyzes what you know of to analyze. However, one of the things I hear frequently from business users of social media is that they’d like to “spot trends“, to “know what’s going to happen before it actually happens“, to “get ahead of the conversation and see where it’s going.” Sentiment analysis, knowledge mining, aggregating conversations into trends, these are all possible when you know what you’re looking for (and you have the technology that’s capable.) It also helps — and it’s no mean feat — if the tools you use are smart enough to tell you what there is to know, but that you don’t know to ask for.
http://www.zdnet.com/blog/hinchcliffe/how-social-media-and-big-data-will-unleash-what-we-know/1533