The dust has barely settled on last week’s Hadoop World and the blogosphere is still abuzz with the implications of various announcements, analyst reports and miscellaneous provocations that are inevitable in events like this. No better time than this blogger to jump into the fray!
My takeaway is the evidence has never been stronger that Big Data is real, and several related announcements and data points bear this out.
First, we have a ready market, according to a report by the UK-based analyst firm Ovum,titled “Big Data Interest Bubbling under the Surface”. In a broad survey conducted with enterprises across multiple geographies (North America, Europe, Asia-Pac), they found 44% are budgeting for Big Data projects in the next 2-5 years, with 33% next year. These are big numbers. The last time I remember anything in enterprise software being a budgeted priority for nearly half of all enterprises, it was the heyday of ERP in the late 90’s, and virtualization in the mid 2000’s. Those vendors generally did pretty well for themselves.
Second, a lot of investment money is flowing towards Big Data technologies. Mainstream vendors and startups alike are investing heavily. Accel Partners, an SF Bay Area based venture capital firm, announced a $100 million fund for Big Data related technology startup investments. A large sum by any measure, it is particularly eye-opening for technologies still early in their maturity curve. Accel partner Ping Li, who also invested early in Cloudera, so knows a thing or two about identifying good investments in this space, says the funds will be disbursed to two types of vendors: (1) those building innovative applications leveraging Big Data and (2) those building infrastructure, security and management solutions for Big Data. In short – they are investing in the two things an early technology needs to become mainstream: Making it easier for the Business to use, and making it easier for IT to manage.
Which brings us to the third: Are we, as a society, really ready to adopt these new technologies? James Markarian, EVP and CTO of Informatica, mentioned during his HadoopWorld keynote that Hadoop may be the greatest technology innovation he has seen in 30 years. 30 years ago saw the introduction of relational databases and personal computers – Both changed the world, in their own ways. Big Data technologies are similarly poised to change the world. Are we ready for this change?
The answer appears to be yes. Much management research in recent years has gone into understanding the “data-driven enterprise”, where decisions become more data-driven and less about intuition and “gut feel” as data volumes grow and (more importantly) the ability to analyze it increases. The Corporate Executive Board recently published research indicating 62% of global corporations still fall short of being able to make effective data-driven decisions, with cultural biases being the main barrier. But those organizations that do become data-driven enjoy a four- to six-percentage-point improvement in productivity, straight to the bottom line. In hyper-competitive markets, that difference can’t be ignored. Not to say that intuitive styles of management won’t have their place – there will always be contexts where hard data is lacking, and ultimately all business success will depend greatly on human judgment and insight – but it is a change for many organizations to suspend disbelief and trust what the data tells them, wherever it does exist.
In short, trends are lining up to produce one of the greatest industry changes we have seen in decades, leading to a sea change in how we organize, make decisions, and get things done. Change is coming. The choice is yours – Are you ready for it?