More than three-quarters of companies are investing or planning to invest in big data in the next two years, according to a recent survey of IT and business leaders by Gartner, Inc. A Capgemini survey across multiple companies revealed that only 13 percent respondents believed that their Big Data implementations are really in full-scale production and predictive insights are extensively integrated into business operations. While 35 percent respondents believed that their Big Data implementations are in “partial” production, rest 53 percent respondents confessed that they are either at POC stage (doing or thinking) or awaiting budgets for doing a POC!.

So, where’s the gap?

In my experience while doing multiple Big Data implementations and building componentized platforms for doing so, here are the key challenges which are impeding accelerated adoption of Big Data and suggested solutions for overcoming them.

Skill Gap

While Data warehousing and ETL skills as well as newer NoSQL skills like MongoDB, Cassandra etc. and infrastructural skills like Amazon AWS, Apache Hadoop etc are reasonably available, one of the high ticket promises of Big Data – Predictive Modeling / Machine Learning which require strong “Data Science” skills, are scarce. Data Science is ideally the intersection of following three pillars: Strong mathematical & statistical background, Hacking (aka programming) skills, and (Deep) Domain Expertise and experience.

Integrating multi-channel and variety of data sources at the modern volume

Another challenge is handling volume, velocity & variety: from structured data sources from RDBMS/Data Warehouses to unstructured data like Social media to clickstream & sensors. There are concerns like ETL, Homogenization, clean-up, enrichment, and semantic associations.

Hard ROI?

A budget allocation for “Big Data Projects” in enterprises is one of the biggest challenges today where teams struggle in terms of justifying hard ROI on investing in Big Data projects. While this shall ease out as the field goes up the standard maturity continuum, a few ideas which can work well are - starting small (and not be too ambitious) to showcase quick results and using innovative “pay for results” business models that play in well with the psyche of people on the edge of decision making around investment in Big
Data projects.

Getting the right data and Infra architecture for performance and scalability

This is an obvious technical challenge which although should be easily addressable, but gets tremendously complex due to many variables. Recent investments in legacy infrastructure severely restricts coming up with an ideal “future-ready” architecture. The endeavor is to come up with a most “optimal” architecture which allows for as much re-use of legacy infrastructure as possible.

The Big Data tech stack has been evolving way too fast with technologies getting up on hype curve and suddenly losing favor due to a newer alternative (ex. STORM v/s SPARK).Until that happens, a detailed “assessment” of what is trending in market and more importantly of the internal stack and future needs, the technology architecture roadmap should be very thoughtfully crafted.

Turn-around time from Data acquisition to insights

One of the most common problems we’ve encountered is the high overall turn-around time which it takes from data acquisition, clean-up, modeling and deploying models at scale in production, in many cases high enough for data to not stay very relevant. A typical flow would look like:

• Data ingestion (from multiple sources)
• Data clean-up and transformations/ enrichment
• Iterative model development (Data Science)
• Deploying models in production

NextPage >>