Data are flooding in at rates never seen before—doubling every 18 months—as a result of greater access to customer data from public, proprietary, and purchased sources, as well as new information gathered from Web communities and newly deployed smart assets. These trends are broadly known as “big data.” Technology for capturing and analyzing information is widely available at ever-lower price points. But many companies are taking data use to new levels, using IT to support rigorous, constant business experimentation that guides decisions and to test new products, business models, and innovations in customer experience. In some cases, the new approaches help companies make decisions in real time. Big data is generating an intense amount of attention among businesses, media and even consumers, along with analytics, cloud-based technologies, digital channels and data visualization.

What is big data?
Big data involves the data produced by different devices and applications, like data from Black Box, Social Media, Stock Exchange, Power Grid, Transport, Search Engine etc. Thus Big Data includes huge volume, high velocity, and extensible variety of data and thus could be of three broad types –
• Structured data – Relational Data
• Semi Structured Data – XML Data
• Unstructured Data – Word, PDF, Text, Media Logs

It also could be typically characterized by the four “V’s”:
• Volume: the amount of data being created is vast compared to traditional data sources
• Variety: data comes from different sources and is being created by machines as well as people
• Velocity: data is being generated extremely fast — a process that never stops, even while we sleep
• Veracity: big data is sourced from many different places, as a result you need to test the veracity/quality of the data

The impact of big data
To understand how data has transformed our daily lives, look at how the movie rental process has been changed. When movies were rented from nearby stores, the store would recommend on which movies the customer said they liked and also by sharing their own opinions.

Today, movie rental companies and content delivery services can utilize an array of data points to generate recommendations. By analyzing what was viewed, when, on what device, as well as user activities such as internet searches, and browsing and scrolling within a webpage, recommendations can be tailored for customers in real time and approximately 75 percent of views at a leading provider are now driven by these recommendations.

Big data life cycle

Creation
There are various types of data that we were generating for long but rarely been using in a meaningful manner:
1. Location of a Person
2. Restaurant they visit, food they consume
3. Credit Card expense pattern etc.
New technology such as advanced sensors and customized software can now record this information for analysis.

Processing
Too much large volumes of data were not traditionally captured and processed for many reasons, mostly because the cost was far larger than the value - companies could derive from its analysis. But due to many of the factors and new technologies the lowered cost is now leveraging the technology barrier for effective data processing, allowing mid size companies also, to be able to unlock the value contained in different data sources.

For instance, it is difficult for conventional relational databases to handle unstructured data, so software frameworks like Hadoop, for distributed storage and parallel processing of large datasets have been introduced to process non-structured data at high speed; to perform a more detailed analysis of big data.

Output

NextPage >>