Mary Liu talks about Big Data and how Sci-Fi movies would use complex databases that would require a complex method of graphing them, as well as how Neo Technology’s graph database fits into all of these


Big Data with noSQL = Prophecy = Profits

Growing trend of SciFi movies is not just symptomatic of advanced green screen and animation technologies. Thanks to advancements in the big data boom, cinematic fantasies are actually realities in the making as the 2013 big data conference, NoSQL Now!, unveiled.

Here is a glimpse: Imagine the “precogs” from Minority Report or the Oracle from The Matrix actually anticipating future crimes or user actions, down to the second. Picture a network database as complex and comprehensive as the Matrix or the Tree of Souls from Avatar, linking all living beings together and automatically deploying labor resources (or animal troops from Avatar) in critical moments of insufficient capacity.

The prequel to the “precogs” and the Oracle is already here. Target can detect, with accuracy, when a customer is pregnant before she or her doctor does based on her purchase behavior. Women who are pregnant in the first trimester –as early as the initial week – have hyper sense of smell that triggers not only nausea, but also changes in brand loyalty for scented skincare products. Target is not the only prophet. “The best way to tell who can make their flights is whether they preordered a vegetarian meal,” says the founder of Kaggle, Anthony Goldbloom, “the psychology of making that trip personal, by knowing your meal is on it, makes you more likely to get on the plane on time.” Currently, businesses are not racing to implement predictive analytics, but rather to do it better. Leading market shares and profits goes to those who can do it best.

The complex data analytic toolset spans from programs written in statistical data-mining languages such as R, to virtualized distributed computing in a cloud space similar to the proverbial matrix of The Matrix. The databases have gathered intel from your social posts and reviews, purchase patterns, and even surveillance video feeds to forecast your next move. Our unconscious habits, rather than consciously made decisions, influence 45% of our choices according to research from Duke University. Hollywood calls this process artificial intelligence; CIA calls it terrorist intel; and businesses call it business intelligence.

Behind the scenes of business intelligence: Challenges of SQL and non-SQL

Adoptions have heighted from mobile devices, social networking and media, networked devices, and even sensors and surveillance videos. Data stores are billowing up by 50% year over year from terabytes to zettabytes (1B terabytes). By the end of this year, a total of 2.5 ZB will be stored globally, not including the NSA’s new 5 ZB Utah Data Center.

Traditional “Structured Query Language” (SQL) databases, which organizes information into neat relational tables (think Excel), just cannot scale for complex, less structured datasets. Over 80% of data today is unstructured files, such as SlideShare documents, that do not work well on typical relational databases. For example, LinkedIn’s graph networks that capture complex first, second, and third degree connectivity is best stored in databases like Espresso DB or Neo Technology. Twitter stream or Uber’s taxi hailing geo data are best stored in column-family databases like Cassandra or HBase. These databases are non-SQL and complement traditional SQL stores.

Read the Full Article Here.