Interview with Jim Webber on narratives and graphs

Jim Webber, GOTO Distinguished Speaker, discusses narratives and graphs with Ryan Slobojan. Topics covered include his first talk given at a Trifork conference, conferences attended, the benefits of giving talks at conferences, the importance of narratives and history, the challenge of first discovery, commonalities between REST and graphs, the Web as a graph, Semantic Web, graphs and triples, polyglot persistence, Neo4j, drivers of graph adoption, applications of graphs, and combining genetic algorithms and graphs.

Andreas Kollegger talks about an upcoming workshop on Neo4j using graph databases

In this preview video, Andreas Kollegger tells us about his workshop on Neo4j on 11th July and talk on how, when and where to use graph databases. Come to The Fifth Elephant 2013 — 11th, 12th and 13th July — to attend Andreas’s workshop and talk.

DBTA highlights the leaders in NoSQL

When NoSQL first hit the IT consciousness in 2009, an explosion of NoSQL databases seemed to appear out of thin air. Some of these contenders had in fact been around for some time, with others thrown together rather quickly to exploit the NoSQL buzz.  Over time, the NoSQL pack thinned out as leaders in specific categories emerged.  For document-oriented databases, in my opinion, the clear leader is MongoDB; for big table-style databases, both Cassandra and HBase contend for leadership; while in graph databases, Neo4J remains the undisputed king.

Read the full article.

Creating and using a Neo4j Graph Model for specific use cases

Lately I’ve been busy talking at conferences to tell people about our way to create large Neo4j databases. Large means some tens of millions of nodes and hundreds of millions of relationships and billions of properties.

Although the technical description is already on the Xebia blog part 1 and part 2, I would like to give a more functional view on what we did and why we started doing it in the first place.

Our use case consisted of exploring our data to find interesting patterns. The data we want to explore is about financial transactions between people, so the Neo4j graph model is a good fit for us. Because we don’t know upfront what we are looking for we need to create a Neo4j database with some parts of the data and explore that. When there is nothing interesting to find we go enhance our data to contain new information and possibly new connections and create a new Neo4j database with the extra information.

This means it’s not about a one time load of the current data and keep that up to date by adding some more nodes and edges. It’s really about building a new database from the ground up everytime we think of some new way to look at the data.

First try without hadoop

Before we created our Hadoop based solution, we used the batchimport framework provided with Neo4j (the batch inserter API). This allows you to insert a large amount of nodes and edges without the transactional overhead (Neo4j is ACID compliant). The batch importer API is a very good fit for the medium sized graphs, or the one time imports of large datasets, but in our case recreating multiple databases a day, the running time was too long.

Scaling out

To speed the process we wanted to use our Hadoop cluster. If we could make the process of creating a Neo4j database work in a distributed way, we could make use of the total amount of cluster machines instead of the single machine batchimporter.

But how do you go about that? The batch import framework was built upon the idea of having a single place to store the data. Having a server running somewhere the cluster could connect to had multiple downsides:

• How to handle downtime of the Neo4j server
• You’re back to being transactional
• You need to check if nodes are already existing

So the idea became to build the database really from the ground up. Would it be possible to build the underlying filestructure without having the need of Neo4j running somewhere? Would be cool right?
Read the full article.

Here is the accompanying video:

O’Reilly Strata sits down with Emil Eifrem about Graph databases

Neo4j is the original graph database. [Discussed at 5:25]

UCSC uses Graph Databases to predict Cancer

Stuart’s group will build a separate database, called the Biomedical Evidence Graph (BMEG), for storing and analyzing interpretive information derived from the raw sequence data stored in the CGHub. Like Facebook’s social graph, the BMEG will use a graph database structure designed for lightning-fast access to complex, interconnected datasets.

Read more: http://phys.org/wire-news/132928262/major-grant-funds-ucsc-researchers-using-big-data-to-predict-can.html

Emil Eifrem, CEO of Neo Technology exposes how Graph Search works

Think of a person. Imagine what they are doing and what they look like right now. Who did you pick? Your spouse, your mother, your boss, your crush, your idol? Why out of everyone in your life, did you pick this person? How are you connected?

What you stand to learn from connected data is vastly superior, more interesting and more actionable than what you can learn from disconnected data. This is the key insight and difference between Search as we know it and Graph Search. A deeper treatment of this idea is explained by social researcher James Fowler (author of the book “Connected”).

Looking at a commercial example: What is the top rated restaurant of Chinese cuisine in the city? What is the top rated one amongst my friends (whose opinions I trust)? What is the top rated one according to my Chinese friends? You are connected to this restaurant, just not directly, but by your friends, or maybe friends of friends.

Facebook has built systems to apply this idea to the Social Graph. It won’t be long before we see similar search systems against the other consumer graphs (Intent, Consumption, Interest and Mobile) identified by Gartner.

How Does It Work?

Think about the last time you went to a foreign country and tried your best to ask for directions in a language you weren’t fluent in. “Excusez moi, ou se trouve le pyramide?” You asked how to get to the pyramid. The Parisian hearing you makes assumptions about what you are trying to do even if you have incorrect sentence structure, horrible pronunciation and make little sense. Being in Paris, you probably mean the glass pyramid in front of the Louvre Museum, not the pyramids in Egypt. “Take the Subway at the Opera towards Villejuif, get out at the Palais Royal, walk south for bit and you’ll get there.”

Graph Search makes this kind of interaction possible with a machine. A user types what they are looking for in their own words. Their words are parsed in context and compared against a grammar of relationships the system understands. Once the system understands the question, it finds the answers by combining relationships to find paths from the user to the subject in question. These paths are filtered by permission and ordered by probability. Finally the answers are displayed with some hint of how the system connected the paths.

Facebook built a massive graph traversal search engine called “Unicorn” which stores the relationships of the social graph. These are the “friend” relationships between users, “likes” relationships between users and any entity (people, pages, places, etc.), as well as “lives in”, ”tagged”, ”works at”, and many others. These relationships are combined to traverse the graph and find paths. There are 1 billion people on Facebook, 150 billion friend connections and 2.5 billion new “likes” relationships added every day. The data is so massive Facebook has to use tricks to limit the number of relationships traversed with each query, and still has not opened up Graph Search to all its users.

Can I Have My Own?

While navigating the social graph is interesting, of more interest may be navigating the richer data available within your own company. The task of building your own graph management system is very daunting. Luckily, commercial alternatives that handle this work are available under the category of “graph databases”.

Which existing customers are like a prospective customer? Customers which bought product x are likely to buy which other products? Which customers match fraudulent patterns? What is the best way to route a package to a customer? Who are the experts in a topic? What drugs bind to protein X and do not interact with drug Y? Whose network would be most valuable on my board of directors? These are the kind of questions business users are trying to figure out and graph databases can answer.

There is a game called “Six Degrees of Kevin Bacon” which is played by connecting random actors to Kevin Bacon by the co-stars and movies they appeared in over multiple hops. This game highlights the theory that every person is six or fewer steps away from any other person in the world. In the Facebook graph, the average number of hops was just 4.74 and dropping. McDonalds has 28 million likes on Facebook, Starbucks has 35 million, Coca Cola has 70 million. Marketers understand that the once hidden “word of mouth” network of customers and influencers is becoming visible and more importantly becoming accessible and easier to leverage. The next time you think of a product, remember that it has a network, you may be in it, and if not, someone, somewhere is trying to connect you.

Read more: http://insights.wired.com/profiles/blogs/how-graph-search-works#ixzz2W7CtT942

First Milestone of Spring Data Release Train Babbage includes Spring Data Neo4j 2.3 M1

Most of the changes of this release have made it into Spring Data Commons to build a solid foundation for the next generation of Spring Data projects and make sure that foundation matures fastly. The other modules released in this train station have been adapted to these changes and thus benefit from them as well.

We’ve upgraded to Querydsl 3.x APIs to accomodate the changes introduced in their major release. The repositories abstraction has added support for ordering ignoring case as well as count…By…(…) projection for derived queries. We also gave the mapping metadata implementation a serious performance overhaul so that especially the MongoDB and Neo4j modules should see a ~20% performance increase for mapping operations.

Another big chunk of work went into the overhaul of the pagination and web support, especially in combination with Spring HATEOAS. Creating paginated resource representations for you Spring MVC controllers has never been easier, as you can see in the reference documentation. The changes in Spring Data Commons are rounded off by some improvements in the CDI integration as well as the move of the ChainedTransactionManager from Spring Data Neo4j into the core module.
In Spring Data MongoDB we added support for customizing the field names through a global strategy and ship a CamelCaseAbbreviatingFieldNamingStrategy out of the box. We’ve introduced XML namespace elements for MongoTemplate and GridFsTemplate, added support for the background attributes for indexing and now also support DBRefs in Map values. The Neo4j module brings updates to the latest Neo4j and Cypher releases.

Read the full article.

H-Online reports: Spring Batch plugs into Neo4j and MongoDB

Spring Data support and Java configuration are highlights of the new Spring Batch release from Pivotal’s SpringSource division. Spring Batch is a lightweight framework for developing batch applications and builds upon the Spring framework’s development approach. It is designed to address the need for periodically executed business critical tasks. The new 2.2.0 version of Spring Batch follows in the footsteps of other Spring projects which are integrating NoSQL databases and other “big data” sources using the Spring Data project and moving over to a programmatic configuration model rather than an XML-based one.

Read the full article.

Graph Databases book authored by Ian Robinson, Jim Webber, and Emil Eifrém, covers the Graph based NoSQL database technology and different options available for storing “Connected Data” in real world applications. Authors discuss topics like data modeling with graph based domain models and predictive analysis using Graph Theory techniques.

InfoQ spoke with co-authors Ian Robinson and Jim Webber about the book, role of Graph Databases in the NoSQL database space, and what’s coming up in the Graph Databases.

InfoQ: Congratulations on the new book. What is the current status of the book?

Ian and Jim: Thanks Srini! The book was written over the course of the last year, with the free rough cut version made available about 8 months into the process. We’re now at the final editing phase with O’Reilly and the completed book will be out on June 10th. We’ll be giving away copies to all the attendees at the GraphConnect conferences this year, as well as at many of our Neo4j community events.

InfoQ: How does a Graph database compare with a relational database and with other NoSQL databases?

Ian and Jim: Alistair Jones coined the term “relational crossroads” to describe the differences. According to Alistair, we’re at a fork in the road. One path leads to the approach adopted by most NOSQL databases, whereby data is heavily denormalized, and we then rely on the application to join it together, typically at high latency, to gain insight. The other path leads to the approach adopted by graph databases, whereby we use the expressive power of the graph to build a flexible, connected model of the problem at hand, which we then query at low latency to gain insight.

The relational database sits in the middle. Much like a graph database, the relational database has a query-centric model. But this model isn’t as powerful as that of a graph database. In particular, it lacks the ability to create arbitrarily extensive, semantically rich and variably connected structures at runtime. To create any kind of extensive structure in a relational database, we have to plan our joins up front. To allow for variation, we end up creating lots of nullable columns. The result: sparse tables, fancy (expensive) joins, and object-relational impedance problems, even when used in otherwise straightforward applications.

InfoQ: What are the use cases where Graph database is a better fit?

Ian and Jim: Like relational databases, the majority of graph databases are general purpose OLTP databases, and can be applied in a wide range of solutions. That said, they really shine when the solution to a problem depends upon us first understanding how things are connected. This is more common than you might think. And in many cases, it’s not just a matter of knowing that things are connected; often, we need to know something about the different relationships in our domain – their names, qualities, weights, and so on.

In short, connectedness is key. Graphs are the best abstraction we have for modeling and querying connectedness; graph databases, in turn, give application developers and data specialists the power to apply this abstraction to their own particular problems. To that end, we’ve seen them used for social networking, recommendations engines, authorization and access control, routing and logistics, product catalogues, datacenter  management, careers management, fraud detection, policing, geospatial, and even emigration applications. The key theme that binds all of these solutions together is that they depend on being able to model and query connected data. Simply having keys and values is not enough; nor is having the data sort-of connected through semantically impoverished joins. We need both connectivity and contextual richness to make these solutions work.

InfoQ: Can you discuss the design considerations developers need to keep in mind when using Graph databases?

Ian and Jim: The most important design decisions concern the data model and queries for any particular application. The key here, as we describe in the book, is to drive out your model from the questions you need to ask of your data. By undercovering the questions at the heart of an application goal or end user need, you identify both the things you’re interested in and the ways in which these things are connected. It’s a short step then to turn these questions into an expressive graph model, and thereafter into the queries you execute against that model.

The resulting model and associated queries are really just projections of the questions you want to ask of your data. With Neo4j’s Cypher query language the complementary nature of these projections becomes obvious: the paths you use to create the graph structure are the same paths you use to query it.

As a good first test of your design, you should be able to read what you have written. Most importantly, you should be able to read your questions in what you have written, without having to make any out-of-band assumptions or inferences. A structure such as ‘(Emil)-[:WROTE]->(Graph Databases)<-[:PUBLISHED]-(O’Reilly)’ reads well, and clearly helps us answer the questions, “Which books has Emil written?”, “Who published Graph Databases?” and “For which publishers has Emil written?”.

InfoQ: What are some architecture and design patterns supported by Graph databases?

Ian and Jim: Graph databases aren’t invasive in that respect – they’re just databases, albeit ones that perform orders of magnitude faster for connected data. As such any application patterns that you want to bring to bear will still work. You like MVC? Sure – that’s a good fit. You want to work entirely in Javascript with callbacks? Sure, there’s a great Node.js connector for Neo4j.

InfoQ: Do Graph databases, by their nature, have any limitations in terms of scalability?

Ian and Jim: When we talk about scaling we’re talking about three different things: scaling for large datasets, scaling for read performance, and scaling for write performance.

Regarding scaling for large datasets, there are really no inherent limitations. Neo4j currently has an arbitrary upper limit on the size of the graph (on the order of 10^10, which is large enough to support most graphs we see in the real world, including a Neo4j deployment that has more than half of Facebook’s social graph in one Neo4j cluster), but this limit is going away later this year. This removes the majority of concerns people have with regard to scaling graphs for “big” data.

Scaling for reads similarly presents no problem. Today, in Neo4j, this is accomplished using master-slave replication. Read requests are load balanced across the cluster, where they execute against local data whose structure has been optimized for connected queries. Aside from transactional integrity, Neo4j has historically focused on read performance. To enable fast traversals we have a graph-native stack all the way down to disk. Some other graph stores offer a graph interface over a nonnative graph store, such as a column store. While this may help with other scaling characteristics, it tends to increase query latencies, because the joins are performed in library code, rather than at the level of the data model.

Scaling for writes can be accomplished by scaling vertically, but at some point, for very heavy write loads, it requires the ability to distribute the data across multiple machines. This is the real challenge. While distributing the data may help write performance, it can harm read performance. So far, nobody has implemented a graph database that optimizes and combines fast local traversals with slower (over the network) distributed traversals.

Scaling graph data by distributing it across multiple machines is much more difficult than scaling some of the simpler data models, but it is possible. Scaling a key-value, column-family or document store is relatively easy because in each case you’re dealing with discrete aggregates that can be placed in their entirety in one location or another. Scaling a graph is more difficult because by its very nature the data is connected. When distributing a graph, we want to avoid having relationships that span machines as much as possible; this is called the minimum point-cut problem. On top of the problem of balancing the graph so that there are as few machine-spanning relationships as possible, things are made even more difficult because the graph is always changing. What looks like a good distribution one moment may no longer be optimal a few seconds later. This is known to be an NP-hard problem in the general case.

While we can’t claim to have solved the NP-hard general graph partitioning problem, we’ve found some pretty interesting ways of side-stepping it. In fact we have an R&D team busily experimenting on those ideas right now, and at some point they’ll make it into future product versions. On top of that, we have some tricks up our sleeve to increase the write scaling limits with the current technology.

So to answer the question, we’d say that the connected nature of graph data imposes a theoretical challenge to scaling writes by distributing the graph. But there are practical solutions to the distributed graph problem, which we’re exploring.

Read the full article.

Philip Rathle discusses the five graphs of telecommunications and how companies like Deutsche Telekom, Maaii, Telenor and HP are now using graph databases in their everyday business.

Communication companies are an obvious fit for graphs, as so many of the critical data sets in telecommunications are graphs. From the network graph, to the social graph, to the call center graph, and the master data graph, Telcos worldwide have begun to use graph databases to achieve competitive gain. Neo4j provides thousand-fold performance improvements and massive agility benefits over relational databases, enabling new levels of performance and insight.