Scaling Writes in Neo4j
Gregyoung explains why he thinks Max De Marzi’s way of putting commands into Neo4j is the best way of doing it from his recent blog
Basically he reaches the point of saying “write into rabbitmq, read from Neo4j”. This is actually a really good way of doing things but it can actually be even better if you mix eventsourcing into the problem.
In the blog post Max puts neo4j commands into the queue and then reads them out as fast as possible updating neo4j. There is also some dubious in memory batching that could result in data loss but for many circumstances that is acceptable.
There are two distinct issues that can come up in the model (and some opportunities for other cool stuff!). The first problem is that since you are putting neo commands in, you can only ever have it work with neo (wouldn’t it be even cooler to also be able to use the same exact mechanism to maintain a star schema?). The second problem is that it will require manual configuration (you need to explicitly create the rabbit queues). When there is only one thing listening its no big deal but if you get to twenty it becomes a problem in the post there is only one so this is a minor quip.
This particular use of Event Sourcing is exactly what I hit on in my polyglot data talk. With event sourcing you would not put the neo commands into the queue. Instead put events into the queue that represent the business concept that has happened say CustomerCreated or ContractTerminated. Then you would write a small handler of these events that created the neo4j commands something like (in a typescript type language):