Deploy a cluster

Introduction

This section describes how to set up a new cluster. Two scenarios are covered:

A four-instance cluster with one Single instance as Primary server and three Read Replica instances as Secondary servers. This scenario is ideal for reporting and analytical workloads.
A three-instance cluster with three Core instances as Primary servers. This scenario is ideal for transactional workloads.

Additionally, the process to turn a Secondary server into a standalone instance by detaching it from an existing cluster is also described.

Configure a cluster with Single and Read Replica instances

The following configuration settings are important to consider when deploying a new cluster with a Single instance as a Primary server. See also Settings reference for more detailed descriptions and examples.

This configuration is optimized for best scalability and it is recommended to be used for reporting and analytical workloads. Clusters configured in this way do not provide automatic failover and fault tolerance. In case of fault, if a cluster is not supported by appropriate external tooling, data may be lost.

In the current version of Neo4j, the clustering-related parameters use the causal_clustering namespace. This will be replaced with a more suitable namespace in an upcoming release.

Table 1. Important settings for clusters with Single instance as Primary server
Option name	Servers	Description
`dbms.default_advertised_address`	All (Primary and Secondary)	The address that other machines are told to connect to. In the typical case, this should be set to the fully qualified domain name or the IP address of this server.
`dbms.mode`	Primary	The operating mode of the server instance. The Primary server is set as `SINGLE`.
`dbms.mode`	Secondary	The operating mode of the server instance. The Secondary servers are set as `READ_REPLICA`.
`dbms.clustering.enable=true`	Primary	Allows a single instance to form a cluster and is only evaluated when `dbms.mode=SINGLE`.
`causal_clustering.initial_discovery_members`	Secondary	This setting needs to be specified on Read Replica instances and contains the network address for at least the primary instance, but can also include Secondary servers. This parameter must be set to the same value on all cluster members. The behavior of this setting can be modified by configuring the setting `causal_clustering.discovery_type`. This is described in detail in Discovery.

The following example shows how to set up a cluster with a Single instance as Primary server and three Read Replica instances as Secondary servers.

Example 1. Configure a cluster with a Single instance as Primary server

In this example, one Primary server named single.example.com and three Secondary servers, read_replica01.example.com, read_replica02.example.com and read_replica03.example.com are configured. All instances have Neo4j Enterprise Edition installed. To form a cluster, the neo4j.conf needs to be configured on each server. The Primary server, set as Single instance, is configured as such:

neo4j.conf on single.example.com:

dbms.mode=SINGLE
dbms.clustering.enable=true
dbms.default_advertised_address=single.example.com

The neo4j.conf on the Secondary servers, set as Read Replica instances, is identical across all instances:

neo4j.conf on read_replica01.example.com, read_replica02.example.com and read_replica03.example.com:

dbms.mode=READ_REPLICA
dbms.default_advertised_address=read_replica<xx>.example.com
causal_clustering.initial_discovery_members=single.example.com:5000

Once all neo4j.conf files have been configured, the instances can be started and the cluster is ready. After the cluster has started, it is possible to connect to any of the instances and run CALL dbms.cluster.overview() to check the status of the cluster. This shows information about each member of the cluster:

CALL dbms.cluster.overview();

+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| id                                     | addresses                                                                          | databases                                       | groups |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| "8e4133d7-4de1-469e-88ac-864571cb0a92" | ["bolt://read_replica1.example.com:7687", "http://read_replica1.example.com:7474"] | {neo4j: "READ_REPLICA", system: "READ_REPLICA"} | []     |
| "eb6a4e88-9a5f-405b-b230-5bbbd681ec9e" | ["bolt://read_replica2.example.com:7687", "http://read_replica2.example.com:7474"] | {neo4j: "READ_REPLICA", system: "READ_REPLICA"} | []     |
| "274e36db-d96f-4736-8a99-68851b1bbb0b" | ["bolt://read_replica3.example.com:7687", "http://read_replica3.example.com:7474"] | {neo4j: "READ_REPLICA", system: "READ_REPLICA"} | []     |
| "6fd05bc6-760e-4644-bf02-05117a5d777d" | ["bolt://single.example.com:7687", "http://single.example.com:7474"]               | {neo4j: "LEADER", system: "LEADER"}             | []     |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
4 rows available after 8 ms, consumed after another 3 ms

Configure a cluster with Core instances

The following configuration settings are important to consider when deploying a new cluster with Core instances as Primary servers. See also Settings reference for more detailed descriptions and examples.

This configuration is optimized for fault tolerance, automatic failover and best scalability, and it is recommended to be used for transactional workloads. In many cases and when they are correctly configured, these clusters safeguard data and they do not require any particular external tooling.

Table 2. Important settings for clusters with Core instances as Primary servers
Option name	Servers	Description
`dbms.default_listen_address`	All (Primary and Secondary)	The address or network interface this machine uses to listen for incoming messages. Setting this value to `0.0.0.0` makes Neo4j bind to all available network interfaces.
`dbms.default_advertised_address`	All (Primary and Secondary)	The address that other machines are told to connect to. In the typical case, this should be set to the fully qualified domain name or the IP address of this server.
`dbms.mode`	Primary	The operating mode of the server instance. The Primary servers are set as `CORE`.
`dbms.mode`	Secondary	The operating mode of the server instance. The Secondary servers are set as `READ_REPLICA`.
`causal_clustering.minimum_core_cluster_size_at_formation`	Primary	The minimum number of Core instances in the cluster at formation. A cluster will not form without the number of Cores defined by this setting, and this should in general be configured to the full and fixed amount.
`causal_clustering.minimum_core_cluster_size_at_runtime`	Primary	The minimum number of Core instances which will exist in the consensus group.
`causal_clustering.initial_discovery_members`	All (Primary and Secondary)	The network addresses of an initial set of Core cluster members that are available to bootstrap this Core or Read Replica instance. In the default case, the initial discovery members are given as a comma-separated list of address/port pairs, and the default port for the discovery service is `:5000`. It is good practice to set this parameter to the same value on all Core Servers. The behavior of this setting can be modified by configuring the setting `causal_clustering.discovery_type`. This is described in detail in Discovery.

Listen configuration

Listening on 0.0.0.0 makes the ports publicly available. Make sure you understand the security implications and strongly consider setting up encryption.

The following example shows how to set up a simple cluster with three Core servers:

Example 2. Configure a Core-only cluster

In this example, three Core instances named core01.example.com, core02.example.com and core03.example.com are configured. Neo4j Enterprise Edition is installed on all three servers. They are configured by preparing neo4j.conf on each server. Note that they are all identical, except for the configuration of dbms.default_advertised_address:

neo4j.conf on core01.example.com:

dbms.default_listen_address=0.0.0.0
dbms.default_advertised_address=core01.example.com
dbms.mode=CORE
causal_clustering.initial_discovery_members=core01.example.com:5000,core02.example.com:5000,core03.example.com:5000

neo4j.conf on core02.example.com:

dbms.default_listen_address=0.0.0.0
dbms.default_advertised_address=core02.example.com
dbms.mode=CORE
causal_clustering.initial_discovery_members=core01.example.com:5000,core02.example.com:5000,core03.example.com:5000

neo4j.conf on core03.example.com:

dbms.default_listen_address=0.0.0.0
dbms.default_advertised_address=core03.example.com
dbms.mode=CORE
causal_clustering.initial_discovery_members=core01.example.com:5000,core02.example.com:5000,core03.example.com:5000

The Neo4j servers are ready to be started. The startup order does not matter.

After the cluster has started, it is possible to connect to any of the instances and run CALL dbms.cluster.overview() to check the status of the cluster. This shows information about each member of the cluster:

CALL dbms.cluster.overview();

+----------------------------------------------------------------------------------------------------------------------------------------+
| id                                     | addresses                                  | databases                               | groups |
+----------------------------------------------------------------------------------------------------------------------------------------+
| "8e07406b-90b3-4311-a63f-85c45af63583" | ["bolt://core1:7687", "http://core1:7474"] | {neo4j: "LEADER", system: "FOLLOWER"}   | []     |
| "aeb6debe-d3ea-4644-bd68-304236f3813b" | ["bolt://core3:7687", "http://core3:7474"] | {neo4j: "FOLLOWER", system: "FOLLOWER"} | []     |
| "b99ff25e-dc64-4c9c-8a50-ebc1aa0053cf" | ["bolt://core2:7687", "http://core2:7474"] | {neo4j: "FOLLOWER", system: "LEADER"}   | []     |
+----------------------------------------------------------------------------------------------------------------------------------------+

Startup time

The instance may appear unavailable while it is joining the cluster. If you want to follow along with the startup, you can follow the messages in neo4j.log.

Add a Core Server to an existing cluster

Core Servers are added to an existing cluster by starting a new Neo4j instance with the appropriate configuration. The new server will join the existing cluster and become available once it has copied the data from its peers. It may take some time for the new instance to perform the copy if the existing cluster contains large amounts of data.

The setting causal_clustering.initial_discovery_members shall be updated on all the servers in the cluster to include the new server.

Example 3. Add a Core Server to an existing cluster

In this example, a Core Server, core04.example.com, is added to the cluster created in Configure a Core-only cluster.

Configure the following entries in neo4j.conf:

neo4j.conf on core04.example.com:

dbms.default_listen_address=0.0.0.0
dbms.default_advertised_address=core04.example.com
dbms.mode=CORE
causal_clustering.minimum_core_cluster_size_at_formation=3
causal_clustering.minimum_core_cluster_size_at_runtime=3
causal_clustering.initial_discovery_members=core01.example.com:5000,core02.example.com:5000,core03.example.com:5000,core04.example.com:5000

Note that the configuration is very similar to that of the previous servers. In this example, the new server is not intended to be a permanent member of the cluster, thus it is not included in causal_clustering.initial_discovery_members on the other Core members of the cluster.

Now start the new Core Server and let it add itself to the existing cluster.

Add a Secondary server to an existing cluster

In the 4.4 version of Neo4j, all Secondary servers are Read Replica instances. The initial configuration for Read Replica instances is provided via neo4j.conf, as mentioned above in Configure a cluster with Single and Read Replica instances. Since Read Replicas do not participate in cluster quorum decisions, their configuration is shorter; they only need to know the addresses of at least one primary instance which they can bind to in order to discover the cluster.

It is recommended to specify the addresses for all existing primary instances in a cluster when adding a Read Replica. They can then select an appropriate Primary server from which to copy data.

Example 4. Add a Secondary server to an existing cluster with a Single instance as Primary server

In this example, a Read Replica instance, replica04.example.com, is added to the cluster created in Configure a cluster with a Single instance as Primary server.

Configure the following entries in neo4j.conf:

neo4j.conf on replica01.example.com:

dbms.default_advertised_address=read_replica04.example.com
dbms.mode=READ_REPLICA
causal_clustering.initial_discovery_members=single.example.com:5000

Now start the new Read Replica and let it add itself to the existing cluster.

Example 5. Add a Secondary server to an existing cluster with Core servers as Primary servers

In this example, a Read Replica, replica05.example.com, is added to the cluster created in Configure a Core-only cluster.

Configure the following entries in neo4j.conf:

neo4j.conf on replica05.example.com:

dbms.default_advertised_address=read_replica05.example.com
dbms.mode=READ_REPLICA
causal_clustering.initial_discovery_members=core01.example.com:5000,core02.example.com:5000,core03.example.com:5000

Now start the new Read Replica and let it add itself to the existing cluster.

When adding a Secondary server to an existing cluster, only Primary servers need to be listed in causal_clustering.initial_discovery_members. It is not necessary to include existing Secondary servers, i.e. other Read Replica instances.

Detach a Secondary server from an existing cluster

It is possible to turn a Secondary server into a standalone instance that thus contains a snapshot of the data in the cluster. This can, in theory, be done for a Core Server as well, but this is not recommended for performance and safety reasons. As mentioned above, in the 4.4 version of Neo4j, all Secondary servers are Read Replica instances.

Example 6. Detach a Read Replica and turn it into a stand alone instance

In this example, a Read Replica, replica01.example.com, is detached from a cluster. See Add a Secondary server to an existing cluster above on how to add a Read Replica to a cluster.

First, check if the Read Replica is as up-to-date as desired. Use SHOW DATABASE to see where the different members of the cluster are in terms of committed transactions compared to the leader.

neo4j@system> SHOW DATABASE test00 YIELD name,serverID,address,role,lastCommittedTxn,replicationLag;

Note that SHOW DATABASES uses serverID as it lists databases and there may be more than one database per server, while dbms.cluster.overview() uses only id as it is only concerned with servers.

+---------------------------------------------------------------------------------------------------------------------------+
| name     | serverID                               | address          | role           | lastCommittedTxn | replicationLag |
+---------------------------------------------------------------------------------------------------------------------------+
| "test00" | "aeb6debe-d3ea-4644-bd68-304236f3813b" | "core3:7687"     | "leader"       | 21423            | 0              |
| "test00" | "8e07406b-90b3-4311-a63f-85c45af63583" | "core1:7687"     | "follower"     | 21422            | -1             |
| "test00" | "b99ff25e-dc64-4c9c-8a50-ebc1aa0053cf" | "core2:7687"     | "follower"     | 21423            | 0              |
| "test00" | "0bf3f6c1-0f48-47c2-a943-18fa8362c918" | "replica4:7687"  | "read_replica" | 21409            | -14            |
| "test00" | "0e9c1b28-c8c0-4c65-a1f2-39d326411280" | "replica6:7687"  | "read_replica" | 21421            | -2             |
| "test00" | "82524236-3058-48a2-b198-6580003475af" | "replica5:7687"  | "read_replica" | 21413            | -10            |
+---------------------------------------------------------------------------------------------------------------------------+

Based on the results, decide which Read Replica to detach and proceed to shut it down.

Once the Read Replica is shut down, configure the following entry in neo4j.conf:

neo4j.conf on replica01.example.com:

dbms.mode=SINGLE

Start the instance again. It is now a standalone instance containing the data committed to it at the time of shutdown.

There is always a chance that the Read Replica is behind the Core Servers at any time (see above on how to check the state of your cluster members). If a transaction is being processed at the time of the shutdown of the Read Replica, this transaction is eventually reflected in the remaining Cluster, but not on the detached Read Replica. A way to ensure that a Read Replica contains a snapshot of a database in the cluster at a point in time, is to pause the read Replica before shutting it down. See dbms.cluster.readReplicaToggle() for more information.