Change Multiple Single Node Installation To Cluster Installation

This document applies to 3.x version only. And there should be no data in the database

Introduction

1.Install TG single node on all machines. And switch to tigergraph user after installation

2.Copy ssh key to all other machines (if ssh key login is not enabled, skip this step)

ssh-copy-id -i /home/tigergraph/.ssh/tigergraph_rsa <ip>

3.Create a new configuration template for the cluster

gadmin config init --template > /tmp/template

4.Add the nodes with node name (m1, m2, …) and corresponding IPs, SSH configs, and all the roots Note: please use the same roots as your single node installation. You can get them by running

gadmin config get System.DataRoot
gadmin config get System.AppRoot
gadmin config get System.LogRoot
gadmin config get System.TempRoot

Change the template file accordingly

System.HostList: [{"ID":"m1","Hostname":"192.168.55.40"},{"ID":"m2","Hostname":"192.168.1.2"}]
System.SSH.User.Username: tigergraph
System.SSH.User.Password:
System.SSH.User.Privatekey: /home/tigergraph/.ssh/tigergraph_rsa
System.DataRoot: /home/tigergraph/tigergraph/data
System.AppRoot: /home/tigergraph/tigergraph/app/3.0.0
System.LogRoot: /home/tigergraph/tigergraph/log
System.TempRoot: /tmp

5.Create a new TG config from the template

gadmin config init -i /tmp/template --file /tmp/tg.cfg

If you would like to enable HA for the cluster you can use the --ha flag followed by the replication factor. Eg. --ha 2 would enable an HA cluster with a replication factor of 2

6.Login to each node and stop the services, and remove old data folder

gadmin stop all -y
rm -rf $(gadmin config get System.DataRoot --file ~/.tg.cfg)
rm ~/.tg.cfg

7.Apply the config and init the cluster with the config

cp --remove-destination /tmp/tg.cfg ~/.tg.cfg
gadmin init cluster -y --skip-stop

Note: For most cases, it should be all set after step 7 and you can ignore the following instructions

8.In some rare cases where ssh is not usable by TigerGraph. The "gadmin init" command will fail You can do the following steps instead of "gadmin init" to achieve the same effect. Copy the config file to each node manually

scp /tmp/tg.cfg <m1_ip>:~/.tg.cfg
scp /tmp/tg.cfg <m2_ip>:~/.tg.cfg
...
scp /tmp/tg.cfg <mx_ip>:~/.tg.cfg

9.Start executor manually on each node Note: This part is only for version 3.2.0 or higher

gadmin start EXE_1   # on m1
gadmin start EXE_2   # on m2
...

Then go to step 10

Note: This part is only for version 3.0.x or 3.1.x.

gadmin start exe --dry-run

This command will print the executor start commands for each node similar to the following:

EXE_1:
mkdir -p /home/tigergraph/tigergraph/log/executor /home/tigergraph/tigergraph/log/executor;exec env -i TG_TOKEN=XXX PATH=XXX  LC_ALL=en_US.UTF-8 LANG=en_US.UTF-8 /home/tigergraph/tigergraph/app/3.0.5/bin/tg_infr_exed -c /home/tigergraph/.tg.cfg --partition 1
EXE_2:
mkdir -p /home/tigergraph/tigergraph/log/executor /home/tigergraph/tigergraph/log/executor;exec env -i TG_TOKEN=XXX PATH=/XXX LC_ALL=en_US.UTF-8 LANG=en_US.UTF-8 /home/tigergraph/tigergraph/app/3.0.5/bin/tg_infr_exed -c /home/tigergraph/.tg.cfg --partition 2

Save the start command of each executor into a shell file and append the log path to the command:

.... --partition 1 > /home/tigergraph/tigergraph/log/executor/exe.out 2>&1 &

Save the corresponding (be careful since the commands are different for different nodes) executor start script on to each node with executable permission and start them manually by nohup.

scp exe_1.sh <m1_ip>:<path_to_file>/exe_1.sh
ssh <m1_ip>
nohup <path_to_file>/exe_1.sh
scp exe_2.sh <m2_ip>:<path_to_file>/exe_2.sh
ssh <m2_ip>
nohup <path_to_file>/exe_2.sh
...

10.After all the executors are started, do the following commands

gadmin start ctrl
gadmin config apply --initial
gadmin init kafka -y
gadmin start ZK KAFKA IFM ETCD
gadmin start

11.The license must be reapplied once the cluster is created

gadmin license set LICENSE_KEY

Additional Information

Reference for license setting: System Administration FAQs - TigerGraph Server