Clone TigerGraph 3.x to Another Cluster
Introduction
This article describes how to clone a TigerGraph database from one cluster to another cluster having the same partition configuration, while having same or different replicas setting. For illustration, we call the two clusters SourceCluster and TargetCluster, respectively. Nodes in two clusters, called src_m1, src_m2,…, tgt_m1,…, etc.
TigerGraph provides its backup and restore utility (GBAR), which allows users to backup the database and restore it to the same cluster. Each archive, indexed by a unique backup tag, has the complete database snapshot as well as the cluster layout. When restoring, the users only need to specify the backup tag and GBAR will automatically take care of the whole procedure.
However, when users want to clone the database to another cluster using GBAR, some extra steps are needed. These steps will ensure the same system environments and the correct cluster layout in the archives, so GBAR will restore the archive as if it were doing the same system restore.
Instructions
1.Setup TargetCluster with same partition as SourceCluster
TigerGraph only allows users to clone the database to another system having the same cluster layout. Specifically, 'SourceCluster and TargetCluster need to have the same number of partitions', which can be calculated by dividing the total cluster nodes with the replica number, while the replica number can be different. For example, the partition number will be 3 if SourceCluster is a 6-node cluster w/ HA enabled on default replica number 2, then TargetCluster has to be a 3-node cluster w/o HA, or 6-node w/ HA, or 9-node w/ HA on replica number 3, and so on. If users want TargetCluster larger, they should setup TargetCluster the same as SourceCluster.
In your SourceCluster, run
gsshIt will print out cluster information
Usage: gssh m1|gpe_1#1|gse1_1#1|...Usage: ----------------Available hosts--------------Host * IdentityFile /home/tigergraph/.ssh/tigergraph_rsa Port 22Host m1 ADMIN#1 admin#1 CTRL#1 ctrl#1 DICT#1 dict#1 ETCD#1 etcd#1 EXE_1 exe_1 GPE_1#1 gpe_1#1 GSE_1#1 gse_1#1 GSQL#1 gsql#1 GUI#1 gui#1 IFM#1 ifm#1 KAFKA#1 kafka#1 KAFKACONN#1 kafkaconn#1 KAFKASTRM-LL_1 kafkastrm-ll_1 NGINX#1 nginx#1 RESTPP#1 restpp#1 TS3_1 ts3_1 TS3SERV#1 ts3serv#1 ZK#1 zk#1 HostName ip.ip.ip.ipHost m2 ADMIN#2 admin#2 CTRL#2 ctrl#2 EXE_2 exe_2 GPE_2#1 gpe_2#1 GSE_2#1 gse_2#1 IFM#2 ifm#2 KAFKA#2 kafka#2 KAFKACONN#2 kafkaconn#2 KAFKASTRM-LL_2 kafkastrm-ll_2 NGINX#2 nginx#2 RESTPP#2 restpp#2 TS3_2 ts3_2 HostName ip.ip.ip.ip#cluster.nodes: m1:172.31.85.22,m2:172.31.80.184#admin.servers: m1,m2#ctrl.servers: m1,m2#dict.servers: m1#etcd.servers: m1#exe.servers: m1,m2#gpe.servers: m1,m2#gse.servers: m1,m2#gsql.servers: m1#gui.servers: m1#ifm.servers: m1,m2#kafka.servers: m1,m2#kafkaconn.servers: m1,m2#kafkastrm-ll.servers: m1,m2#nginx.servers: m1,m2#restpp.servers: m1,m2#ts3.servers: m1,m2#ts3serv.servers: m1#zk.servers: m1#log.root: /home/tigergraph/tigergraph/log#app.root: /home/tigergraph/tigergraph/app/3.0.5#data.root: /home/tigergraph/tigergraph/dataSetup your TargetCluster the same way and run gssh afterwards to verify. The example above is a cluster with 2 machines.
2.Config GBAR at SourceCluster and TargetCluster
In SourceCluster’s m1 Run
$ gadmin config entry backup...# change it to trueSystem.Backup.Local.Enable [ false ]: Backup data to local pathNew: true# change it to the path of choiceSystem.Backup.Local.Path [ ]: The path to store the backup filesNew: ~/backup...Leave other configurations unchanged.
Then run
gadmin config applyThen do the same thing in TargetCluster’s m1 if it has not been configured yet.
Note: This step can be done in any machine other than just m13.Generate Archive
In SourceCluster, runRun
gbar backup -t your_tag_of_choiceIt will ask you to type username and password of GSQL. The default of both are tigergraph if never changed.
A directory of name <your_tag_of_choice> with will be written to the path you just entered in the gadmin configuration
$ pwd/home/tigergraph$ ls your_tag_of_choice tigergraphThis backup data directory will be written to all machines of SourceCluster while you only need to run gbar in one machine. Each machine will only have its own data backed up in this directory.
4.Copy the Archive from SourceCluster to TargetCluster
You need to copy archives from all machines in SourceCluster to TargetCluster.
You can either copy source m1 to target m1, source m2 to target m2 etc or copy all source archive to the same target machine if one of your target machine has enough storage.
Our recommendation is to copy each source node to its corresponding target node.
In this example, I setup a 4 nodes 2 replica cluster and copied
1.source m1 → target m1
2.source m2 → target m2
3.source m3 → target m3
4.source m4 → target m4
We have a script to do most of the work. You only need to modify 7 arguments at the top of the script.
###################################################
# Please Replace Variables Below with Your Values #
###################################################
source_public_ips=('IP' 'IP' 'IP' 'IP')
source_private_ips=('IP' 'IP' 'IP' 'IP')
target_public_ips=('IP' 'IP' 'IP' 'IP')
target_private_ips=('IP' 'IP' 'IP' 'IP')source_pem=tg1.pem target_pem=tg2.pem
tag=tt-20201130065703
# # Start # # for i in "${!source_public_ips[@]}" do echo "m$$i+1 source:${source_public_ips[$i]} target:${target_private_ips[$i]}" done echo
o1="UserKnownHostsFile=/dev/null" o2="StrictHostKeyChecking=no"
echo "First, copy ssh key of target cluster to each node of source cluster" echo for i in "${!source_public_ips[@]}" do echo "copy $target_pem to — source m$$i+1 ${source_public_ips[$i]}" scp -i $source_pem $target_pem -o $o1 -o $o2 tigergraph@${source_public_ips[$i]}:~/ echo done
echo "copy each source node’s backup data to corresponding target node" for i in "${!source_public_ips[@]}" do echo "scp -o $o1 -o $o2 -i $target_pem -r backup tigergraph@${target_private_ips[$i]}:~/" ssh -i $source_pem tigergraph@${source_public_ips[$i]} "scp -o $o1 -o $o2 -p -i $target_pem -r backup tigergraph@${target_private_ips[$i]}:~/" done echo
echo "change IPs in layout files of all target nodes" for i in "${!target_public_ips[@]}" do echo "m$$i+1" ssh -i $target_pem -o $o1 -o $o2 tigergraph@${target_public_ips[$i]} " src_ips=\"$(echo ${source_private_ips[]})\"; src_ips=(\$src_ips); tgt_ips=\"$(echo ${target_private_ips[]})\"; tgt_ips=(\$tgt_ips); for j in \${!src_ips[@]}; do sed -i \"s/\${src_ips[\$j]}/\${tgt_ips[\$j]}/\" backup/$tag/$tag-archive.layout done;" done
First you need to fill in the IP addresses of all machines of source cluster and target cluster. Since your working machine may be in a different network from the cluster is, we distinguish public IPs from private IPs. If your working machine is in the same network of clusters, fill the same value.
Our example has 4 nodes, but your cluster may have different length of IPs.
source_public_ips=('IP' 'IP' 'IP' 'IP')source_private_ips=('IP' 'IP' 'IP' 'IP')target_public_ips=('IP' 'IP' 'IP' 'IP')target_private_ips=('IP' 'IP' 'IP' 'IP')Then in your working machine, you need to have the SSL private key (The default path is <tigergraph_user_home>/.ssh/tigergraph_rsa) of the tigergraph account of both source and target cluster. Fill in their paths.
source_pem=source.pemtarget_pem=target.pemtag is the backup tag. Run gbar list in any source node to get.
tag=tt-20201130065703After fill in all variables, run
bash clone.shIf you saw permission too open error, run
chmod 600 your_ssh_keyThis script will copy backup files from source to target machines and get care of other necessary modifications.