Backing up to GCP

Introduction

GBAR is TigerGraph’s backup utility. While it doesn’t support backups to GCP natively, you can set up automated backups by running a few simple scripts.

 Note: You must have a service account to run the backups with a storage admin role or set permissions to backup and download from the bucket
 https://cloud.google.com/docs/authentication/getting-started#windows

Instructions

Download gcloud on ALL NODES

#set key for gcloud on ALL NODES

gcloud auth activate-service-account  --key-file=/path/to/key

Setting up Backups

Step 1: Configure GBAR

Run the following script on any node. replace <Backup> with your desired directory

gadmin config set System.Backup.Local.Enable true
gadmin config set System.Backup.Local.Path ~/<Backup>
gadmin config apply -y

Step 2: Running your backup

Single node backups

#Take backup
GSQL_USERNAME=tigergraph GSQL_PASSWORD=tigergraph gbar backup -t daily
#Tar dir - prevents checksum issues
name=$(ghostname | grep -o m[0-9])-$(date +"%Y-%m-%d")
todays_backup=$(gbar list | grep "$(date +"%Y-%m-%d")" | awk '{print $1}')
tar -cvf  $name.tar $(gadmin config get System.Backup.Local.Path)/$todays_backup
#Hash and upload backup to ensure backup uploads properly
hash=$(gsutil  hash $name.tar 2>&1  | grep md5 | awk '{ printf $3 }')
gsutil -h Content-MD5:$hash cp  -r $name.tar gs://jordans-bucket-tigergraph
#remove files from local host
rm -f $name.tar && rm -rf $todays_backup

Multi-node backups

GCS_BUCKET=gs://SOME_BUCKET
#set GSQL tg user creds
GSQL_USERNAME=tigergraph
GSQL_PASSWORD=tigergraph
#set backup path
BACKUP_PATH=$(gadmin config get System.Backup.Local.Path)
echo "------set BACKUP_PATH to $BACKUP_PATH------"
#Take backup
echo "------running backup------------"
GSQL_USERNAME=$GSQL_USERNAME GSQL_PASSWORD=$GSQL_PASSWORD gbar backup -t daily
#extract backup name
echo "------backup finished, retrieiving backup name------"
DAILY_BACKUP_NAME=$(gbar list | grep "$(date +"%Y-%m-%d")" | awk '{print $1}')
echo "------retrieved backup name as $DAILY_BACKUP_NAME------"
#Tar files
echo "------tar files on all nodes------"
grun_p all "PATH=$PATH; cd $(gadmin config get System.Backup.Local.Path); tar -zcvf \$(ghostname | grep -o -E 'm[0-9]+')-$(date +"%Y-%m-%d")\.tar $DAILY_BACKUP_NAME"
echo "------file tar finished. Displaying name on all nodes...------"
grun_p all "PATH=$PATH; cd $(gadmin config get System.Backup.Local.Path); ls \$(ghostname | grep -o -E 'm[0-9]+')-$(date +"%Y-%m-%d")\.tar"
echo "------begin upload to GCS------"
#upload all backup files to GCP
grun_p all "PATH=$PATH; cd $BACKUP_PATH; echo \"copy \$(ghostname | grep -o -E 'm[0-9]+')-$(date +"%Y-%m-%d")\.tar to $GCS_BUCKET\"; gsutil cp \$(ghostname | grep -o -E 'm[0-9]+')-$(date +  "%Y-%m-%d")\.tar $GCS_BUCKET"
echo "------finish upload to GCS------"
#remove files from local host
echo "------removing backup files from all nodes------"
grun_p all "cd $BACKUP_PATH; echo \"removing $DAILY_BACKUP_NAME\"; rm -rf $DAILY_BACKUP_NAME"
grun_p all "PATH=$PATH; cd $BACKUP_PATH; echo \"removing \$(ghostname | grep -o -E m[0-9]+)-$(date +"%Y-%m-%d")\.tar\"; rm -rf \$(ghostname | grep -o -E m[0-9]+)-$(date +"%Y-%m-%d")\.tar"
echo "------finished removing files from all nodes------"
echo "------done------"

Step 3: Restore your backup

GCS_BUCKET=gs://SOME_BUCKET
RESTORE_DATE=2022-07-07
#Copy files from GCP to local backup path
echo ""------copy files from $GCS_BUCKET to each node"------"
grun_p all "PATH=$PATH; echo \"copy \$(ghostname | grep -o -E m[0-9]+)\-$RESTORE_DATE.tar to $(gadmin config get System.Backup.Local.Path)\"; gsutil cp -r $GCS_BUCKET/\$(ghostname | grep -o -E 'm[0-9]+')\-$RESTORE_DATE.tar $(gadmin config get System.Backup.Local.Path)"
#Unpack tar files
grun_p all "PATH=$PATH; cd $(gadmin config get System.Backup.Local.Path); tar -xvf \$(ghostname | grep -o -E m[0-9]+)\-$RESTORE_DATE.tar"
#Restore with gbar
GSQL_USERNAME=tigergraph GSQL_PASSWORD=tigergraph gbar restore $(gbar list | grep $(date +"%Y-%m-%d") | awk '{print $1}') -y