Profiling TigerGraph Memory Usage

Problem

In certain situations when you believe your TigerGraph system’s memory behavior isn’t normal, you can take these steps to profile its memory usage.

Error message

N/A

Diagnosis

N/A

Solution

Below, we assume your OS is Ubuntu. Also note, TigerGraph restart is needed when enabling or disabling the profiler.

1.Install Google Perf Tool

Method 1: For Ubuntu, use "apt-get install"

sudo apt-get update
sudo apt-get install google-perftools

For CentOS/RedHat, use "yum install"

sudo yum update
sudo yum install google-perftools

Method 2: Build from the Google Perf Tool source code (you need to install gcc first)

curl https://github.com/gperftools/gperftools/releases/download/gperftools-2.5/gperftools-2.5.tar.gz -O -L
tar xzf gperftools-2.5.tar.gz
cd gperftools-2.5

You’ll need to configure your build first. With the following command, we use the default configuration. You can find all the supported options with this command : ./configure — help

 ./configure

Sometimes, you might get an error message complaining "no frame pointer or lib unwind". To work around this, you can run the following command instead :

 ./configure --enable-frame-pointers

Build and install the package using the following command. By default, the package will be installed to /usr/local/bin and /usr/local/lib. You can install the package to any destination, using command-line options to configure. You’ll need to use sudo to do this.

sudo make install

2.Memory Profiling

Change the runtime configuration of the component you want for memory profiling

gadmin --config runtime

Append "HEAPPROFILE=/tmp/profile_folder/profiler_res LD_PRELOAD=/usr/local/lib/libprofiler.so:/home/tigergraph/tigergraph/bin/libtcmalloc.so" to any components you will be profiling.

HEAPPROFILE specifies the filename prefix for the output files (The folder should exist, so please run this command : mkdir -p /tmp/profile_folder.

LD_PRELOAD specifies the path to libprofiler.so and libtcmalloc.so.

Below is an example for GSE memory profiling

 # place your run time options here, per each component
 # E.g.,
 # GPE: LD_PRELOAD=${JEMALLOC_PATH}/lib/libjemalloc.so.1 GLOG_v=100
 # GSE:  LD_PRELOAD=/da1/tigergraph_4.2/bin/libtcmalloc.so AdaptorCache=0 GCOUT_v=10
 # By default, no options
 GPEAG: ""
 GPEEG: ""
 GPE: ""
 GPP: ""
 GSE: "HEAPPROFILE=/tmp/profile_folder/profiler_res LD_PRELOAD=/usr/local/lib/libprofiler.so:/home/tigergraph/tigergraph/bin/libtcmalloc.so"
 RESTPP: ""
 RESTPP-LOADER: "LineBatch=128"
 KAFKA-LOADER: "LineBatch=128"

Once you have added the line to the components that will be profiled, restart the service.

For example, gadmin restart gse -y. You’ll be able to find a .heap file in the /tmp/profile_folder.

3.Visualize Profiling Result

Once the profiling results are outputted, we can convert the profiling files to PDF format, making it easier to analyze.

pprof --pdf /home/tigergraph/tigergraph/bin/ids_worker /tmp/profile_folder/profiler_res*.heap > memory_profile.pdf

The first argument is the path of your binary, in this case it is /home/tigergraph/tigergraph/bin/ids_worker. GSE : ids_worker GPE : poc_gpe_server REST: poc_rest_server

The second argument is the path of your output files : /tmp/profile_folder/profiler_res*.heap.

You can specify one heap file to visualize the heap usage during a specific period, or you can also specify multiple heap files which will be aggregated. An unwanted side effect of this is potentially having repeated data.

If you run this command in a Linux environment, it may let you know that some libraries are missing. You will need to install those (e.g., on Ubuntu):

sudo apt-get install graphviz ghostscript -y

For CentOS/RedHat:

sudo yum install graphviz ghostscript -y

The PDF file will be generated.