GPE service won’t start, max segment ID reached

Problem

Gssh error : getcwd: cannot access parent directories: No such file or directory

Error message

Log file created at: 2022/06/09 14:44:44
Running on machine: tigger-graphdb-1
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E0609 14:44:44.629555 3428619 gsystem.cpp:799] GSystem|KafkaWatcher|Disk status changed: 0 -> 1
F0609 14:44:45.766184 3428698 topologyvertexstatus.hpp:61] Check failed: segmentid < max_segments_ 131073|131072
E0609 14:44:45.786608 3428698 glogging.cpp:132] ============ Crashed with stacktrace ============
  0# FailureSignalHandler at /home/graphsql/product/src/engine/utility/gutil/glogging.cpp:132
 1# 0x00007F51B24683C0 in /var/tigergraph/app/3.5.2/.syspre/usr/lib_ld3/libpthread.so.0
 2# 0x00007F51B1D5D03B in /var/tigergraph/app/3.5.2/.syspre/usr/lib_ld3/libc.so.6
 3# 0x00007F51B1D3C859 in /var/tigergraph/app/3.5.2/.syspre/usr/lib_ld3/libc.so.6
 4# 0x00007F51B55C57B8 in /var/tigergraph/app/3.5.2/bin/libtigergraph.so
 5# 0x00007F51B55BEA1E in /var/tigergraph/app/3.5.2/bin/libtigergraph.so
 6# 0x00007F51B55BE96A in /var/tigergraph/app/3.5.2/bin/libtigergraph.so
 7# 0x00007F51B55BE350 in /var/tigergraph/app/3.5.2/bin/libtigergraph.so
 8# 0x00007F51B55C16C9 in /var/tigergraph/app/3.5.2/bin/libtigergraph.so
 9# topology4::DeltaRebuilder::CheckSegment(topology4::QueryState&, unsigned long, unsigned long&, unsigned long&, topology4::SegmentMeta*, unsigned long, bool, bool&) at /home/graphsql/product/src/engine/core/topology/topology4/deltarebuilder.cpp:809
10# topology4::DeltaRebuilder::TryUnScheduledRebuild(unsigned long) at /home/graphsql/product/src/engine/core/topology/topology4/deltarebuilder.cpp:1821
11# topology4::DeltaRebuilder::RebuildManagementThread() at /home/graphsql/product/src/engine/core/topology/topology4/deltarebuilder.cpp:2257
12# 0x00007F51B55ACD59 in /var/tigergraph/app/3.5.2/bin/libtigergraph.so
13# 0x00007F51B245C609 in /var/tigergraph/app/3.5.2/.syspre/usr/lib_ld3/libpthread.so.0
14# 0x00007F51B1E39163 in /var/tigergraph/app/3.5.2/.syspre/usr/lib_ld3/libc.so.6
============ End of stacktrace ============

Diagnosis

Found that user environment reached the maximum number of segments allowed in TigerGraph

Workaround

Prior to 3.7.0, a patch from TigerGraph Engineering is required to resolve this issue. Starting in version 3.7.0, the maximum segments is a configurable value via gadmin config. If you find yourself needing to increase this value, it’s recommended to reach out to TigerGraph support.

Solution

Download patch to m1

curl -o libtigergraph.so -X GET "https://storage.googleapis.com/storage/v1/b/jiyatest/o/libtigergraph_patch.so?alt=media"

Stop GPE, GSE, RESTPP

gadmin stop gpe gse restpp

Make backup of current libtigergraph.so on all nodes

grun all "mv /var/tigergraph/app/3.5.2/bin/libtigergraph.so /var/tigergraph/app/3.5.2/bin/libtigergraph.so.bak"

Copy new libtigergraph.so (i.e. patch) from m1 to all nodes

gscp all libtigergraph.so /var/tigergraph/app/3.5.2/bin/

Verify file permissions/ownership of new libtigergraph.so (should be 755 / tigergraph)

grun all "ls -l /var/tigergraph/app/3.5.2/bin/libtigergraph.so"

Start GPE, GSE, RESTPP

gadmin start gpe gse restpp