GSE Reuse Deleted Vertex ID

Symptoms

In this issue, symptoms are not very clear from error messages and log information. Please see below for possible symptoms:

Vertex newly created getting deleted immediately

Unable to update and insert certain vertex types

When TSEs try to reproduce the issue in an internal testing environment, the issue often couldn’t be reproduced, as it’s related with the data specific in the customer’s operating environment. This issue often couldn’t be directly discovered from the GPE and GSE logs.

Error Messages

There are no directly related error messages reported with this issue. However, to confirm this issue, we use the following example to illustrate:

Suppose there’s a UID "5200" of type "Page" missing. You can check whether it exists in GSE or not with the following command:

curl 'localhost:9000/idservice?id=5200&type=Page'

If the "error" in the response is "false", it means it exists in GSE. Then check whether it exists in GPE with the following command:

curl -v 'http://localhost:9000/graph/gsql_demo/vertices/Page/5200'

Cause

In this known issue, the GSE may allocate a vertex ID previously marked as deleted to a vertex, which will cause the data inconsistency between the two major TigerGraph engine components: Graph Processing Engine (GPE) and Graph Storage Engine (GSE).

The root cause for this data inconsistency problem is that GSE reused deleted VIDs at the end of segments after GSE restarted. This will cause the vertex only exists in GSE but not in the GPE, as in the GPE, it has been marked as deleted.

As a result, the data get inconsistent between GPE and GSE, which may cause many problems described in the Symptoms section.

Resolution

To resolve this issue, please upgrade the system to TG3.5.2 or later versions.

To diagnosis this issue, please use the following steps:

Step 1: Check if this symptom is related to this deletion vertex issue with the following command. It will return “check failed” if there’s any data inconsistency between GSE and GPE.

curl -s -H "GSQL-TIMEOUT: 36000000" -X GET "http://localhost:9000/deleted_vertex_check?threadnum=10&verbose=1" | jq > deleted_vertex_check.json

Step 2: Check the result from Step 1 and see if there are deleted vertex issue from deleted_vertex_check.json file

Step 3: After the deletion vertex issue is confirmed, use the following command to align the GPE and GSE data:

gadmin config set GSE.BasicConfig.Env "$(gadmin config get GSE.BasicConfig.Env); DeleteByForce=1"
gadmin config apply -y
gadmin restart gse -y
curl -s -H "GSQL-TIMEOUT: 36000000" 'http://localhost:9000/deleted_vertex_check?threadnum=10&verbose=1&fixGSE=true&fixGPE=true'

Step 4: When the mismatched data is large, and the system response time is long. Then, if the customer can reload the data, alternatively, we can also clear the graph data for fast processing with the following command:

CLEAR GRAPH STORE -HARD

Step 5: Rerum the following command to double check if this issue has been fixed. It should return “check passed” after this issue has been fixed.

curl -s -H "GSQL-TIMEOUT: 36000000" -X GET "http://localhost:9000/deleted_vertex_check?threadnum=10&verbose=1" | jq > deleted_vertex_check.json

Step 6: Reload data if there is an upgrade from previously affected version.