API calls to :14240/gsqlserver fail with 5xx HTTP response
Problem
User reported issues with all requests to GSQL server API endpoints. These were failing with HTTP 5xx error codes on all nodes in the cluster.
Error message
<html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx</center>
</body>
</html>Diagnosis
First, we confirmed the request path. User was sending requests to an external load balancer which was forwarding them to TigerGraph. Additionally looked into the GSQL server logs and saw the following messages:
I@20220622 23:24:28.346 (BaseHandler.java:759) Forwarded request /127.0.0.1:37266 from /127.0.0.1:36078/requesttoken
I@20220622 23:24:28.346 (SessionManager.java:235) Add session with cookie: AWSALB=59kasdgkq643b/q3l90sm2jufg734mdpas89f[...]
I@20220622 23:24:28.347 (Util.java:2837) Error when parsing cookie: AWSALB=59kasdgkq643b/q3l90sm2jufg734mdpas89f[...]Attempted to reproduce by setting a cookie with curl’s -b flag:
$ curl -b 'TEST=asldgkasldgkasdlg' localhost:14240/restpp/requesttoken -d '{"secret": "42p[...]nb0"}'
<html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx</center>
</body>
</html>This confirmed the issue was related to the GSQL server handling cookies.