Server encountered internal error while handling request

Anton_P · 2024-08-09T12:05:56.518Z

We are facing thousands of “Server encountered internal error while handling request” errors starting from “2024-07-12”. At first, I thought they would disappear over time or wouldn’t be a big issue but it looks like magnitude growth day by day. At the start, it was like several errors per day; now it is more like hundredths per day.

We also started to receive user reports that seem related to Device Sync so, it may be related to the errors we see in logs.

Errors are also visible under the Device Sync - Metrics tab

image1926×508 39.2 KB

Several error examples:

{
  "_id" : "66b5f66eda89c60fc51d55fc",
  "app_id" : "606fdd5a64896dd874cae095",
  "co_id" : "66b5f632da89c60fc516ec6e",
  "completed" : "2024-08-09T10:58:59.79Z",
  "domain_id" : "606fdd5a64896dd874cae096",
  "error" : "Server encountered internal error while handling request",
  "function_call_location" : "US-VA",
  "function_call_provider_region" : "aws-us-east-1",
  "group_id" : "606fd776216356502b39baf5",
  "remote_ip_address" : "88.6.30.82",
  "request_method" : "GET",
  "request_url" : "/api/client/v2.0/app/divtracker-uhfjk/realm-sync",
  "started" : "2024-08-09T10:58:54.789Z",
  "status" : 500,
  "type" : "API"
}, {
  "_id" : "66b5f675da89c60fc51d9939",
  "app_id" : "606fdd5a64896dd874cae095",
  "co_id" : "66b5f639da89c60fc517bceb",
  "completed" : "2024-08-09T10:59:01.355Z",
  "domain_id" : "606fdd5a64896dd874cae096",
  "error" : "Server encountered internal error while handling request",
  "framework_name" : "Android",
  "framework_version" : "34",
  "function_call_location" : "US-VA",
  "function_call_provider_region" : "aws-us-east-1",
  "group_id" : "606fd776216356502b39baf5",
  "platform" : "Android",
  "platform_version" : "14",
  "remote_ip_address" : "31.4.136.137",
  "request_method" : "GET",
  "request_url" : "/api/client/v2.0/app/divtracker-uhfjk/realm-sync",
  "sdk_name" : "Java",
  "sdk_version" : "10.17.0",
  "started" : "2024-08-09T10:59:01.222Z",
  "status" : 500,
  "type" : "API"
}, {
  "_id" : "66b5f680da89c60fc51ea5d1",
  "app_id" : "606fdd5a64896dd874cae095",
  "co_id" : "66b5f644da89c60fc519cfd2",
  "completed" : "2024-08-09T10:59:45.969Z",
  "domain_id" : "606fdd5a64896dd874cae096",
  "error" : "Server encountered internal error while handling request",
  "function_call_location" : "US-VA",
  "function_call_provider_region" : "aws-us-east-1",
  "group_id" : "606fd776216356502b39baf5",
  "remote_ip_address" : "124.120.206.248",
  "request_method" : "GET",
  "request_url" : "/api/client/v2.0/app/divtracker-uhfjk/realm-sync",
  "started" : "2024-08-09T10:59:12.388Z",
  "status" : 500,
  "type" : "API"
}, {
  "_id" : "66b5f682da89c60fc51ecb35",
  "app_id" : "606fdd5a64896dd874cae095",
  "co_id" : "66b5f646da89c60fc51a09fc",
  "completed" : "2024-08-09T10:59:45.971Z",
  "domain_id" : "606fdd5a64896dd874cae096",
  "error" : "Server encountered internal error while handling request",
  "framework_name" : "Android",
  "framework_version" : "34",
  "function_call_location" : "US-VA",
  "function_call_provider_region" : "aws-us-east-1",
  "group_id" : "606fd776216356502b39baf5",
  "platform" : "Android",
  "platform_version" : "14",
  "remote_ip_address" : "107.116.79.101",
  "request_method" : "GET",
  "request_url" : "/api/client/v2.0/app/divtracker-uhfjk/realm-sync",
  "sdk_name" : "Java",
  "sdk_version" : "10.17.0",
  "started" : "2024-08-09T10:59:14.813Z",
  "status" : 500,
  "type" : "API"
}

Is it a known error? Is there any ETA for resolution?

Jonathan_Lee · 2024-08-09T16:37:52.341Z

Hi,

I took a look at your app on our end, and it seems the internal server errors have to do with difficulties connecting to your MongoDB cluster. These types of errors are benign, as Device Sync has retryability built into it. I’ll follow up with the team to prevent this class of error from being reported as an internal server error.

Anton_P · 2024-08-11T10:07:25.973Z

I appreciate your help @Jonathan_Lee

However, even if Device Sync retries some errors, I see in logs that multiple clients might be affected for an hour or so during the day which might cause a bad experience for users. Are those connection errors you mentioned something that can be fixed?

Anton_P · 2024-08-11T11:05:13.058Z

Usually, everything works as expected but then there are hours when errors come in big numbers. For example, this is the latest incident. For ~15 mins errors are spiking:

started: 2024-08-11T09:05:15.197Z
started: 2024-08-11T09:05:33.205Z
started: 2024-08-11T09:05:57.596Z
started: 2024-08-11T09:06:46.517Z
started: 2024-08-11T09:06:49.263Z
started: 2024-08-11T09:07:36.217Z
started: 2024-08-11T09:07:36.217Z
started: 2024-08-11T09:07:36.217Z
started: 2024-08-11T09:07:37.723Z
started: 2024-08-11T09:07:37.783Z
started: 2024-08-11T09:09:13.594Z
started: 2024-08-11T09:09:46.633Z
started: 2024-08-11T09:09:57.089Z
started: 2024-08-11T09:09:57.107Z
started: 2024-08-11T09:10:30.163Z
started: 2024-08-11T09:10:30.214Z
started: 2024-08-11T09:10:30.225Z
started: 2024-08-11T09:10:30.521Z
started: 2024-08-11T09:10:30.538Z
started: 2024-08-11T09:10:57.184Z
started: 2024-08-11T09:10:57.249Z
started: 2024-08-11T09:11:33.601Z
started: 2024-08-11T09:11:41.914Z
started: 2024-08-11T09:11:46.689Z
started: 2024-08-11T09:11:57.207Z
started: 2024-08-11T09:11:57.213Z
started: 2024-08-11T09:11:57.227Z
started: 2024-08-11T09:12:40.281Z
started: 2024-08-11T09:12:40.281Z
started: 2024-08-11T09:12:43.051Z
started: 2024-08-11T09:13:28.205Z
started: 2024-08-11T09:14:25.998Z
started: 2024-08-11T09:14:47.709Z
started: 2024-08-11T09:14:57.361Z
started: 2024-08-11T09:14:57.418Z
started: 2024-08-11T09:14:57.471Z
started: 2024-08-11T09:15:46.757Z
started: 2024-08-11T09:15:46.81Z
started: 2024-08-11T09:17:53.453Z
started: 2024-08-11T09:17:53.457Z
started: 2024-08-11T09:18:22.904Z
started: 2024-08-11T09:18:59.278Z
started: 2024-08-11T09:19:01.462Z
started: 2024-08-11T09:19:08.676Z
started: 2024-08-11T09:19:12.605Z
started: 2024-08-11T09:19:12.605Z
started: 2024-08-11T09:19:12.605Z
started: 2024-08-11T09:19:14.59Z
started: 2024-08-11T09:19:18.653Z
started: 2024-08-11T09:19:20.361Z
started: 2024-08-11T09:19:20.362Z
started: 2024-08-11T09:19:20.364Z
started: 2024-08-11T09:19:20.377Z
started: 2024-08-11T09:19:20.509Z
started: 2024-08-11T09:21:16.552Z

Anton_P · 2024-08-16T08:44:23.729Z

The issue seems resolved, thanks

image1928×722 60.2 KB