Sort exceeded memory limit of 104857600 bytes

Chandra_Mohan1 · 2024-03-06T08:08:43.878Z

I was creating a Online Archive on one of the collections and this collection had objects more than 50 millions. On the confirmation page we 4 steps

Review all the details associated with archiving
Run the query in mongo shell which has the sort on date column
Query to create index on date column.
Click begin archiving.

Running query in the 2nd step was causing this above mentioned error.

My concern is
will this error have any impact while the online archiver schedule job starts archiving the data on a given date & time.

Prem_PK_Krishna · 2024-03-15T18:24:38.344Z

Hello,

You are likely seeing this error message in the 2nd step because there is no index on the date field based on which the Online Archive is getting configured.

From our documentation:

Atlas runs an index sufficiency query to determine the efficiency of the archival process. If the number of documents scanned to the number of documents returned is 10 or more, the query result triggers an Index Sufficiency Warning . This warning indicates that you have insufficient indexes for an efficient archival process. For date-based archives, you must index the date field.

Once the index is created on the date field, you can rerun Step-2 and the error message should not be seen.

Thanks !

Chandra_Mohan1 · 2024-03-19T05:58:50.524Z

This issue seems to be resolved on non-prod environment which had around 10-15 million records. Our prod environment has around 4-6 trillion records, its going to be a big challenge of creating index on a date field. Currently we have a index on a column which is not a date field.

Prem_PK_Krishna · 2024-03-19T16:18:43.060Z

Hello,

Online Archive based on a date field must have an index. This is mentioned as part of our documentation.

As part of creating an Online Archive, as part of Step-2 of the process, it is mentioned in the UI:

Date Field

This field must already have an index. Learn more about indexes.

As part of Step-4, in the summary page as well, we mention about having the required indexes.

Regarding indexes, note that Atlas can also use a compound index. You need not create a separate index if the date time field is part of a compound index. As long as the date time field is one of the fields in the compound index, you should not get any index sufficiency error for online archive.

Only if there are no indexes or compound indexes that contain the date time field, will you receive the index sufficiency error.