Listing Collection Names Performance

Amitoj_Singh1 · 2024-09-18T18:36:40.300Z

What is the performance cost for a line like this, where we use listCollections and a filter to get if a collection already exists in a database?

There is a chance there may be a lot of collections in our database. Would a line like this be costly?

col_list = db.list_collection_names(filter={"name": name})

Vivekanandan_Sakthivelu · 2024-09-19T02:49:32.836Z

You can either use runCommand or show collection on terminal to list collections on database . This will not have any performance impact , as the information about the collection is pulled from metadata. Usually this metadata information is kept in memory for fast access .

Ref:

mongodb.com

listCollections - MongoDB Manual v7.0

Implementation of List Command

github.com

mongodb/mongo/blob/05e7767cea66a858fbb906b9d01f489e97064ae5/src/mongo/db/commands/list_collections.cpp#L350


      
          void doCheckAuthorization(OperationContext* opCtx) const final {
              AuthorizationSession* authzSession = AuthorizationSession::get(opCtx->getClient());
              uassertStatusOK(authzSession->checkAuthorizedToListCollections(request()));
          }
          
          NamespaceString ns() const final {
              return NamespaceString(request().getDbName());
          }
          
          Reply typedRun(OperationContext* opCtx) final {
              CommandHelpers::handleMarkKillOnClientDisconnect(opCtx);
              unique_ptr<MatchExpression> matcher;
              const auto as = AuthorizationSession::get(opCtx->getClient());
          
              const auto listCollRequest = request();
              const auto dbName = listCollRequest.getDbName();
              const bool nameOnly = listCollRequest.getNameOnly();
              const bool authorizedCollections = listCollRequest.getAuthorizedCollections();
          
              // We need to copy the serialization context from the request to the reply object
              const auto respSerializationContext =

github.com

mongodb/mongo/blob/master/src/mongo/db/catalog/collection_catalog.cpp#L925


      
                  return;
              }
          
              LOGV2_DEBUG(22546, 1, "Reloading view catalog for database", logAttrs(dbName));
          
              uncommittedCatalogUpdates.replaceViewsForDatabase(dbName,
                                                                loadViewsForDatabase(opCtx, *this, dbName));
              PublishCatalogUpdates::ensureRegisteredWithRecoveryUnit(opCtx, uncommittedCatalogUpdates);
          }
          
          const Collection* CollectionCatalog::establishConsistentCollection(
              OperationContext* opCtx,
              const NamespaceStringOrUUID& nssOrUUID,
              boost::optional<Timestamp> readTimestamp) const {
              if (_needsOpenCollection(opCtx, nssOrUUID, readTimestamp)) {
                  auto coll = _openCollection(opCtx, nssOrUUID, readTimestamp);
          
                  // Usually, CappedSnapshots must be established before opening the storage snapshot. Thus,
                  // the lookup must be done from the in-memory catalog. It is possible that the required
                  // CappedSnapshot was not properly established when this operation was collection creation,
                  // because a Collection instance was not found in the in-memory catalog.

Amitoj_Singh1 · 2024-09-19T15:03:14.230Z

For the sake of argument, what if the collection list had thousands or even tens of thousands of items? Would the query still be lightweight enough, or would it have a noticeable impact on performance?

steevej · 2024-09-19T21:36:47.718Z

Amitoj_Singh1:

Would the query still be lightweight enough

No it won’t.

Each collection is at least 1 file. If your server has to manage that many files you would end up with an anti-pattern.

You will have other issues a lot sooner than any issue with listing the collections.

Vivekanandan_Sakthivelu · 2024-09-20T00:47:30.913Z

@Amitoj_Singh1 , run command returns cursor not the entire resultSet. There is a filter option in list Command so that can help to limit your result . As long as you don’t write that in a while loop your good

The anti pattern @steevej mentioned is about having large numbers of collection when you can still optimize to have smaller numbers.