Find and insert data, from different collections, inside different databases, without replica set

Working with Data
, ,
1

Hello there,

Due to some reasons, cannot have a replica set enable, so cannot use (change streams), etc…

I am trying to basically compare 2 collections, see where are the differences, and send the missing data from one db to another. (It can happens both directions)

Where did I arrive?

I have a working solution ( although I need to do more tests), but I want to know if there is a better way to do this. (without using change streams…)

I start with a QMap(called matrix) (I am using QT with cpp) where I store the differences between database 1 and 2, and with this I can create a query, for every missing element in the other db.

mongocxx::options::find opts{};
    opts.projection(make_document(kvp("_id", 0)));
    bsoncxx::builder::basic::array array;

    for (auto map = matrix.constBegin(); map != matrix.constEnd(); map++) {
        int machineId = map.key();
        int value = map.value();
        qDebug() << machineId << value;
        array.append(
            make_document(
                kvp("machineId", machineId),
                kvp("operationId", make_document(kvp("$gt", value)))
            )
        );
    }

Since that, I can create a cursor

    auto cursor = collectionSource.find(make_document(kvp("$or", array)), opts);

And after that, I just need to create a QVector and iterate over the cursor to add documents to the vector, so I insert it into the ‘collectionTarget’. I decided to do 50 documents at a time, to have less calls to the db.

    QVector<bsoncxx::document::view> documentsToInsert;
    for (auto &&doc : cursor) {
        documentsToInsert.push_back(doc);
        qDebug() << bsoncxx::to_json(doc);
        if (documentsToInsert.size() >= 50) {
            collectionTarget.insert_many(documentsToInsert);
            documentsToInsert.clear(); // Clear after insertion
            qDebug() << "Inserted 50 documents...";
        }
    }

    // Insert remaining documents
    if (!documentsToInsert.isEmpty()) {
        collectionTarget.insert_many(documentsToInsert);
        qDebug() << "Inserted remaining documents: " << documentsToInsert.size();
    }

Is there a better way to approach this??