Insert performance

lvn2007 · 2024-09-05T21:13:08.944Z

Hi all,

Thank you in advance for your reply.
I am using mongocxx r3.10.2 (Client) and v7.0.14 (Server) to conduct benchmarks to better understand MongoDB’s performance characteristics when dealing with time-series data.

The testbed includes 1x Server machine and x8 Client machines. All of which have the same underlying hardware:

Dual socket Intel Xeon Silver 4108 @1.80GHz
64GB RAM
100 Gbit Ethernet
4x Samsung NVMe SSD 970 EVO Plus (250GB) as RAID 0 array

The problem that I am observing is that when launching 256 clients where each client performs a single sequential insertion at a time for 1,000 times, the observed round-trip latency is often well-above 20,000 (ms). On the other hand, the same test is conducted with the exception that the inserting data is organized as random access pattern, the avg. latency is around 220 (ms) per insert.
For the above, a single insert include a timestamp with the rest of 32 datapoints where each data point is 32 bytes. Totaling 1,024 bytes + timestamp for a single insert.

On server’s end, the test database is configured as a time-series collection with granularity as seconds.

On client’s end, the code is as following:

  // Init MongoDB instances
  mongocxx::uri uri(fmt::format("mongodb://{}:{}/?socketTimeoutMS=86400000", SERVER_ADDR, SERVER_PORT));
  mongocxx::client mongoClient(uri);
  mongocxx::options::insert insertOptions;
  insertOptions.ordered(false);

  // Access DB and collection
  const mongocxx::database mongoDB = mongoClient["BENCH_DB"];
  mongocxx::collection mongoCollection = mongoDB["BENCH_DB"];

  // Create document
  bsoncxx::builder::basic::document currDocument = makeSingleDocument(&insertData->at(i));

  // Dispatch insert
  start = std::chrono::high_resolution_clock::now();
  const core::optional<mongocxx::result::insert_one> &insertResult = mongoCollection.insert_one(currDocument.view(), insertOptions);
  end = std::chrono::high_resolution_clock::now();

I’m not sure what cause such a large difference in performance and how to alleviate such bottleneck when the only difference in the underlying access pattern of data. I suspect the issue in SEQ pattern might be because of lock contention since data is essentially inserting to the same bucket one after another. While in RAND pattern, it’s inserting to different buckets so there’s little to no lock. However, I also think 20k (ms) latency for a single insert to complete is quite strange.

lvn2007 · 2024-09-23T02:03:52.776Z

Refreshing this thread …

Rishabh_Bisht · 2024-09-23T19:15:19.966Z

Hi @lvn2007

Are you able to replicate the increased latency if you perform same action via mongosh or perhaps another driver (like C Driver)?

lvn2007 · 2024-10-02T12:20:27.573Z

I have not tried to replicate the same using different driver. However, I had used pymongo to perform the same operation with 1 client thread and that seems to work okay.
That indicates Mongo server might be overwhelmed which ultimately caused the increased in latency. Although, I’m still puzzled with 20k (ms) latency when handling 256 concurrent clients.