Can we get an RDD of documents with mongo-spark-connector 10.1.1?

Basant_Gurung · 2024-01-04T07:38:40.854Z

I am working on a spark application that involves reading data from Mongo where the collections may have millions of data with different structures.
Recently, I have migrated from using mongo-spark-connector 2.4.4 to 10.1.1.
The connector 10.1.1 samples 1000 (default) documents to infer the schema for creating the data frame.
The connector 2.4.4 used to have MongoSpark.load() which returns MongoRDD[Document].
While dealing with collections of millions of documents with different structures, using 10.1.1 could lead to data loss as the schema would be inferred using only 1000 docs. We cannot set this value to a million.
Is there a way to overcome this problem? Or is there still a way to get an RDD of documents like it used to have earlier?

Prakul_Agarwal · 2024-02-02T07:41:28.554Z

The support for RDD is not exposed in the v10.x of Spark Connector Drivers. you still see a dataframe inferred using https://www.mongodb.com/docs/spark-connector/current/batch-mode/batch-read/#schema-inference

You have two options - (a) Explicitly specify the schema instead of relying on the auto-inference, (b) Increase the number of documents that are sampled for auto-inferring the schema (however this will lead to proportional increase in initial time latency) .

mongodb.com

Batch Read Configuration Options — MongoDB Spark Connector

sampleSize The number of documents to sample from the collection when inferring the schema.

Basant_Gurung · 2024-02-20T13:06:58.840Z

Thanks @Prakul_Agarwal for the response!

system · 2024-02-27T07:41:06.296Z

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.