Hi all,
Been using the MongoDB Connector for Spark a while now and recently just upgraded to the latest 10.3.0 version with all the breaking changes solved since I was using the Mongo API as is, with filters and such.
After upgrading both the connector and the MongoDB Scala libraries:
val mongoScalaDriver = "org.mongodb.scala" %% "mongo-scala-driver" % "4.8.2" exclude("org.mongodb", "mongodb-driver-reactivestreams")
val mongoSparkConnector = "org.mongodb.spark" %% "mongo-spark-connector" % "10.3.0" exclude("org.mongodb", "mongodb-driver-reactivestreams")
I’m getting the following error when the project is compiled and deployed, running in local runs just fine:
2024-07-29T09:58:14,917 ERROR [main] boot.SpringApplication: Application run failed
java.lang.IllegalStateException: Failed to execute ApplicationRunner
at ...
Caused by: java.lang.ClassNotFoundException:
Failed to find data source: mongodb. Please find packages at
https://spark.apache.org/third-party-projects.html
... 27 more
Caused by: java.lang.ClassNotFoundException: mongodb.DefaultSource
at java.base/java.net.URLClassLoader.findClass(Unknown Source)
... 38 more
The way I write into Mongo has changed, so that may be related too…
As of now, I’m writing as:
myDf
.write
.format("mongodb")
.option("connection.uri", "...")
.option("database", "...")
.option("collection", "...")
.option("idFieldList", "...")
.option("upsertDocument", value = true)
.mode(SaveMode.Overwrite)
.save()
Prior the update this has work flawlessly, so I can only imagine this is related to some extra dependencies I’m missing or something? Hope someone can shed a light in here…