Introducing the Newest Version of the MongoDB Spark Connector

Robert Walters
March 31, 2022 | Updated: April 5, 2022

MongoDB has just released an all-new version of our Spark Connector. This article discusses the background behind the MongoDB Spark Connector and some of the key features of the new release.

Why a new version?

The current version of the MongoDB Spark Connector was written in 2016 and is based on Version 1 of the Spark Data Source API. This API is still supported, but Databricks has released an updated version of the API, making it easier for data sources like MongoDB to work within the Spark ecosystem. By using Version 2 of the MongoDB Spark Connector, you’ll immediately benefit from capabilities such as tighter integration with Spark Structured Streaming.

MongoDB will continue to support Version 1 until Databricks deprecates its Datasource API, but no new features will be implemented, and upgrades to the Connector will include only bug fixes and support for the current version.

Which version should I use?

The new Spark Connector (Version 10.0) is not intended to be a direct replacement for applications that use the current MongoDB Spark Connector. Note that the new connector uses a different namespace, “com.mongodb.spark.sql.connector.MongoTableProvider”, versus the original Spark Connector, which uses “com.mongodb.spark.DefaultSource”. Having a different namespace makes it possible to use both versions of the Connector within the same Spark application. This is helpful in unit testing your application with the new Connector and making the transition on your timeline.

Also note a change with versioning of the MongoDB Spark Connector. The current version of the existing MongoDB Spark Connector is 3.0. Up until now, as MongoDB released versions of the connector, the number was aligned with the version of Spark that was supported—i.e., Version 2.4 of the MongoDB Spark Connector works with Spark 2.4. Going forward, this will not be the case. MongoDB's documentation will make clear which versions of Spark the Connector supports and provide the appropriate information.

Structured Streaming to MongoDB

Apache Spark comes with a stream processing engine called Structured Streaming, which is based on Spark's SQL engine and DataFrame APIs. Spark Structured Streaming treats each incoming stream of data as a microbatch, continually appending each microbatch to the target dataset. This makes it easy to convert existing Spark batch jobs into streaming jobs. Structured Streaming provides maximum throughput via the same distributed capabilities that have made Spark such a popular platform. In the following example, we’ll show you how to stream data to MongoDB using Structured Stream.

Consider a CSV file that contains natural gas prices. The following PySpark code will read the CSV file into a stream, compute a moving average, and stream the results into MongoDB.

from pyspark.sql.types import StructType, DateType, StringType, TimestampType, DoubleType
from pyspark.sql import functions as F
from pyspark.sql.window import Window
from pyspark.sql.functions import lit, count

sc.setLogLevel('DEBUG')

readSchema = ( StructType()
  .add('Type', StringType())
  .add('Date', TimestampType())
  .add('Price', DoubleType())
)


ds = (spark
  .readStream.format("csv")
  .option("header", "true")
  .schema(readSchema)
  .load("daily*.csv"))


slidingWindows = (ds
  .withWatermark("Date", "1 minute")
  .groupBy(ds.Type, F.window(ds.Date, "7 day"))
  .avg()
  .orderBy(ds.Type,'window'))

dsw = (
  slidingWindows
    .writeStream
    .format("mongodb")
    .queryName("7DaySlidingWindow")
    .option("checkpointLocation", "/tmp/pyspark/")
    .option("forceDeleteTempCheckpointLocation", "true")
    .option('spark.mongodb.connection.uri', 'MONGODB CONNECTION HERE')
    .option('spark.mongodb.database', 'Pricing')
    .option('spark.mongodb.collection', 'NaturalGas')
    .outputMode("complete"))

query = dsw.start()
query.processAllAvailable()
query.stop()

For more information and examples on the new MongoDB Spark Connector V10.0, check out our documentation.

Ask questions and give feedback on the MongoDB Community Forum. The Connector is open sourced; feel free to contribute at GitHub.

← Previous

How These Women Are Leading Teams and Growing Their Careers at MongoDB

Each year, MongoDB highlights some of our most influential leaders in celebration of International Women’s Day and Women’s History Month. These women were nominated by their colleagues for making a huge impact on their teams. They’re taking their careers (and MongoDB) to new heights and inspire us every day! Reba Cox , Director of Procurement I currently lead the Procurement and Travel teams, and I was recently asked to build out a new function for Sustainability at MongoDB. My team ensures that MongoDB spends money safely, efficiently, and compliantly! I have somehow turned my personal passion for shopping into a full-time job. I started at MongoDB as a manager and have since been promoted twice and expanded my leadership to three functions. I have gained more experience in four years at MongoDB than I gained in all of my career prior! I think a good leader has to have a combination of traditional leadership skills, such as making and implementing multiyear strategic goals, being well-spoken, and standing up for what’s right, but they also need to have compassion and put a huge emphasis on developing their team. If you are not spending the majority of your time building leaders, then you are missing a great opportunity to multiply your positive impact at a company and in people’s lives. The greatest lesson I have learned as a leader is to have compassion. I can be very demanding of my team, but the one thing they know is that I always have their back and I honestly try to do what’s best for them every single day. If my team doesn’t trust or believe in me, then how can I ask great and difficult things from them? If you want to be a leader someday, then showing up, speaking up, and consistently doing what you say will help you get there. I always tell my team to be someone who people can rely on and trust. Be more helpful, more kind, and more ‘can do’ than anyone expects you to be! You will gain respect among your peers and put yourself in the right position to receive additional responsibilities and opportunities. The opportunities at MongoDB are limitless. As a female leader at MongoDB, I have been able to create so many lasting friendships and mentorship opportunities with an endless list of incredible women. The most rewarding part of my job is mentoring and working with the many fabulous women I come in contact with on a day-to-day basis. I love how the culture of MongoDB allows us each to be unique and really lean into who we are as women and leaders. The growth you can experience here, both personally and professionally, is endless. Anokhee Mepani , Director of Operations, Technical Services For the past several years, I’ve led customer success and operations teams at rapidly growing startups. I deeply enjoy working through the opportunities and challenges that come with fast, exponential scaling, and I wanted to expand my experience at a larger but equally fast-growing tech company. My previous companies used MongoDB to power their businesses, and I saw firsthand how impactful MongoDB is. I knew I would find the work here meaningful and exciting, so it was an easy decision to join my team. Technical Services Operations supports the Technical Services team , which provides 24/7 support to our customers. Our mission is to enable Technical Services to serve our customers successfully. As a leader, it’s my responsibility to ensure my team is best set up for success, and my continued growth and learning are a big part of that. I am focused on learning through inclusion — learning from my team’s and my colleagues’ different perspectives and varied experiences. As an individual contributor, I learned a lot from industry best practices and standard approaches; as a leader, I believe my greatest growth comes from seeking novel and diverse solutions to challenges. What I believe makes someone a good leader is truly caring about your team and providing support and advocacy that is in the team’s best interest. I truly believe that if you set high standards and take care of your people, then success will follow. Cristina Castillo , Director of Sales Development, EMEA I currently lead the South and Central European regions for the Sales Development organization. Our team plays a crucial role for MongoDB’s Sales team. We generate business opportunities for the most strategic accounts in all types of industries and work with the Enterprise organization to plan and execute pipeline-generation strategies. Working in the Sales Development organization is really exciting. As a leader, you have a huge responsibility for the development of emerging professionals, and I want to make my team’s time in Sales Development a memorable experience and empower them to own their careers . From a leadership perspective, compassion is key. You need to understand your team’s feelings, goals, strengths, and areas of development, and be passionate and committed to supporting their success and career growth. Self-awareness is also really important; we are an example for the people around us, so we need to be able to self-reflect, ask for feedback, and analyze our development areas to always become better as leaders. In addition, you need to be able to build trust with your team so you can create a safe environment where people will be encouraged to speak up and communicate not only the positives, but also the negatives so you can take action. If your team is happy, then they will deliver at their best! If you are new to a leadership role, then you may face many challenging situations along the way. It is important that you believe in yourself and trust your gut. I also think it’s helpful to find a mentor you can trust who will give you guidance when you are lost and feedback to help you accelerate your skills. Last but not least, remember that, at the end of the day, we are humans dealing with humans, so make time for fun! Melissa Mahoney , Lead Technical Writer, Cloud I am fortunate to lead the incredibly smart, resourceful, and compassionate technical writers of the Cloud Documentation team. We produce procedural, reference, and tutorial content for MongoDB’s Cloud services and strive to provide users with delightful learning experiences and self-serve support through documentation. I have always had a passion for mentorship, both in learning from others and in helping those around me grow. I have been fortunate to turn that passion into a key part of my job. I transitioned from an individual contributor role to a team lead about a year and a half ago, and every day I learn something new about myself and how best to serve my team and their goals. I think a good leader must be grounded in empathy. Empathy for your users drives you to fully understand their needs, sparks creativity to meet those needs, and inspires the best possible deliverable. Empathy for your team requires that you listen actively and openly and communicate clearly and honestly, and it motivates you to seek collaborative solutions. Empathy for yourself forces you to delegate, seek help when you need it, and allow yourself to learn and grow from mistakes. The biggest lesson I’ve learned as a leader is that I can’t (and shouldn’t!) do it all. The transition from an individual contributor to a manager was difficult for me; at first, it was hard to let go of projects or meetings. But as my calendar filled up and the work I promised to do fell behind, I realized I had become the bottleneck I had tried so hard to prevent. Learning to delegate not only freed up my time to be a better manager, but also created new opportunities for leadership and expertise for members of the team. My advice to someone new to leadership is to learn to manage your time early on. Your calendar will fill quickly, so ask yourself if you really need to be there or if someone else could use this opportunity to learn or lead. Interested in making an impact at MongoDB? We have several open roles on our teams around the globe and would love for you to transform your career with us!

March 30, 2022

Next →

AI-Powered Call Centers: A New Era of Customer Service

Customer satisfaction is critical for insurance companies. Studies have shown that companies with superior customer experiences consistently outperform their peers. In fact, McKinsey found that life and property/casualty insurers with superior customer experiences saw a significant 20% and 65% increase in Total Shareholder Return , respectively, over five years. A satisfied customer is a loyal customer. They are 80% more likely to renew their policies, directly contributing to sustainable growth. However, one major challenge faced by many insurance companies is the inefficiency of their call centers. Agents often struggle to quickly locate and deliver accurate information to customers, leading to frustration and dissatisfaction. This article explores how Dataworkz and MongoDB can transform call center operations. By converting call recordings into searchable vectors (numerical representations of data points in a multi-dimensional space), businesses can quickly access relevant information and improve customer service. We'll dig into how the integration of Amazon Transcribe, Cohere, and MongoDB Atlas Vector Search—as well as Dataworkz's RAG-as-a-service platform— is achieving this transformation. From call recordings to vectors: A data-driven approach Customer service interactions are goldmines of valuable insights. By analyzing call recordings, we can identify successful resolution strategies and uncover frequently asked questions. In turn, by making this information—which is often buried in audio files— accessible to agents, they can give customers faster and more accurate assistance. However, the vast volume and unstructured nature of these audio files make it challenging to extract actionable information efficiently. To address this challenge, we propose a pipeline that leverages AI and analytics to transform raw audio recordings into vectors as shown in Figure 1: Storage of raw audio files: Past call recordings are stored in their original audio format Processing of the audio files with AI and analytics services (such as Amazon Transcribe Call Analytics ): speech-to-text conversion, summarization of content, and vectorization Storage of vectors and metadata: The generated vectors and associated metadata (e.g., call timestamps, agent information) are stored in an operational data store Figure 1: Customer service call insight extraction and vectorization flow Once the data is stored in vector format within the operational data store, it becomes accessible for real-time applications. This data can be consumed directly through vector search or integrated into a retrieval-augmented generation (RAG) architecture, a technique that combines the capabilities of large language models (LLMs) with external knowledge sources to generate more accurate and informative outputs. Introducing Dataworkz: Simplifying RAG implementation Building RAG pipelines can be cumbersome and time-consuming for developers who must learn yet another stack of technologies. Especially in this initial phase, where companies want to experiment and move fast, it is essential to leverage tools that allow us to abstract complexity and don’t require deep knowledge of each component in order to experiment with and realize the benefits of RAG quickly. Dataworkz offers a powerful and composable RAG-as-a-service platform that streamlines the process of building RAG applications for enterprises. To operationalize RAG effectively, organizations need to master five key capabilities: ETL for LLMs: Dataworkz connects with diverse data sources and formats, transforming the data to make it ready for consumption by generative AI applications. Indexing: The platform breaks down data into smaller chunks and creates embeddings that capture semantics, storing them in a vector database. Retrieval: Dataworkz ensures the retrieval of accurate information in response to user queries, a critical part of the RAG process. Synthesis: The retrieved information is then used to build the context for a foundational model, generating responses grounded in reality. Monitoring: With many moving parts in the RAG system, Dataworkz provides robust monitoring capabilities essential for production use cases. Dataworkz's intuitive point-and-click interface (as seen in Video 1) simplifies RAG implementation, allowing enterprises to quickly operationalize AI applications. The platform offers flexibility and choice in data connectors, embedding models, vector stores, and language models. Additionally, tools like A/B testing ensure the quality and reliability of generated responses. This combination of ease of use, optionality, and quality assurance is a key tenet of Dataworkz's "RAG as a Service" offering. Diving deeper: System architecture and functionalities Now that we’ve looked at the components of the pre-processing pipeline, let’s explore the proposed real-time system architecture in detail. It comprises the following modules and functions (see Figure 2): Amazon Transcribe , which receives the audio coming from the customer’s phone and converts it into text. Cohere ’s embedding model, served through Amazon Bedrock , vectorizes the text coming from Transcribe. MongoDB Atlas Vector Search receives the query vector and returns a document that contains the most semantically similar FAQ in the database. Figure 2: System architecture and modules Here are a couple of FAQs we used for the demo: Q: “Can you explain the different types of coverage available for my home insurance?” A: “Home insurance typically includes coverage for the structure of your home, your personal belongings, liability protection, and additional living expenses in case you need to temporarily relocate. I can provide more detailed information on each type if you'd like.” Q: “What is the process for adding a new driver to my auto insurance policy?" A: “To add a new driver to your auto insurance policy, I'll need some details about the driver, such as their name, date of birth, and driver's license number. We can add them to your policy over the phone, or you can do it through our online portal.” Note that the question is reported just for reference, and it’s not used for retrieval. The actual question is provided by the user through the voice interface and then matched in real-time with the answers in the database using Vector Search. This information is finally presented to the customer service operator in text form (see Fig. 3). The proposed architecture is simple but very powerful, easy to implement, and effective. Moreover, it can serve as a foundation for more advanced use cases that require complex interactions, such as agentic workflows , and iterative and multi-step processes that combine LLMs and hybrid search to complete sophisticated tasks. Figure 3: App interface, displaying what has been asked by the customer (left) and how the information is presented to the customer service operator (right) This solution not only impacts human operator workflows but can also underpin chatbots and voicebots, enabling them to provide more relevant and contextual customer responses. Building a better future for customer service By seamlessly integrating analytical and operational data streams, insurance companies can significantly enhance both operational efficiency and customer satisfaction. Our system empowers businesses to optimize staffing, accelerate inquiry resolution, and deliver superior customer service through data-driven, real-time insights. To embark on your own customer service transformation, explore our GitHub repository and take advantage of the Dataworkz free tier .

November 27, 2024