Diana Annie Jenosh

Senior Partner Solutions Architect

At Ignite, MongoDB announced a new milestone in their relationship with the integration of MongoDB Atlas with Microsoft Azure OpenAI Service. This significant integration is a testimony to the commitment of our collaboration and serves as a milestone to help customers on the journey of building generative AI applications grounded in enterprise data, providing for rich and meaningful user interactions.

## Advanced AI and rich data: Best of both worlds

Retrieval-augmented generation (RAG) is becoming the norm for generative AI applications. Without RAG, large language models (LLMs) can only answer based on the public data which they have been trained upon. This is a significant limitation and inhibits the use of these LLMs for production use cases. RAG helps overcome this challenge by grounding the LLM in the enterprise-owned data which the LLM otherwise had no knowledge of. 

Azure OpenAI Service helps to build state-of-the-art generative AI applications by choosing powerful models such as GPT-4o and adding more context to the LLMs from enterprise data, which now can be from MongoDB Atlas, to get more relevant and insightful conversations. Azure OpenAI Service is available as an API, SDK, and in Azure AI Foundry. Thus, customers using Azure AI Foundry can now add MongoDB Atlas vector index in the chat playground and evaluate the LLMs with the vector data stored in MongoDB Atlas collections. The API can directly be invoked from applications to easily build RAG applications grounded in MongoDB Atlas data.

Azure AI Foundry is a good starting point for enterprises aiming to build innovative generative AI applications in a secure and responsible way. The model catalog in Azure AI Foundry is a hub of models from Azure OpenAI Service and other leading providers, offering foundation, open, task, and industry-specific models. AI Foundry supports the entire lifecycle of designing an AI application starting from model evaluation and benchmarking, customization of the application,  flow creation, and debugging, to managing generative AI applications without compromising security and following responsible AI best practices. On the other hand, MongoDB Atlas is one of the world’s most popular document model databases with native vector search capabilities. [MongoDB Atlas Vector Search](https://mongodb.prakticum-team.ru/lp/cloud/atlas/try4?webxp579=true&utm_source=google&utm_campaign=search_gs_pl_evergreen_vector-search_product_prosp-brand_gic-null_ww-multi_ps-all_desktop_eng_lead&utm_term=mongodb%20atlas%20vector%20search&utm_medium=cpc_paid_search&utm_ad=p&utm_ad_campaign_id=21100816499&adgroup=162920454394&cq_cmp=21100816499&gad_source=1&gclid=Cj0KCQjwsoe5BhDiARIsAOXVoUuaMs42NEwXdNZYR3vcROrgzN-cQLCgsqmNWRc1nZSWbH_3J4UaqvkaAoe-EALw_wcB) was recognized by the [Retool Survey](https://mongodb.prakticum-team.ru/resources/solutions/use-cases/retool-2024-state-of-ai-in-production) as a leader with the highest Net Promoter Score (NPS) in 2023 and 2024\. MongoDB Atlas stores vector data alongside operational data without any additional overhead, owing to its inherently flexible document data model. MongoDB Atlas supports full text and hybrid searches along with vector search on dedicated search nodes in Azure which ensures that operational workloads are not impacted by generative AI application workloads.

## MongoDB Atlas with Azure OpenAI Service: RAG 

MongoDB Atlas is preferred as a vector database for RAG applications as it can store vectors seamlessly with the rest of the operational data, which means no ETL, reduced architectural complexity, and cost savings. MongoDB’s native vector search uses the uniform MongoDB API which implies it can be combined with other MongoDB Query API constructs to enable pre-filtering and post-filtering. MongoDB Atlas is designed for reliability, scalability, and security which are of prime importance when it comes to generative AI applications. Enabling easy access to MongoDB data via Azure OpenAI Service makes RAG implementations possible for even non-developers.

The diagram below shows the RAG architecture implemented through this integration. The user’s question is converted to embeddings using an embedding model such as “text-embedding-ada-002.” In the retrieval stage, the embeddings pertaining to the user query are used for a vector search in MongoDB Atlas to get similar chunks from the MongoDB Atlas collection. In the Augmented stage, these chunks retrieved from MongoDB Atlas are added to the actual user query as context to create a prompt for the LLM. In the Generation stage, the LLM is invoked passing it the augmented prompt, and now the LLM answers to the query grounded in the context provided in the prompt. The result is that now, the LLM provides accurate and relevant answers. Refer to the citations to see what data was retrieved from MongoDB Atlas that led to the answer provided by the LLM. 

![RAG with MongoDB Atlas and Azure OpenAI Service][1]

## MongoDB Atlas with Azure OpenAI Service: A seamless experience  

The integration is available as a REST API that can be invoked from your applications like any other REST API. Developers need to add the MongoDB collection and vector index information in the payload specifications along with the prompt and the system message/role information while invoking the API. The API will return the response along with the citations which point to the chunks retrieved from the MongoDB vector database along with the metadata of the chunk for easy cross verification. 

The integration is accessible from the chat playground in AI Foundry. It's a few clicks wizard experience to point the LLM to use the data from MongoDB Atlas collection while responding to the queries, providing for enhanced AI experience. MongoDB Atlas is available in the drop-down menu in the “Add your data” experience in the portal’s chat playground. Users just need to select MongoDB Atlas and point to the Atlas vector index to leverage the robust vector search capability of MongoDB Atlas and equip the LLMs with the precise information required to answer the queries. This makes creating chatbots and copilots for internal applications or customer-facing portals extremely easy.

Watch the video to see a [demo](https://youtu.be/ucOG5vdHbI8) of the integration in Azure AI Foundry and witness how easy it is now to bring your valuable data in MongoDB Atlas to your LLMs to build elegant conversational experiences.

> Join the [MongoDB AI Applications Program (MAAP)](https://mongodb.prakticum-team.ru/services/consulting/ai-applications-program) framework to accelerate your AI journey and let MongoDB help you take your AI application from ideation to production.
>
> Refer to the [Microsoft official documentation](https://aka.ms/OYDMongoDoc) for more details on the integration.


  [1]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt9f7f54f7cfa3032e/673bc2144ece01fcc8836450/image1.png

RAG Made Easy with MongoDB Atlas and Azure OpenAI

MongoDB is excited to present a mirroring solution for our customers who are looking to bring operational data from MongoDB Atlas to Microsoft Fabric for big data analytics, AI, and BI, combining it with the rest of the data estate of the enterprise. Mirroring in Fabric provides a unique way to bring in data from operational data stores into the uniform data layer of OneLake in Fabric. Open Mirroring is designed to enable Data ISVs to extend it because it is based on an open Delta Lake table format. In addition, Database Mirroring provides public APIs for Data ISVs to integrate with the open mirroring approach. Once mirroring is enabled for a MongoDB Atlas collection, the corresponding table in OneLake is kept in sync with the changes in source MongoDB Atlas collection, unlocking opportunities of varied analytics and AI and BI in near real-time. To learn more about the Open Mirroring capabilities, please visit [Introducing Open Mirroring in Microsoft Fabric](https://aka.ms/mirroring/intro-open-mirroring).

## MongoDB Atlas in Fabric ecosystem

Today, there are [multiple ways](https://mongodb.prakticum-team.ru/developer/products/atlas/unlock-value-data-mongodb-atlas-intelligent-analytics-microsoft-fabric/) data from MongoDB Atlas can be brought into OneLake in Microsoft Fabric. [Data pipelines](https://learn.microsoft.com/en-us/fabric/data-factory/connector-mongodb-atlas-overview) and [Dataflow Gen2](https://learn.microsoft.com/en-us/fabric/data-factory/connector-mongodb-atlas-sql-overview) are easy mechanisms to bring data from MongoDB Atlas to OneLake in batches or micro-batches. The Data Pipeline connector for MongoDB Atlas provides an easy experience via the [Copy data assistant](https://learn.microsoft.com/en-us/fabric/data-factory/create-first-pipeline-with-sample-data) which allows users to choose MongoDB Atlas as a source and OneLake as a target to push the operational data to OneLake. It also supports MongoDB Atlas as a target to ingest the analytics and enriched data back to MongoDB Atlas for persistence. Dataflow Gen2 is an [Atlas SQL-based connector](https://learn.microsoft.com/en-us/power-query/connectors/mongodb-atlas-sql-interface) and allows you to pull data from MongoDB Atlas to OneLake, do transformations such as filtering and flattening of data, and run power queries to create rich Power BI visualizations. Additionally, the [MongoDB Spark connector](https://mongodb.prakticum-team.ru/docs/spark-connector/v10.4/) can be used to create Fabric Spark notebooks that pull/push data from MongoDB Atlas in batch and streaming mode, as well. Data Pipeline Copy activity, Dataflow Gen2, and Spark notebooks can be orchestrated by Data Pipeline to build enterprise workflows with MongoDB as a key source and destination. 

## Mirroring MongoDB Atlas to Fabric OneLake

Joint customers will gain value from Mirroring, which will enable them with Spark-based analytics, SQL-based warehousing capabilities, SynapseML-based AI/ML predictions, and KQL-based, real-time intelligence on the current and up-to-date data in MongoDB Atlas.

The insert, update, and delete events occurring on the source MongoDB Atlas collection will be mirrored to the target OneLake table in near real-time. The conversion of the format to Parquet, data type conversions, and handling schema changes is all handled by mirroring. Mirroring generates a [SQL analytics endpoint](https://learn.microsoft.com/en-us/fabric/data-warehouse/get-started-lakehouse-sql-analytics-endpoint) to enable SQL analytics using T-SQL on the mirrored data tables in a read-only fashion. It also generates a [default semantic model](https://learn.microsoft.com/en-us/fabric/data-warehouse/semantic-models) to build Microsoft Power BI reports and dashboards out of the mirrored OneLake tables. 

The following diagram depicts the mirroring integration architecture. 

![MongoDB Atlas mirroring in Fabric][1]

## MongoDB Atlas Mirroring implementation

Our implementation of mirroring utilizes the mirroring open extensibility platform in Fabric and is intended to benefit customers who have been looking to achieve this functionality since it was pre-announced at Microsoft Ignite two years back. The mirroring open extensibility platform, announced at Ignite this year, provides a set of APIs to be used to create a MirrorDB. Once a Landing Zone is created within the MirrorDB in OneLake, then all the Parquet files pushed to the Landing Zone will be replicated into the corresponding MirrorDB tables.

Microsoft has published a set of APIs and formats for any third-party developer, partner, customer, or ISV to push change data to the Landing Zone. 

The solution requires a one-time execution of a simple Python script that invokes the Microsoft Fabric mirroring APIs from the mirroring open extensibility platform to create the MirrorDB for MongoDB Atlas. 

Once the Landing Zone in MirrorDB is created, you click the **Deploy to Azure** button, available at the bottom in the provided GitHub repo, as shown below.

![MongoDB Fabric Replication][2]

It will take you to an ARM template configuration screen in your Azure tenant. Once the required details are input and the **Create** button is selected, it will create an **App** **service**, deploy the Python app, and start the mirroring application. The mirroring application will trigger **Initial Sync** for the one-time historical data ingestion into OneLake and **Listening** which then keeps replicating changes in the source MongoDB Atlas collections to the target tables in the MirrorDB. 

The solution also can also be deployed in a VM in your Azure tenant or any bare metal server. If deployed in a VM, the Azure Vnet of the VM can be peered or private linked with MongoDB Atlas for secure communication. 

The following diagram depicts how the app service powered by Python script enables mirroring. 

![MongoDB Atlas mirroring Solution][3]

Refer to the [short video](https://youtu.be/SHsnhax9ISo) for the MongoDB Atlas to OneLake mirroring solution explanation.

## MongoDB Atlas mirroring use cases 

Mirroring will benefit multiple industries and their use cases where near real-time analytics are required. The mirrored MongoDB database in OneLake comes with a SQL analytics endpoint which simplifies the data warehousing experience by using T-SQL or the visual query editor. This can be used to combine data from other lakehouses and warehouses in Fabric for a holistic enterprise data-estate analysis. The semantic model can help build reports and dashboards using Power BI and the mirrored data. These reports and outputs can help take responsive actions in near real-time, especially in highly regulated industries. 

The mirrored data can be used in near real-time, ML-based analytics like credit scoring, fraud detection in the financial services industry, for dynamic pricing, inventory management in retail industry, or for predictive maintenance in the manufacturing industry.

## Conclusion

Mirroring for MongoDB Atlas will enable you to move your data from MongoDB Atlas collections to OneLake without any complex setup or ETL process involved. This near real-time data synchronization into the Delta format allows the mirrored data to be utilized by the variety of Fabric tools to gather accurate and meaningful insights.

[Get started with mirroring](https://github.com/mongodb-partners/MongoDB_Fabric_Mirroring) for your MongoDB Atlas data.


  [1]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt78131f4073929380/673bc3e634c039710e0242f7/image1.png
  [2]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt3668fe516d79c962/673bc4158581943ff80e099e/image3.png
  [3]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt883b92a85b788816/673bc434b35d667cc66d8f92/image2.png

Near Real-time Analytics Powered by Mirroring in Microsoft Fabric for MongoDB Atlas

Venkatesh Shanbhag

Partner Solution Architect

Many enterprise customers using MongoDB Atlas as their core operational database also use BigQuery for their Batch and AI/ML based analytics, making it pivotal for seamless transfer of data between these entities. Since the announcement of the [Dataflow templates](https://cloud.google.com/blog/products/data-analytics/mongodb-atlas-and-bigquery-dataflow-templates) (in October Of 2022) on moving data between MongoDB and BigQuery, we have seen a lot of interest from customers as it made it effortless for an append-only, one-to-one migration of data. Though the three Dataflow templates provided cater to most of the common use cases, there was also a demand to be able to do transformations as part of these templates.

We are excited to announce the addition of the ability to write your own user-defined functions (UDFs) in these Dataflow pipelines! This new feature allows you to use UDFs in JavaScript to transform and analyze data within BigQuery. With UDFs, you can define custom logic and business rules that can be applied to your data as it is being processed by Dataflow. This allows you to perform complex transformations like transforming fields, concatenating fields, deleting fields, converting embedded documents to separate documents, etc. These UDFs take unprocessed documents as input parameters and return the processed documents as output.

To use UDFs with BigQuery Dataflow, simply write your JavaScript function and store it in the Google cloud storage bucket. Use the Dataflow templates’ optional parameter to read these UDFs while running the templates. The function will be executed on the data as it is being processed, allowing you to apply custom logic and transformations to your data during the transfer.

## How to set it up
Let’s have a quick look at how to set up a sample UDF to process (transform a field, flatten an embedded document, and delete a field) from an input document before writing the processed data to BigQuery.

### Set up MongoDB 

1. MongoDB Atlas setup through [registration](https://mongodb.prakticum-team.ru/cloud/atlas/register).
2. MongoDB Atlas setup through [GCP Marketplace](https://console.cloud.google.com/marketplace/product/mongodb/mdb-atlas-self-service?utm_source=mongodb_devrel&utm_medium=static_site&utm_campaign=devrel&utm_id=mongodb). (MongoDB Atlas is available pay as you go in the GC marketplace).
3. Create your MongoDB [cluster](https://mongodb.prakticum-team.ru/docs/atlas/tutorial/create-new-cluster/).
4. Click on **Browse collections** and click on **+Create Database**.

![MongoDB Atlas cluster landing page](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/mongodb_atlas_cluster_landing_page_ad369de42d.png)

5: Name your database **Sample_Company** and collection **Sample_Employee**.
![MongoDB Atlas create collection step](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/create_database_30d4e827a2.png)
6: Click on **INSERT DOCUMENT**.
![Insert document to MongoDB collection from Atlas UI](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/insert_document_2e7d4f16ef.png)
Copy and paste the below document and click on **Insert**.
```
{
  "Name":"Venkatesh",
  "Address":{"Phone":{"$numberLong":"123455"},"City":"Honnavar"},
  "Department":"Solutions Consulting",
  "Direct_reporting": "PS"
}
```
7: To have authenticated access on the MongoDB Sandbox cluster from Google console, we need to create database users.

Click on the **Database Access** from the left pane on the Atlas Dashboard. 

Choose to **Add New User** using the green button on the left. Enter the username `appUser` and password `appUser123`. We will use built-in roles; click **Add Default Privileges** and in the **Default Privileges** section, add the roles readWriteAnyDatabase. Then press the green **Add User** button to create the user.

![Create a database user on MongoDB Atlas](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/create_database_user_774995251a.png)

8: Whitelist the IPs.

For the purpose of this demo, we will allow access from any ip, i.e 0.0.0.0/0. However, this is not recommended for a production setup, where the recommendation will be to use VPC Peering and private IPs.

![IP whitelisting on MongoDB Atlas to allow external access](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/ip_whitelisting_f5bc31c2bb.png)

![Whitelist 0.0.0.0/0](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/whitelisting_c0f1df1e03.png)

### Set up Google Cloud

1. Create a cloud storage bucket.
2. On your local machine, create a Javascript file **transform.js** and add below sample code.

```
function transform(inputDoc) {
   var outputDoc = new Object();
   inputDoc["City"] = inputDoc["Address"]["City"];
   delete doc.Address;
   outputDoc = doc;
   return returnObj;
}
```

This function will read the document read from MongoDB using the Apache beam MongoDB IO connector. Flatten the embedded document Address/City to City. Delete the Address field and return the updated document.

3: [Upload](https://cloud.google.com/storage/docs/uploading-objects) the javascript file to the Google Cloud storage bucket.

4: [Create](https://cloud.google.com/bigquery/docs/datasets#create-dataset) a BigQuery Dataset in your project in the region close to your physical location.

5: Create a [Dataflow pipeline](https://console.cloud.google.com/dataflow).

a. Click on the **Create Job from the template** button at the top.

b. Job Name: **mongodb-udf**.

c. Region: Same as your BigQuery dataset region.

d. MongoDB connection URI: Copy the connection URI for connecting applications from MongoDB Atlas.

e. MongoDB Database: **Sample_Company**.

f. MongoDB Collection: **Sample_Employee**.

g. BigQuery Destination Table: Copy the destination table link from the BigQuery

h. Dataset details page in format: bigquery-project:**sample_dataset.sample_company**.

i. User Option: **FLATTEN**.

j. Click on **show optional parameters**.

k. Cloud storage location of your Javascript UDF: Browse your UDF file loaded to bucket location. This is the new feature that allows running the UDF and applies the transformations before inserting into BigQuery.

l. Name of your Javascript function: **transform**.

![Create Dataflow Job with UDF on Google Cloud console](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/create_dataflow_job_cfdbd2357e.png)

6: Click on **RUN JOB** to start running the pipeline. Once the pipeline finishes running,  your graph should show **Succeeded** on each stage as shown below.

![Dataflow Job Graph view on Google Cloud console](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/dataflow_job_graph_ff48458c96.png)

7: After completion of the job, you will be able to see the transformed document inserted into BigQuery.

![Document read from MongoDB flattened and loaded to BigQuery](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/document_read_ed7e24b805.png)

## Conclusion
In this blog, we introduced UDFs to MongoDB to BigQuery Dataflow templates and their capabilities to transform the documents read from MongoDB using custom user defined Javascript functions stored on Google Cloud storage buckets. This blog also includes a simple tutorial on how to set up MongoDB Atlas, Google Cloud, and the UDFs.

### Further reading

* A data pipeline for [MongoDB Atlas and BigQuery using Dataflow](https://cloud.google.com/blog/products/data-analytics/mongodb-atlas-and-bigquery-dataflow-templates).
* A data pipeline for [MongoDB Atlas and BigQuery using the Confluent](https://mongodb.prakticum-team.ru/developer/products/atlas/mongodb-bigquery-pipeline-using-confluent/) connector.
* Run analytics using BigQuery using [BigQuery ML](https://cloud.google.com/bigquery-ml/docs).
* [Set up your first MongoDB cluster](https://mongodb.prakticum-team.ru/docs/atlas/billing/gcp-self-serve-marketplace/) using Google Marketplace.


Learn how to transform the MongoDB Documents using user-defined JavaScript functions in Dataflow templates.

UDF Announcement for MongoDB to BigQuery Dataflow Templates

To win in this competitive digital economy, enterprises are striving to create smarter intelligent apps. These apps provide a superior customer experience and can derive insights and predictions in real-time. 

Smarter apps use data — in fact, lots of data, AI and analytics together. [MongoDB Atlas](https://mongodb.prakticum-team.ru/cloud/atlas/lp/try4) stores valuable operational data and has capabilities to support operational analytics and AI based applications. This blog details MongoDB Atlas’ seamless integration with [Microsoft Fabric](https://www.microsoft.com/en-us/microsoft-fabric) to run large scale AI/ML and varied analytics and BI reports across the enterprise data estate, reshaping how teams work with data by bringing everyone together on a single, AI-powered platform built for the era of AI. Customers can leverage MongoDB Atlas with Microsoft Fabric as the foundation to build smart and intelligent applications.

## Better together 

MongoDB was showcased as a key partner at Microsoft Ignite, highlighting the collaboration to build seamless integrations and joint solutions complementing capabilities to address diverse use cases. 

![Satya Nadella at Microsoft Ignite][1]

During the first [Keynote at Ignite](https://ignite.microsoft.com/en-US/sessions/bff678f5-f533-4ad0-9905-6ac35c9704ad?source=sessions), Satya Nadella, Chairman and Chief Executive Officer of Microsoft, announced that Microsoft Fabric is now generally available for purchase. Satya addressed the strategic plan to enable MongoDB Atlas mirroring in Microsoft Fabric to enable our customers to use mirroring to access their data in OneLake. 

MongoDB Atlas’ flexible data model, versatile query engine, integration with LLM frameworks, and inbuilt [Vector Search](https://mongodb.prakticum-team.ru/products/platform/atlas-vector-search), [analytical nodes](https://mongodb.prakticum-team.ru/docs/atlas/cluster-config/multi-cloud-distribution/#analytics-nodes-for-workload-isolation), [aggregation framework](https://mongodb.prakticum-team.ru/docs/manual/aggregation/), [Atlas Data Lake](https://mongodb.prakticum-team.ru/atlas/data-lake), [Atlas Data Federation](https://mongodb.prakticum-team.ru/cloud/atlas/lp/data-federation), [Charts](https://mongodb.prakticum-team.ru/products/charts), etc. enables operational analytics and application-driven intelligence from the source of the data itself. However, the analytics and AI needs of an enterprise span across their data estate and require them to combine multiple data sources and run multiple types of analytics like big data, Spark, SQL, or KQL-based ones at a large-scale. They bring data from sources like MongoDB Atlas to one uniform format in [OneLake](https://learn.microsoft.com/en-us/fabric/onelake/onelake-overview) in Microsoft Fabric to enable them to run Batch Spark analytics and AI/ML of petabyte scale and use data warehousing abilities, big data analytics, and real-time analytics across the delta tables populated from disparate sources. 

![MongoDB Atlas Developer Data Platform][2]

Let's discuss the integration mechanisms between MongoDB Atlas and Microsoft Fabric for both batch and real-time scenarios. 

## MongoDB integration mechanisms for Microsoft Fabric

MongoDB Atlas serves as the operational data layer (ODL) of many enterprise applications. Atlas as an ODL stores data from internal applications, customer-facing services, and third-party APIs from multiple channels. By using Microsoft Fabric pipelines, you can combine MongoDB Atlas data with relational data from other traditional applications and unstructured data from sources like logs, clickstreams, etc.

### Batch/microbatches

There are multiple options to bring MongoDB Atlas data into Microsoft Fabric in batch mode one time or bring it in micro batches which runs on a specified frequency. In this section, we will discuss the out-of-the-box (OOTB) approaches that can be applied to fetch data from MongoDB to Microsoft Fabric in batch mode. 

#### MongoDB Atlas SQL Dataflow Gen2 connector

The [MongoDB Atlas SQL connector ](https://learn.microsoft.com/en-us/power-query/connectors/mongodb-atlas-sql-interface)is a Microsoft-certified connector which can be accessed from the “[Dataflow Gen2](https://learn.microsoft.com/en-us/fabric/data-factory/create-first-dataflow-gen2)” feature from “[Data Factory](https://learn.microsoft.com/en-us/fabric/data-factory/data-factory-overview)” in Microsoft Fabric.

Dataflow Gen2 selection takes us to the familiar Power Query interface of Microsoft Power BI. To bring data from MongoDB Atlas collections, search the MongoDB Atlas SQL connector from the “Get Data” option on the menu.

![Dataflow Gen2][3]

Providing the connection URI and authentication details, we can set up a connection to MongoDB Atlas. Note that the connection string is not that of a normal connection but that of Atlas SQL or a federated database. Learn more about [connecting using the Atlas SQL](https://mongodb.prakticum-team.ru/docs/atlas/data-federation/query/sql/powerbi/connect/) or [set up an Atlas federated database](https://mongodb.prakticum-team.ru/docs/atlas/data-federation/tutorial/getting-started/) and get a connection string for the same. Also, note that the connector needs a Gateway set up to communicate from Fabric and schedule refreshes. Get more details on [Gateway setup](https://powerbi.microsoft.com/en-in/gateway/).

Once data is retrieved from MongoDB Atlas into Power Query, the magic of Power Query can be used to transform the data, including flattening object data into separate columns, unwinding array data into separate rows, or changing data types. These are typically required when converting MongoDB data in JSON format to the relational format in Power BI. Additionally, the blank query option can be used for a quick query execution. Below is a sample query to start with:

```
let
  Source = MongoDBAtlasODBC.Query("<Atlas SQL connection string>", “<DB name>", "select * from <collection name>", null)
in
  Source
```

####  MongoDB Data Pipeline connector (preview)

The [announcement at Microsoft Ignite](https://www.microsoft.com/en-us/microsoft-fabric/blog/) of the Data Pipeline connector being released for MongoDB Atlas in Microsoft Fabric is definitely good news for MongoDB customers. The connector provides a quick and similar experience as the [MongoDB connector in Data Factory and Synapse Pipelines](https://learn.microsoft.com/en-us/azure/data-factory/connector-mongodb?tabs=data-factory). 

The connector is accessed from the “[Data Pipelines](https://learn.microsoft.com/en-us/fabric/data-factory/create-first-pipeline-with-sample-data)” feature from “[Data Factory](https://learn.microsoft.com/en-us/fabric/data-factory/data-factory-overview)” in Fabric. Choose the “Copy data” activity to use the MongoDB connector to get data from MongoDB or to push data to MongoDB. To get data from MongoDB, add MongoDB in Source. Select the MongoDB connector and create a linked service by providing the **connection string** and the **database** to connect to in MongoDB Atlas.

![MongoDB Data Pipeline Connector][4]

We can run the pipeline by setting up Lakehouse in Microsoft Fabric as the Sink to receive the data from MongoDB Atlas.

![Microsoft Fabric Data Details][5]

Thus, the connector provides an easy mechanism to copy data between MongoDB Atlas and Microsoft Fabric. Similar to the MongoDB connector in Microsoft Azure Data Factory, it is available for both on-prem and self-hosted MongoDB instances and MongoDB Atlas. As this is the first release of the connector, it has few limitations when compared to the MongoDB connector in Azure Data Factory or Synapse Pipelines. 

#### MongoDB mirroring in Microsoft Fabric

As mentioned earlier, Satya Nadella, CEO of Microsoft, announced the strategic plan to enable MongoDB Atlas mirroring in Microsoft Fabric. This will provide the easiest and fastest approach to replicate the data from MongoDB Atlas to OneLake in Microsoft Fabric. However, this will provide the fewest options to do data transformations while replicating the data. The availability of this feature is expected to be coming in CY24.

![Microsoft Fabric Compute Engines][6]

### Real-time sync

Bringing data using the PULL mechanism of batch mode serves multiple use-cases, including the near real-time ones. However, in certain cases, we may need a real-time sync mechanism, which will replicate the data changes from MongoDB Atlas into Microsoft Fabric in real-time. This has to be a Change Data Capture method, a PUSH mechanism from MongoDB Atlas to Microsoft Fabric. In this section, we will detail two such custom approaches that can be leveraged for these CDC-based, real-time sync use cases.

#### Atlas triggers and functions

Real-time sync from MongoDB Atlas to Microsoft Fabric can be achieved by using [MongoDB Atlas Triggers](https://mongodb.prakticum-team.ru/docs/atlas/app-services/triggers/database-triggers/) to capture the change events in a MongoDB collection and using an [Atlas function](https://mongodb.prakticum-team.ru/docs/atlas/app-services/functions/) to trigger an Azure function. The Azure function can directly write to the Lake House in Microsoft Fabric or to ADLS Gen2 storage using [ADLS Gen2 APIs](https://learn.microsoft.com/en-us/rest/api/storageservices/data-lake-storage-gen2). ADLS Gen2 storage accounts can be referenced in Microsoft Fabric using [shortcuts](https://learn.microsoft.com/en-us/fabric/onelake/onelake-shortcuts), eliminating the need for an ETL process to move data from ADLS Gen2 to OneLake. Data in Microsoft Fabric can be accessed using the existing ADLS Gen2 APIs but there are some changes and constraints which can be referred to in the [Microsoft Fabric documentation](https://learn.microsoft.com/en-us/fabric/onelake/onelake-access-api). 

![Atlas Triggers and Functions with Azure][7]

#### MongoDB Spark streaming

MongoDB’s [Spark connector v10.1](https://mongodb.prakticum-team.ru/blog/post/introducing-mongodb-spark-connector-version-10-1) provides [streaming capabilities](https://mongodb.prakticum-team.ru/docs/spark-connector/current/structured-streaming/) which allows structured streaming of changes from MongoDB or to MongoDB in both continuous and micro-batch modes. Using the connector, we just need a simple code that reads a stream of changes from the MongoDB collection and writes the stream to the Lakehouse in Microsoft Fabric or to ADLS Gen2 storage which can be referenced in Microsoft Fabric using shortcuts. MongoDB Atlas can be set up as a source for structured streaming by referring to the [MongoDB documentation](https://mongodb.prakticum-team.ru/docs/spark-connector/current/structured-streaming/#configuring-a-read-stream-from-mongodb). Refer to the [Microsoft Fabric documentation](https://learn.microsoft.com/en-us/fabric/data-engineering/lakehouse-streaming-data) on setting up Lakehouse as Sink for structured streaming. 

![Microsoft Fabric Data Pipeline with MongoDB Atlas][8]

Watch a full demo of MongoDB Atlas integration with Microsoft Fabric:

:youtube[]{vid=0-jTG9Y6nyQ}

## Conclusion

AI is set to reshape businesses and provide innovative solutions across industries. Smart applications can be built with the right database solution combined with the right analytics and AI solution. These applications can thrive, delivering accurate, context-aware, dynamic, data-driven user experiences that meet the growing demands of today's fast-paced digital landscape. MongoDB Atlas data and its operational analytics, when combined with the entire enterprise data estate in Microsoft Fabric to run multiple analytics and reporting at a large scale, is a powerful combination to build such smart applications.

Learn more about [MongoDB Atlas on Azure](https://azure.microsoft.com/en-us/solutions/mongodb/) and get started for free today on [Azure Marketplace](https://azuremarketplace.microsoft.com/en-us/marketplace/apps/mongodb.mongodb_atlas_self_serve_prod_2022?ocid=msft_ignite_fabricblog_FY24). 


  [1]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltc783c7ca51ffc321/655678560e64b945e26edeb7/Fabric_Keynote.png
  [2]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt11079cade4dbe467/6553ef5253e8ec0e05c46baa/image2.png
  [3]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt133326ec100a6ccd/6553ef7a9984b8c9045a9fc6/image5.png
  [4]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltd2c2a8c4741c1849/6553efa09984b8a1685a9fca/image6.png
  [5]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltc1faa0c6b3e2a93d/6553f00253e8ecacacc46bb4/image3.png
  [6]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt4789acc89ff7d1ef/6553f021647c28121d4e84f6/image7.png
  [7]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt58b3b258cfeb5614/6553f0410e64b9dbad6ece06/image1.png
  [8]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltba161d80a4442dd0/6553f06ac2479d218b7822e0/image4.png

Learn how you can use Microsoft Fabric with MongoDB Atlas for intelligent analytics for your data. 

Diana Annie Jenosh

Content by Diana Annie Jenosh

RAG Made Easy with MongoDB Atlas and Azure OpenAI

Near Real-time Analytics Powered by Mirroring in Microsoft Fabric for MongoDB Atlas

UDF Announcement for MongoDB to BigQuery Dataflow Templates

Unlock the Value of Data in MongoDB Atlas with the Intelligent Analytics of Microsoft Fabric