1Data - PeerIslands Data Sync Accelerator

Today’s enterprises are in the midst of digital transformation, but they’re hampered by monolithic, on-prem legacy applications that don’t have the speed, agility, and responsiveness required for digital applications. To make the transition, enterprises are migrating to the cloud. MongoDB has partnered with PeerIslands to develop 1Data, a reference architecture and solution accelerator that helps users with their cloud modernization.

This post details the challenges enterprises face with legacy systems and walks through how working with 1Data helps organizations expedite cloud adoption.

Modernization Trends

As legacy systems become unwieldy, enterprises are breaking them down into microservices and adopting cloud native application development. Monolith-to-microservices migration is complex, but provides value across multiple dimensions. These include:

  • Development velocity
  • Scalability
  • Cost-of-change reduction
  • Ability to build multiple microservice databases concurrently

One common approach for teams adopting and building out microservices is to use domain driven design to break down the overall business domain into bounded contexts first. They also often use the Strangler Fig pattern to reduce the overall risk, migrate incrementally, and then decommission the monolith once all required functionality is migrated. While most teams find this approach works well for the application code, it’s particularly challenging to break down monolithic databases into databases that meet the specific needs of each microservice. There are several factors to consider during transition:

  1. Duration. How long will the transition to microservices take?
  2. Data synchronization. How much and what types of data need to be synchronized between monolith and microservice databases?
  3. Data translation in a heterogeneous schema environment. How are the same data elements processed and stored differently?
  4. Synchronization cadence. How much data needs syncing, and how often (real-time, nightly, etc.)?
  5. Data anti-corruption layer. How do you ensure the integrity of transaction data, and prevent the new data from corrupting the old?

Simplifying Migration to the Cloud

Created by PeerIslands and MongoDB, 1Data helps enterprises address the challenges detailed above.

Migrate and synchronize your data with confidence with 1Data

  1. Schema migration tool. Convert legacy DB schema and related components automatically to your target MongoDB instance. Use the GUI-based data mapper to track errors.
  2. Real-time data sync pipeline. Sync data between monolith and microservice databases nearly in real time with enterprise grade components.
  3. Conditional data sync. Define how to slice the data you’re planning to sync.
  4. Data cleansing. Translate data as it’s moved.
  5. DSLs for data transformation. Apply domain-specific business rules for the MongoDB documents you want to create from your various aggregated source system tables. This layer also acts as an anti-corruption layer.
  6. Data auditing. Independently verify data sync between your source and target systems.
  7. Go beyond the database. Synchronize data from APIs, Webhooks & Events.
  8. Bidirectional data sync. Replicate key microservice database updates back to the monolithic database as needed.

Get Started with Real-Time Data Synchronization

With the initial version of 1Data, PeerIslands addresses the core functionality of real-time data sync between source and target systems.

Here’s a view of the logical architecture:

Real time data sync architecture chart

Source System. The source system can be a relational database like Oracle, where we’ll rely on CDC, or other sources like Events, API, or Webhooks.

**Data Capture & Streaming.**Captures the required data from the source system and converts them into data streams using either off-the-shelf DB connectors or custom connectors, depending on the source type. 1Data implements data sharding and throttling, which enable data synchronization at scale, in this phase.

Data Transformation. The core of the accelerator, when we convert the source data streams into target MongoDB document schemas. We use LISP-based Domain Specific Language to enable simple, rule-based data transformation, including user-defined rules.

Data Sink & Streaming. Captures the data streams that need to be updated into the MongoDB database through stream consumers. The actual update into the target DB is done through sink connectors.

Target system. The MDB database used by the microservices.

Auditing. Most data that gets migrated is enterprise-critical; 1Data audits the entire data synchronization process for missed data and incorrect updates.

Two-way sync. The logical architecture enables data synchronization from the MongoDB database back to the source database.

We used MongoDB, Confluent Kafka and Debezium to implement this initial version of 1Data:

1Data Architecture Chart

The technical architecture is cloud agnostic, and can be deployed on-prem as well. We’ll be customizing it for key cloud platforms as well as fleshing out specific architectures to adopt for common data sync scenarios.

Conclusion

The 1Data solution accelerator lends itself to multiple use cases, from single view to legacy modernization.

Please reach out to us for technical details and implementation assistance, and watch this space as we develop the 1Data accelerator further.

← Previous

Announcing Azure Private Link Integration for MongoDB Atlas

We’re excited to announce the general availability of Azure Private Link as a new network access management option in MongoDB Atlas . MongoDB Atlas is built to be secure by default . All dedicated Azure clusters on Atlas are deployed in their own VNET. For network security controls, you already have the options of an IP Access List and VNET Peering . The IP Access List in Atlas offers a straightforward and secure connection mechanism, and all traffic is encrypted with end-to-end TLS. But it requires that you provide static public IPs for your application servers to connect to Atlas, and to list all such IPs in the Access List. And if your applications don’t have static public IPs or if you have strict requirements on outbound database access via public IPs, this won’t work for you. The existing solution to this is VNET Peering, with which you configure a secure peering connection between your Atlas cluster’s VNET and your own VNET(s). This is easy, but the connections are two way. While Atlas never has to initiate connections to your environment, some customers perceive VNET peering as extending the perceived network trust boundary anyway. Although Access Control Lists (ACLs) and security groups can control this access, they require additional configuration. MongoDB Atlas and Azure Private Link Now, you can use Azure Private Link to connect a VNET to MongoDB Atlas. This brings two major advantages: Unidirectional: connections via Private Link use a private IP within the customer’s VNET, and are unidirectional such that the Atlas VNET cannot initiate connections back to the customer's VNET. Hence, there is no extension of the network trust boundary. Transitive: connections to the Private Link private IPs within the customer’s VNET can come transitively from another VNET peered to the Private Link-enabled VNET, or from an on-prem data center connected with ExpressRoute to the Private Link-enabled VNET. This means that customers can connect directly from their on-prem data centers to Atlas without using public IP Access Lists. Azure PrivateLink offers a one-way network peering service between an Azure VNET and a MongoDB Atlas VNET Meeting Security Requirements with Atlas on Azure Azure Private Link adds to the security capabilities that are already available in MongoDB Atlas, like Client Side Field-Level Encryption , database auditing , BYO key encryption with Azure Key Vault integration , federated identity , and more. MongoDB Atlas undergoes independent verification of security and compliance controls , so you can be confident in using Atlas on Azure for your most critical workloads. Ready to try it out? Get started with MongoDB Atlas today! Sign up now

October 15, 2020
Next →

Customer Service Expert Wati.io Scales Up on MongoDB

Wati.io is a software-as-a-service (SaaS) platform that empowers businesses to develop conversation-driven strategies to boost growth. Founded by CEO Ken Yeung in 2019, Wati started as a chatbot solution for large enterprises, such as banks and insurance companies. However, over time, Yeung and his team noticed a growing need among small and medium-sized businesses (SMBs) to manage customer conversations more effectively. To address this need, Wati used MongoDB Atlas and built a solution based on the WhatsApp Business API. It enables businesses to manage and personalize conversations with customers, automate responses, improve commerce functions, and enhance customer engagement. Speaking at MongoDB.local Hong Kong in September 2024, Yeung said, “The current solutions on the market today are not good enough. Especially for SMBs [that] don’t have the same level of resources as enterprises to deal with the number of conversations and messages that need to be handled every day.” Supporting scale: From MongoDB Community Edition to MongoDB Atlas “From the beginning, we relied on MongoDB to handle high volumes of messaging data and enable businesses to manage and scale their customer interactions efficiently,” said Yeung. Wati originally used MongoDB Community Edition , as the company saw the benefits of a NoSQL model from the beginning. As the company grew, it realized it needed a scalable infrastructure, so Wati transitioned to MongoDB Atlas. “When we started reaching the 2 billion record threshold, we started having some issues. Our system slowed down, and we were not able to scale it,” said Yeung. Atlas has now become an essential part of Wati’s infrastructure, helping the company store and process millions of messages each month for over 10,000 customers in 165 countries. “Transitioning to a new platform—MongoDB Atlas—seamlessly was critical because our messaging system needs to be on 24/7,” said Yeung. Wati collaborated closely with the MongoDB Professional Services and MongoDB Support teams, and in a few months it was able to rearchitect the deployment and data model for future growth and demand. The work included optimizing Wati’s database by breaking it down into clusters. Wati then focused on extracting connections, such as conversations, and dividing and categorizing data within the clusters—for example, qualifying data as cold or hot based on the read and write frequencies. This architecture underpins the platform’s core features, including automated customer engagement, lead qualification, and sales management. Deepening search capabilities with MongoDB Atlas Search For Wati’s customers, the ability to search through conversation histories and company documents to retrieve valuable information is a key function. This often requires searching through millions of records to rapidly find answers so that they can respond to customers in real-time. By using MongoDB Atlas Search , Wati improved its search capabilities, ultimately helping its business customers perform more advanced analytics and improve their customer service agents’ efficiency and customer reporting. “[MongoDB] Atlas Search is really helpful because we don’t have to do a lot of technical integration, and minimal programming is required,” said Yeung. Looking ahead: Using AI and integrating more channels Wati expects to continue collaborating with MongoDB to add more features to its platform and keep innovating at speed. The company is currently exploring to build more AI capabilities of Wati KnowBot , as well as how it can expand its integration with other conversation platforms and channels such as Instagram and Facebook. To learn more about MongoDB Atlas, visit our product page . To get started with MongoDB Atlas Search, visit the Atlas Search product page .

November 25, 2024