Empower Modern App Developers with Document Databases

Andrew Davidson
June 5, 2023

Across industries, business success depends on a company’s ability to deliver new digital experiences through software. The speed at which a company can develop and deploy a new application with innovative features is a direct lever on business outcomes. Given the vital role developers play in the success of your business, it stands to reason that equipping them with the tools to maximize their productivity is in your best interest.

Unfortunately, many organizations are unaware of the tax they’re placing on their development teams by using a relational database. While the relational database has been a bedrock for data-driven applications for 50 years, it was developed in an era before the internet and is a poor fit as the foundation of today’s web and mobile applications.

Document databases, which have emerged over the past decade, have cemented themselves as the most popular and widely used alternative to the tabular model found in traditional relational databases. Document databases have become so powerful that even relational databases are trying to emulate them. Built around JavaScript Object Notation (JSON)–like documents, document databases are intuitive for developers to use. Instead of the rigid row-and-column structure of the relational model, document databases map documents directly to objects in code, which is how coders naturally think of and work with data.

Let’s break down the key advantages to document databases in building modern applications. We’ll see why the document model’s flexibility eliminates the complex intergroup dependencies that have traditionally slowed developers down.

The limitations of the relational database model

Relational databases add complexity to a developer’s workload, severely hampering the velocity of work. The rigid row-and-column structure creates a mismatch between the way developers think of code and data, and how they need to store it. Additionally, while the relational model was fine in an age when most applications used a small pre-set of attributes such as last names, ZIP codes, and state abbreviations, the majority of data collected by organizations today is rich in structure. We have given names and sometimes preferred names. We have unique attributes that are relevant to only some of us: For example, people with PhDs have dissertation topics, sports lovers have favorite sports, and our families come in every conceivable shape and size. This richly structured data reflects how we actually think about the real world, and it’s very difficult to flatten, store, analyze, or query using rows and columns.

With relational databases, developers can feel stuck in quicksand with changes to their applications requiring them to carefully collaborate with experts like database administrators (DBAs) who help them translate their schemas and queries to underlying relational data models to ensure that indexing strategies are appropriately employed. The layer of indirection increases cognitive load, is hard to reason about, and slows everything down. More than half of application changes require database schema modifications. Those database modifications take longer to complete than the application changes they are designed to support. You can quickly see why these complicated efforts severely slow the delivery of new software features into production.

Enabling development of modern apps with document databases

With the birth of the internet and the proliferation of mobile and web apps, developers’ roles evolved. The emergence of robust development frameworks, which abstracted away underlying complexity, and the rise of DevOps led organizations to consolidate developer functions. The new generation of full-stack developers wanted databases that better addressed their applications’ requirements and their ways of working with data.

The founders of MongoDB recognized a need for a modern database solution while at the adtech giant DoubleClick in 2007. They still were unable to scale to the 400,000 transactions per second the business required due to the constraints of the relational model. These challenges inspired them to create a new, modern, general-purpose database. This database could address the shortcomings of the relational data model and offer a solution that developers actually wanted to use. The result was a horizontally scalable, document-based NoSQL database called MongoDB. The document database model in general, and MongoDB more specifically, addresses the limitations of relational databases in several notable ways:

Intuitive data model: The documents at the center of document databases have a universal data format. JSON is a lightweight, language-independent, and human-readable format that has become a widely used standard for storing and exchanging data. These documents map directly to data structures in popular programming languages so there is no need for the additional mapping layer often used with relational databases. Because data that is accessed together is stored together, there is less code to write; developers don’t need to decompose data across tables or run joins.

Flexible schema: These JSON-like documents are flexible. Each document can have its own fields, and there’s no need to pre-define the schema in the database. It can be modified at any time. That flexibility enhances developer agility.

Meeting user expectations while simplifying application architectures

The most innovative applications we use in our daily lives — think Netflix and Instagram — have raised user expectations for what every application should be. Today we expect applications to be:

Highly responsive
Able to deliver relevant information
Optimized for mobile devices
Secure
Powered by real-time insights
Continuously improved

Meeting those expectations can be extremely challenging, especially for developers using relational databases. A typical data infrastructure built around a legacy relational database can trap your development team in overly complex and siloed architectures. Document databases, on the other hand, can simplify application architectures. Documents are a superset of all other data models, so developers can store and work with a variety of data types. Development teams can accommodate most of their use cases in a single data model and database

The document data model can help your developers overcome the limitations of the relational model while improving their productivity and velocity. Allowing them to minimize the undifferentiated work of maintaining their infrastructure and to focus on meeting demanding user expectations.

As a result, they can deliver better, more innovative applications faster than before.

Click here to read the original article published on The New Stack.

← Previous

The Guide to openEHR Schema Modeling with MongoDB

Disclosure: This article features valuable insights from MongoDB's experts, customers, or partners to further an understanding of MongoDB in specific industry use cases. Please note that while MongoDB does not validate the accuracy of the text and statements, this resource aims to provide practical knowledge for your reference. The openEHR specification is a widely used standard for storing and managing electronic health records (EHRs). It offers a structured way of organizing clinical data that makes it easy to query and analyze. However, implementing the schema modeling and querying of openEHR data can present unique challenges. In this blog post, we will explore the intricacies of openEHR schema modeling, some challenges, and outline potential solutions that we can implement using MongoDB. Understanding openEHR specification and interoperability standards Before diving into the complexities of openEHR schema modeling and querying, let’s first understand the openEHR specification and how it relates to other interoperability standards such as HL7 FHIR. These standards work together to enable seamless healthcare information exchange across various systems and applications. A clear understanding of these standards will provide a solid foundation for addressing the challenges we’ll encounter. openEHR (open Electronic Health Record) is an open-source standard for the representation and management of electronic health records (EHRs). It is designed to provide a flexible and interoperable framework for the collection, storage, retrieval, and exchange of health data, regardless of the system or application used to generate or consume it. Key building blocks of openEHR include: Archetypes: Archetypes are structured, reusable models that define the content and structure of clinical information. An example is a Vital Signs Archetype which defines the structure and constraints for capturing vital signs measurements. Properties may include elements such as temperature, heart rate, blood pressure, and oxygen saturation. Templates: Templates, on the other hand, are derived from archetypes and are used to define specific subsets of clinical information. An example is an Adult Vital Signs Template — a template derived from the Vital Signs Archetype and customized for adult patients. It includes a subset of vital sign elements specifically relevant to adult patients, such as temperature, heart rate, and blood pressure. Compositions: A composition is an instance of a template that contains actual clinical data. An example is Patient Encounter Composition — a composition that represents a patient encounter, containing various clinical measurements and data. Properties include sections for patient demographics, symptoms, diagnoses, procedures, medications, and vital signs. Within the composition, the vital signs section would follow the structure defined by the Adult Vital Signs Template, containing actual vital signs measurements for a specific encounter. Figure 1: openEHR - Archetype & template openEHR & HL7 FHIR While openEHR defines an “information model” for modeling and persisting data in EMR systems, FHIR is an open-source standard for healthcare information exchange across EMRs and other systems. The diagram below can help you visualize how these standards and technologies work together in healthcare systems. FIgure 2: openEHR, FHIR, & HL7 openEHR schema modeling challenges openEHR schema modeling poses several challenges due to its complex hierarchical structure and the need to handle diverse data types. Additionally, querying this data can be difficult due to the complex relationships between the different data elements. Many of the end user queries are at composition level and each composition is made up of hundreds of fields. This is typically overcome by creating multiple sets of indexes often resulting in performance bottlenecks. Let’s examine some of the typical query patterns and demonstrate how these pose challenges for schema modeling and querying: Complex Hierarchical Structure: The openEHR specification is built on a complex hierarchical structure that represents various healthcare concepts and their relationships. Translating this structure into a database schema can be challenging, as traditional relational databases may struggle to handle the dynamic and nested nature of openEHR data. Finding an efficient and flexible way to model this complex structure is essential for ensuring accurate representation and easy retrieval of data. Data Versioning and Evolution: openEHR supports the concept of versioning and evolution, allowing for changes and updates to healthcare records over time. Modeling and querying evolving data can be complex, as it requires maintaining the history of changes and accommodating different versions of the schema. Ensuring data consistency, efficient versioning, and the ability to query historical data are critical considerations in openEHR schema modeling. Performance and Scalability: Healthcare systems generate a vast amount of data, and efficient querying of openEHR records is crucial for timely analysis and decision-making. Designing a schema that allows for fast and scalable querying is a challenge, particularly when dealing with large datasets and complex query patterns. Optimizing query performance, indexing strategies, and data partitioning techniques are essential for ensuring a responsive and scalable system. Query Patterns: openEHR data is queried based on various patterns, such as retrieving patient records, searching for specific diagnoses, or aggregating data for statistical analysis. Each query pattern may have different performance requirements and may involve traversing complex relationships within the hierarchical structure. Designing an efficient schema that can handle these query patterns and provide fast and accurate results is a significant challenge in openEHR schema modeling. Learn more about how MongoDB works with any healthcare data standard in our whitepaper, What is Radical Interoperability Typical openEHR schema model As you see in the below hierarchical model, the complexity of the openEHR specification can make it challenging to model and query clinical data. In the subsequent sections, we will explore potential solutions for openEHR schema modeling, including the Attribute Pattern and the Flat Hierarchy Pattern. Figure 3: Part of Archetype data captured in JSON format Archetype query language Archetype Query Language (AQL) is a query language specifically designed for querying clinical data stored in openEHR-based electronic health record systems. It provides a standardized and powerful way to retrieve specific clinical information from structured data using archetypes and templates. AQL enables clinicians, researchers, and developers to express complex queries, filter data based on clinical criteria, and retrieve meaningful information for analysis and decision support. For example to “get the latest 5 abnormal blood pressure values that were recorded in a health encounter for a specific patient”, you can write an AQL as follows: As you can see above in the highly nested and hierarchical schema model, while being a flexible and extensible approach to representing clinical data, presents unique challenges in storage, retrieval, and performance. Even a small volume of 250k documents in this format requires a significant amount of storage space for handling. In addition, given the variable nature of nesting hierarchy, typical patterns for indexing are highly inefficient. We will explore possible solution options using MongoDB schema modeling patterns. Addressing openEHR schema modeling challenges with MongoDB MongoDB provides flexible and powerful features that can help address the challenges of openEHR schema modeling and querying. Attribute pattern with standard index One possible solution is to use the attribute pattern modeling style, simplifying the schema model and making it more predictable. This approach allows for efficient indexing, enabling fast retrieval of data. The query for our example scenario to “get the latest 5 abnormal blood pressure values that were recorded in a health encounter for a specific patient” can be simplified as follows. The above query is highly efficient as evaluated against a collection of 5 MM document size. Additionally, the storage is significantly improved taking 833 MB compressed storage space for 5 MM documents, as well as efficient retrieval using index. However the index size is significantly large for this pattern — about 3.5 GB for the 5 MM document size – which may be a blocker for larger data sets. Can we improve on this? What are our options? Learn more about how MongoDB works with any healthcare data standard in our whitepaper, What is Radical Interoperability Flat hierarchy model with wildcard search An alternative approach to addressing challenges discussed above is by leveraging a flattened document model and using wildcard index. The same openEHR schema model can be represented as follows: By using a flattened document data model, the complex hierarchical structure of openEHR schemas can be simplified for storage and retrieval efficiency. While this simplifies the schema model, there are a few considerations like the dimensions need to be moved to the application / configuration layer, like the body temperature is celsius, for example. In addition the query on the above document model is simplified as below. To make the query efficient, we create a wild-card index as follows. With the above index in place, the query is significantly more efficient as we can see in the explain plan below. When comparing the storage size to openEHR standard spec, the flat hierarchy spec model only takes 839 MB compressed storage for 5 MM documents. However, the wildcard index is significantly heavy. For the above volume of documents, the index size is only 705 MB. Can we improve upon this further? Flat hierarchy model with Atlas Search MongoDB Atlas Search brings the power of Apache Lucene to MQL. You can simplify the process of indexing and search significantly. You start by creating an Atlas Search index as follows: Atlas takes care of the various steps leading up to making a Lucene search index available. Once the index is in place, you can query using $search opearator. The query we have seen in above solution options can be constructed as follows. The data storage is identical to the flat hierarchy schema, with only the index built using Atlas search. The search index size is similar to the wildcard index. However the search index provides significantly greater functionalities and capabilities. Keep in mind that this capability is only available on cloud and with MongoDB Atlas . We have seen multiple solution options as detailed above. You can choose one that suits your application as well as infrastructure requirements. Design Validation Typical process of evaluating various possible schema models and their efficiencies will require multiple iterations including NFR validation on production scale of data volume. This helps us to: Better understand fit of MongoDB to the particular use case Understand MongoDB sizing expectations Demonstrate MongoDB performance Understand performance of specific queries Fine tune MongoDB schema for our needs Current tools only replicate a sample data or generate random data and are not suitable for the above purposes as the indexes get skewed and performance of queries is not close to real world. Completely random data also does not provide an accurate view of the MongoDB sizing that will be required. Test data generator Test data generator and Performance testing tools are a solution accelerator from PeerIslands that helps customers generate large volume, customizable, close-to-real-world test data with specific customer schemas. We have used the test data generator to generate a 5 Million documents data set for both attribute and flat hierarchy design patterns. The test data generator takes a configuration file and quickly generates large volumes of data, as shown in the flat hierarchy schema model below. Generating test data for the original openEHR spec requires a more complex configuration as the one below. Results and comparison We generated 5MM data items for each type of design pattern (attribute and flat hierarchy). For base openEHR spec, we generated 250k items given the size and time requirements. The figure below provides a comparison between the data and index sizes for each design approach: In addition the data and index sizes for Atlas Search is shown below for a 214k dataset. The attribute pattern performs significantly better than the base openEHR spec. However it is limited by its index size. The flat hierarchy data model performs the best overall with a wildcard index. When you can use cloud, you have the option of using Atlas Search. While the overall index size is not any better than the wildcard index on the flat hierarchy collection, the Lucene search index provides a significantly expanded feature set for query and retrieval. Final thoughts The MongoDB document data model provides a powerful and intuitive approach to structuring and interacting with healthcare data (read openEHR), which is often complex and variable. It closely aligns with how you think and code, allowing you to store and retrieve data of any shape and form. The powerful query engine and indexing capabilities further enhance its versatility, enabling you to develop complex query patterns and optimize performance for your specific application requirements. Choosing the appropriate modeling approach depends on the specific requirements of the application, query patterns, and performance considerations. Both the Attribute Pattern and the Flat Hierarchy models offer viable solutions for openEHR data storage in MongoDB, providing flexibility and performance optimizations tailored to different use cases. Additionally, MongoDB Atlas Search introduces powerful search capabilities for enhanced query and retrieval functionalities. While we looked into schema modeling and querying solutions for openEHR data in MongoDB, there are other topics that are of interest while developing production-scale applications and environments where you would like both engineers and operations teams to be more productive. We will be exploring the following topics in a future blog. Simplifying openEHR queries: A DSL based approach to convert existing AQL to MQL and using Generative AI to build MQL based on natural language prompts. Strategies for organizing openEHR data such as Multi-tenancy by clinic and horizontal scaling using sharding. Generate production-scale data volume persisted in a Sharded MongoDB cluster. Run NFR validation on dedicated production grade infrastructure and compare performance of various approaches discussed. We hope this comprehensive guide has provided valuable insights into openEHR schema modeling and querying challenges, as well as potential solutions using MongoDB. Learn more about how MongoDB works with any healthcare data standard in our whitepaper, What is Radical Interoperability References: Introduction to openEHR HL7 FHIR & openEHR Choosing the standard that is right for you MongoDB Wildcard Index

June 5, 2023

Next →

AI-Powered Call Centers: A New Era of Customer Service

Customer satisfaction is critical for insurance companies. Studies have shown that companies with superior customer experiences consistently outperform their peers. In fact, McKinsey found that life and property/casualty insurers with superior customer experiences saw a significant 20% and 65% increase in Total Shareholder Return , respectively, over five years. A satisfied customer is a loyal customer. They are 80% more likely to renew their policies, directly contributing to sustainable growth. However, one major challenge faced by many insurance companies is the inefficiency of their call centers. Agents often struggle to quickly locate and deliver accurate information to customers, leading to frustration and dissatisfaction. This article explores how Dataworkz and MongoDB can transform call center operations. By converting call recordings into searchable vectors (numerical representations of data points in a multi-dimensional space), businesses can quickly access relevant information and improve customer service. We'll dig into how the integration of Amazon Transcribe, Cohere, and MongoDB Atlas Vector Search—as well as Dataworkz's RAG-as-a-service platform— is achieving this transformation. From call recordings to vectors: A data-driven approach Customer service interactions are goldmines of valuable insights. By analyzing call recordings, we can identify successful resolution strategies and uncover frequently asked questions. In turn, by making this information—which is often buried in audio files— accessible to agents, they can give customers faster and more accurate assistance. However, the vast volume and unstructured nature of these audio files make it challenging to extract actionable information efficiently. To address this challenge, we propose a pipeline that leverages AI and analytics to transform raw audio recordings into vectors as shown in Figure 1: Storage of raw audio files: Past call recordings are stored in their original audio format Processing of the audio files with AI and analytics services (such as Amazon Transcribe Call Analytics ): speech-to-text conversion, summarization of content, and vectorization Storage of vectors and metadata: The generated vectors and associated metadata (e.g., call timestamps, agent information) are stored in an operational data store Figure 1: Customer service call insight extraction and vectorization flow Once the data is stored in vector format within the operational data store, it becomes accessible for real-time applications. This data can be consumed directly through vector search or integrated into a retrieval-augmented generation (RAG) architecture, a technique that combines the capabilities of large language models (LLMs) with external knowledge sources to generate more accurate and informative outputs. Introducing Dataworkz: Simplifying RAG implementation Building RAG pipelines can be cumbersome and time-consuming for developers who must learn yet another stack of technologies. Especially in this initial phase, where companies want to experiment and move fast, it is essential to leverage tools that allow us to abstract complexity and don’t require deep knowledge of each component in order to experiment with and realize the benefits of RAG quickly. Dataworkz offers a powerful and composable RAG-as-a-service platform that streamlines the process of building RAG applications for enterprises. To operationalize RAG effectively, organizations need to master five key capabilities: ETL for LLMs: Dataworkz connects with diverse data sources and formats, transforming the data to make it ready for consumption by generative AI applications. Indexing: The platform breaks down data into smaller chunks and creates embeddings that capture semantics, storing them in a vector database. Retrieval: Dataworkz ensures the retrieval of accurate information in response to user queries, a critical part of the RAG process. Synthesis: The retrieved information is then used to build the context for a foundational model, generating responses grounded in reality. Monitoring: With many moving parts in the RAG system, Dataworkz provides robust monitoring capabilities essential for production use cases. Dataworkz's intuitive point-and-click interface (as seen in Video 1) simplifies RAG implementation, allowing enterprises to quickly operationalize AI applications. The platform offers flexibility and choice in data connectors, embedding models, vector stores, and language models. Additionally, tools like A/B testing ensure the quality and reliability of generated responses. This combination of ease of use, optionality, and quality assurance is a key tenet of Dataworkz's "RAG as a Service" offering. Diving deeper: System architecture and functionalities Now that we’ve looked at the components of the pre-processing pipeline, let’s explore the proposed real-time system architecture in detail. It comprises the following modules and functions (see Figure 2): Amazon Transcribe , which receives the audio coming from the customer’s phone and converts it into text. Cohere ’s embedding model, served through Amazon Bedrock , vectorizes the text coming from Transcribe. MongoDB Atlas Vector Search receives the query vector and returns a document that contains the most semantically similar FAQ in the database. Figure 2: System architecture and modules Here are a couple of FAQs we used for the demo: Q: “Can you explain the different types of coverage available for my home insurance?” A: “Home insurance typically includes coverage for the structure of your home, your personal belongings, liability protection, and additional living expenses in case you need to temporarily relocate. I can provide more detailed information on each type if you'd like.” Q: “What is the process for adding a new driver to my auto insurance policy?" A: “To add a new driver to your auto insurance policy, I'll need some details about the driver, such as their name, date of birth, and driver's license number. We can add them to your policy over the phone, or you can do it through our online portal.” Note that the question is reported just for reference, and it’s not used for retrieval. The actual question is provided by the user through the voice interface and then matched in real-time with the answers in the database using Vector Search. This information is finally presented to the customer service operator in text form (see Fig. 3). The proposed architecture is simple but very powerful, easy to implement, and effective. Moreover, it can serve as a foundation for more advanced use cases that require complex interactions, such as agentic workflows , and iterative and multi-step processes that combine LLMs and hybrid search to complete sophisticated tasks. Figure 3: App interface, displaying what has been asked by the customer (left) and how the information is presented to the customer service operator (right) This solution not only impacts human operator workflows but can also underpin chatbots and voicebots, enabling them to provide more relevant and contextual customer responses. Building a better future for customer service By seamlessly integrating analytical and operational data streams, insurance companies can significantly enhance both operational efficiency and customer satisfaction. Our system empowers businesses to optimize staffing, accelerate inquiry resolution, and deliver superior customer service through data-driven, real-time insights. To embark on your own customer service transformation, explore our GitHub repository and take advantage of the Dataworkz free tier .

November 27, 2024