From Relational Databases to AI: An Insurance Data Modernization Journey

Jeff Needham, Luca Napoli, and Jack Yallop
March 14, 2024 | Updated: May 13, 2024
#genAI

Imagine you’re a data architect, a developer, or a data engineer at an insurance company. Management has asked you and your team to build a new AI claim adjustment system, a customer-facing LLM-powered chatbot, and an application to streamline the underwriting process.

However, doing so is far from straightforward due to the challenges you face on a daily basis. The bulk of your time is spent navigating your company’s outdated legacy systems, which were built in the 1970s and 1980s. Some of these legacy platforms were written in COBOL and CICS, and today very few people on your team know how to develop and maintain those technologies. Moreover, the data models you work with are another source of frustration. Every interaction with them is a reminder of the intricate structures that have evolved over time, making data manipulation and analysis a nightmare.

In sum, legacy systems are preventing your team—and your company—from innovating and keeping up with both your industry and customer demands.

Whether you’re trying to modernize your legacy systems to improve operational efficiency, boost developer productivity, or if you want to build AI-powered apps that integrate with large language models (LLMs), MongoDB has a solution for that. In this post, we’ll walk you through a journey that starts with a relational data model refactored into MongoDB collections, vectorization and querying of unstructured data and, finally, retrieval augmented generation (RAG): asking large language models (LLMs) questions about data in natural language.

Identifying, modernizing, and storing the data

Our journey starts with an assessment of the data sources we want to work with. As shown below, we can bucket the data into three different categories:

  • Structured legacy data: Tables of claims, coverages, billings, and more. Is your data locked in rigid relations schemas? This tutorial is a step-by-step guide on how to migrate a real-life insurance relational model with the help of MongoDB Relational Migrator, refactoring 21 tables to only five MongoDB collections.

  • Structured data (JSON): You might have files of policies, insurance products, or forms in JSON format. Check out our docs to learn how to insert those into a MongoDB collection.

  • Unstructured data (PDFs, Audios, Images, etc.): If you need to create and store a numerical representation (vector embedding) of, for instance, claim-related photos of accidents or PDFs of policy guidelines, you can have a look at this blog that will walk you through the process of generating embeddings of pictures of car crashes and persisting them alongside existing fields in a MongoDB collection.

Diagram of different types of data being stored in MongoDB. On the left, buckets of Structured Legacy Data, Structured Data, and Unstructured data are presented. Structured Legacy Data connects to the MongoDB Relational Migrator, before flowing into the Converged AI Data Store. Structured Data flows directly to the AI Data Store. And Unstructured Data connects to an Embedding Model, which then flows into the AI Data Store utilizing vectors.
Figure 1: Storing different types of data into MongoDB

Regardless of the original format or source, our data has finally landed into MongoDB Atlas into what we call a Converged AI Data Store, which is a platform that centrally integrates and organizes enterprise data, including vectors, that enable the development of ML- and AI-powered applications.

Accessing, experimenting, and interacting with the data

It’s time to put the data to work. The Converged AI Data Store unlocks a plethora of use cases and efficiency gains, both for the business and for developers. The next step of the journey is about the different ways we can interact with our data:

  • Database and Full Text Search: Learn how to run database queries, start from the basics and move up to advanced features such as facets, fuzzy search, autocomplete, highlighting, and more with Atlas Search.

  • Vector Search: We can finally leverage unstructured data. The Image Search blog we mentioned earlier also explains how to create a Vector Search index and run vector queries against embeddings of photos.

  • RAG: Combining Vector Search and the power of LLMs, it is possible to interact in natural language with our data (see Figure 2 below), asking complex questions and getting detailed answers. Follow this tutorial to become a RAG expert.

Diagram of RAG where we combine custom data with the LLM. In step 1, the user prompts in natural language through the embedding model. Step 2, the vectorized prompt is used to retrieve context through the Converged AI Data Store. Step 3, the LLM is prompted with context and original question. Step 4, the user receives the answer.
Figure 2: Retrieval augmented generation (RAG) diagram where we dynamically combine our custom data with the LLM to generate reliable and relevant outputs

Having explored all the different ways we can ask questions of the data, we made it to the end of our journey. You are now ready to modernize your company’s systems and finally be able to keep up with the business’ demands.

What will you build next?

If you would like to discover more about Converged AI and Application Data Stores with MongoDB, take a look at the following resources:

← Previous

利用生成式人工智能和 MongoDB 应对网络安全的最大挑战

在不断变化的网络安全环境中，企业面临着众多挑战，需要利用尖端技术提供创新解决方案。 最紧迫的问题之一是网络威胁日益复杂，包括恶意软件、勒索软件和网络钓鱼攻击，这些攻击越来越难以检测和缓解。 此外，数字基础设施的快速扩张扩大了攻击面，使安全团队更难监控和保护每个入口和出口点。 另一个重大挑战是缺少熟练的网络安全专业人员（据独立调查估计，全球缺口约为 400 万1），这使得许多组织容易受到攻击。 这些挑战凸显了对先进技术的需求，这些技术可以增强人类保护数字资产和数据的努力。 生成式AI有何帮助？ 生成式人工智能 ( gen AI ) 已成为应对这些网络安全挑战的强大工具。 通过利用大型语言模型 ( LLM ) 在现有数据集的基础上生成新数据或模式，生成式人工智能可以在多个关键领域提供创新解决方案： 强化威胁检测和响应 生成式人工智能可用于模拟网络威胁，包括复杂的恶意软件和网络钓鱼攻击。 这些模拟有助于训练机器学习模型，以更准确地检测新的和不断演变的威胁。 此外，生成式人工智能可以帮助开发实时对威胁做出反应的自动响应系统。 虽然这永远不会消除对人工监督的需求，但可以减少人工干预和劳累，从而更快地缓解攻击。 例如，在适当的监督下，它可以自动为易受攻击的系统打补丁，或调整防火墙规则以阻止攻击载体。 这种自动快速反应能力对于减少零日漏洞尤为重要，因为从发现漏洞到攻击者利用漏洞之间的窗口很短。 从安全事件事后分析中汲取可操作的经验教训 在网络安全事件发生后，进行彻底的事后分析对于了解事件的经过、原因以及今后如何防止类似事件的发生至关重要。 在这一过程中，生成式人工智能可以综合和汇总多种来源的复杂数据（日志、网络流量和安全警报等），发挥关键作用。 通过分析这些数据，生成式人工智能可以识别可能导致安全漏洞的模式和异常，从而提供由于信息量和复杂性而可能被人类分析师忽视的见解。 此外，它还可以生成全面的报告，突出显示关键发现、诱发因素和潜在漏洞，从而简化事后分析过程。 这种能力不仅能加快恢复和学习过程，还能使组织实施更有效的补救策略，最终加强其网络安全态势。 生成用于深度模型训练的合成数据 用于培训网络安全系统的真实数据短缺，这是一个重大障碍。生成式人工智能可以创建真实的合成数据集，反映真实的网络流量和用户行为，而不会暴露敏感信息。 这种合成数据可用于训练检测系统，在不损害隐私或安全的情况下提高其准确性和有效性。 自动检测网络钓鱼 网络钓鱼仍然是最常见的攻击载体之一。 生成式人工智能可以分析网络钓鱼电子邮件和网站中的模式，生成能够高精度预测和检测网络钓鱼尝试的模型。 通过将这些模型集成到电子邮件系统和网络浏览器中，组织可以自动过滤掉网络钓鱼内容，保护用户免受潜在威胁。 综合考虑：机遇与风险 生成式人工智能有望实现复杂流程的自动化、加强威胁检测和响应、提供对网络威胁的更深入了解，从而改变网络安全实践。随着业界不断将生成式人工智能融入网络安全战略，我们必须对这项技术的道德使用和滥用潜力保持警惕。 尽管如此，它在加强数字防御方面所带来的好处是毋庸置疑的，因此成为应对网络威胁的持久战中的宝贵资产。 MongoDB 如何提供帮助？ 有了 MongoDB，您的开发团队就能以任何规模更快地构建和部署强大、正确和差异化的实时网络防御系统。 要了解 MongoDB 如何做到这一点，请考虑 AI 技术堆栈包含三层： 底层计算 (GPU) 和 LLM 微调模型的工具以及用于上下文学习和对训练模型进行推理的工具 人工智能应用程序和相关最终用户体验 MongoDB 在堆栈的第二层运行。 它使客户能够将自己的专有数据带到任何计算基础设施上运行的任何 LLM，以构建生成式人工智能驱动的网络安全应用程序。 为此，MongoDB 解决采用生成式人工智能保障网络安全时最棘手的问题。 MongoDB Atlas 将运营数据、非结构化数据和矢量数据安全地统一在一个完全托管的多云平台中，避免了在不同系统之间复制和同步数据的需要。 MongoDB 基于文档的架构还允许开发团队轻松地对应用程序数据和矢量嵌入之间的关系进行建模。 这样就可以更深入、更快速地分析和见解与安全相关的数据。 图 1： 在统一的 API 和开发者数据平台中，MongoDB Atlas 汇集了构建现代网络安全应用程序所需的所有数据服务。 MongoDB 的开放式架构与丰富的 AI 开发者框架、LLM 和嵌入式提供商的生态系统相集成。这与我们业界领先的多云功能相结合，使您的开发团队能够灵活快速地行动，避免在这个快速发展的领域中被任何特定的云提供商或 AI 技术限制。 请查看我们的 AI 资源页面，了解有关使用 MongoDB 构建 AI 驱动的应用的更多信息。 将生成式人工智能和 MongoDB 应用于现实世界的网络安全应用 威胁情报 ExTrac 利用 AI 驱动的分析技术和 MongoDB Atlas，通过分析数千个来源的数据来预测公共安全风险。该平台最初帮助西方政府预测冲突，现在正扩展到企业的声誉管理等方面。 MongoDB 的文档数据模型使 ExTrac 能够高效管理复杂数据，增强实时威胁识别。 Atlas Vector Search 有助于增强语言模型，并管理文本、图像和视频的矢量嵌入，从而加快功能开发。这种方法使 ExTrac 能够利用 MongoDB 的灵活性和强大功能，有效地为客户建立趋势模型、追踪不断变化的叙事和预测风险，从而处理任何形状和结构的数据。在 ExTrac 案例研究中了解更多信息。 网络安全评估 VISO TRUST 利用 AI 简化对第三方网络风险的评估，使复杂的供应商安全信息能够快速获取，以便做出明智的决策。 VISO TRUST 的平台利用 Amazon Bedrock 和 MongoDB Atlas，实现了供应商安全尽职调查的自动化，大大减少了安全团队的工作量。 其 AI 驱动的方法涉及人工智能，可对安全文档进行分类、检测组织并预测人工智能中的安全控制位置。 MongoDB Atlas 为密集检索系统提供文本嵌入，通过检索增强生成 ( RAG ) 提高 LLM 的准确性，提供即时、可操作的安全见解。 通过创新地使用技术，VISO TRUST 能够提供快速、可扩展的网络风险评估，为 InstaCart 和 Upwork 等企业大大减少了工作量和时间。 MongoDB 灵活的文档数据库和 Atlas Vector Search 在管理和查询海量数据方面发挥了关键作用，支持 VISO TRUST 提供全面网络风险情报的使命。 在 Viso Trust 案例研究中了解更多信息。 开始使用的步骤 由 LLM 驱动的生成式人工智能，辅以编码为矢量嵌入的操作数据，为网络安全领域带来了许多新的可能性。 如果您想进一步了解这项技术及其可能性，请查看我们的 Atlas Vector Search Learning Byte 。在短短 10 分钟内，您将大致了解不同的使用案例以及如何开始。 1 1 Hill, M. （2023 年 4 月 10 日）。 尽管进行了大规模的招聘活动，但网络安全劳动力缺口仍达 400 万。 CSO。

March 13, 2024
Next →

MongoDB and Partners: Building the AI Future, Together

If you’re like me, over the past year you’ve closely watched AI’s developments—and the world’s reactions to them. From infectious excitement about AI’s capabilities, to impatience with its cost and return on investment, every day has been filled with AI twists and turns. It’s been quite the roller coaster. During the ride, from time to time I’ve wondered where AI falls on the Gartner hype cycle, which gives "a view of how a technology or application will evolve over time." Have we hit the "peak of inflated expectations" only to fall into the "trough of disillusionment?" Or is the hype cycle an imperfect guide, as The Economist argues? The reality is that it takes time for any new technology—even transformative ones like AI—to take hold. And every advance, no matter how big, has had its detractors. A famous example is that of Picasso (!), who in 1968 said, “Computers are useless. They can only give you answers.” (!!) For our part, MongoDB is convinced that AI is a once-in-a-generation technology that will enhance every future application—a belief that has been reinforced by the incredible work our partners have shared at MongoDB’s 2024 events. Speeding AI development MongoDB is committed to helping organizations of all sizes succeed with AI, and one way we’re doing that is by collaborating with the MongoDB partner ecosystem to create powerful, user-friendly AI development tools and solutions. For example, Fireworks.ai —which is a member of the MongoDB AI Applications Program ecosystem —created an inference solution that hosts gen AI models and supports containerized deployments. This tool makes it easier for developers to build and deploy powerful applications with a range of easy-to-use tools and customization options. They can choose to use state-of-the-art, open-source language, image, and multimodal foundation models off the shelf, or they can customize and fine-tune models to their needs. Jointly, Fireworks.ai and MongoDB provide a solution for developers who want to leverage highly curated and optimized open-source models and combine these with their organization’s own proprietary data—and to do so with unparalleled speed and security. “MongoDB is one of the most sophisticated database providers, and it’s very easy to use,” said Benny Chen , cofounder of Fireworks.ai. "We want developers to be able to use these tools, and we want to work with providers who enable and empower developers." Nomic , another MAAP ecosystem member, also enables developers with best-in-class solutions across the entire unstructured data workflow. Their Embed offering, available through the Nomic API , allows users to vectorize large-scale datasets for use in text, image, and multimodal retrieval applications, including retrieval-augmented generation (RAG), using only their web browser. The Nomic-MongoDB solution is a highly efficient, open-weight model that developers can use to visualize the unstructured datasets they store in MongoDB Atlas . These insights help users quickly discover trends and articulate data-driven value propositions. Nomic also supported the recently announced vector quantization in MongoDB Atlas Vector Search , which reduces vector sizes while preserving performance. Last—but hardly least!—there’s our new reference architecture with MAAP partners AWS and Anthropic. Announced at MongoDB.local London , the reference architecture supports building memory-enhanced AI agents, and is designed to streamline complex processes and develop smarter, more responsive applications. For more—including a link to the code on Github— check out the MongoDB Developer Center . Making AI work for anyone and everyone The companies MongoDB partners with aren’t just making gen AI easier for developers—they’re building tools for everyone. For example, Capgemini has invested $2 billion in gen AI and is training 100,000 of its employees in the technology. GenYoda, a solution that helps insurance professionals with their daily work, is a product of this investment. GenYoda leverages MongoDB Atlas Vector Search to analyze large amounts of customer data, like policy statements, premiums, claims history, and health information. Using GenYoda, insurance professionals can quickly analyze underwriters’ reports to make informed decisions, create longitudinal health summaries, and streamline customer interactions to improve contact center efficiency. GenYoda can ingest 100,000 documents in just a few hours and respond to users’ queries in two to three seconds—a metric on par with the most widely used gen AI models. And it produces results: in one example, by using Capgemini’s solution an insurer was able to increase productivity by 15%, add new reports 25% faster (thus speeding decision-making), and reduce the manual effort of searching PDFs, increasing efficiency by 10%. Building the future of AI together So, what’s next? Honestly, I’m as curious as you are. But I’m also incredibly excited. At MongoDB, we’re active participants in the AI revolution, working to embrace the possibilities that lie ahead. The future of gen AI is bright, and I can’t wait to see what we’ll build together. To learn more about how MongoDB can accelerate your AI journey, explore the MongoDB AI Applications Program .

November 4, 2024