From Relational Databases to AI: An Insurance Data Modernization Journey

Jeff Needham, Luca Napoli, and Jack Yallop
March 14, 2024 | Updated: May 13, 2024
#genAI

Imagine you’re a data architect, a developer, or a data engineer at an insurance company. Management has asked you and your team to build a new AI claim adjustment system, a customer-facing LLM-powered chatbot, and an application to streamline the underwriting process.

However, doing so is far from straightforward due to the challenges you face on a daily basis. The bulk of your time is spent navigating your company’s outdated legacy systems, which were built in the 1970s and 1980s. Some of these legacy platforms were written in COBOL and CICS, and today very few people on your team know how to develop and maintain those technologies. Moreover, the data models you work with are another source of frustration. Every interaction with them is a reminder of the intricate structures that have evolved over time, making data manipulation and analysis a nightmare.

In sum, legacy systems are preventing your team—and your company—from innovating and keeping up with both your industry and customer demands.

Whether you’re trying to modernize your legacy systems to improve operational efficiency, boost developer productivity, or if you want to build AI-powered apps that integrate with large language models (LLMs), MongoDB has a solution for that. In this post, we’ll walk you through a journey that starts with a relational data model refactored into MongoDB collections, vectorization and querying of unstructured data and, finally, retrieval augmented generation (RAG): asking large language models (LLMs) questions about data in natural language.

Identifying, modernizing, and storing the data

Our journey starts with an assessment of the data sources we want to work with. As shown below, we can bucket the data into three different categories:

  • Structured legacy data: Tables of claims, coverages, billings, and more. Is your data locked in rigid relations schemas? This tutorial is a step-by-step guide on how to migrate a real-life insurance relational model with the help of MongoDB Relational Migrator, refactoring 21 tables to only five MongoDB collections.

  • Structured data (JSON): You might have files of policies, insurance products, or forms in JSON format. Check out our docs to learn how to insert those into a MongoDB collection.

  • Unstructured data (PDFs, Audios, Images, etc.): If you need to create and store a numerical representation (vector embedding) of, for instance, claim-related photos of accidents or PDFs of policy guidelines, you can have a look at this blog that will walk you through the process of generating embeddings of pictures of car crashes and persisting them alongside existing fields in a MongoDB collection.

Diagram of different types of data being stored in MongoDB. On the left, buckets of Structured Legacy Data, Structured Data, and Unstructured data are presented. Structured Legacy Data connects to the MongoDB Relational Migrator, before flowing into the Converged AI Data Store. Structured Data flows directly to the AI Data Store. And Unstructured Data connects to an Embedding Model, which then flows into the AI Data Store utilizing vectors.
Figure 1: Storing different types of data into MongoDB

Regardless of the original format or source, our data has finally landed into MongoDB Atlas into what we call a Converged AI Data Store, which is a platform that centrally integrates and organizes enterprise data, including vectors, that enable the development of ML- and AI-powered applications.

Accessing, experimenting, and interacting with the data

It’s time to put the data to work. The Converged AI Data Store unlocks a plethora of use cases and efficiency gains, both for the business and for developers. The next step of the journey is about the different ways we can interact with our data:

  • Database and Full Text Search: Learn how to run database queries, start from the basics and move up to advanced features such as facets, fuzzy search, autocomplete, highlighting, and more with Atlas Search.

  • Vector Search: We can finally leverage unstructured data. The Image Search blog we mentioned earlier also explains how to create a Vector Search index and run vector queries against embeddings of photos.

  • RAG: Combining Vector Search and the power of LLMs, it is possible to interact in natural language with our data (see Figure 2 below), asking complex questions and getting detailed answers. Follow this tutorial to become a RAG expert.

Diagram of RAG where we combine custom data with the LLM. In step 1, the user prompts in natural language through the embedding model. Step 2, the vectorized prompt is used to retrieve context through the Converged AI Data Store. Step 3, the LLM is prompted with context and original question. Step 4, the user receives the answer.
Figure 2: Retrieval augmented generation (RAG) diagram where we dynamically combine our custom data with the LLM to generate reliable and relevant outputs

Having explored all the different ways we can ask questions of the data, we made it to the end of our journey. You are now ready to modernize your company’s systems and finally be able to keep up with the business’ demands.

What will you build next?

If you would like to discover more about Converged AI and Application Data Stores with MongoDB, take a look at the following resources:

← Previous

利用生成式人工智能和 MongoDB 应对网络安全的最大挑战

在不断变化的网络安全环境中，企业面临着众多挑战，需要利用尖端技术提供创新解决方案。 最紧迫的问题之一是网络威胁日益复杂，包括恶意软件、勒索软件和网络钓鱼攻击，这些攻击越来越难以检测和缓解。 此外，数字基础设施的快速扩张扩大了攻击面，使安全团队更难监控和保护每个入口和出口点。 另一个重大挑战是缺少熟练的网络安全专业人员（据独立调查估计，全球缺口约为 400 万1），这使得许多组织容易受到攻击。 这些挑战凸显了对先进技术的需求，这些技术可以增强人类保护数字资产和数据的努力。 生成式AI有何帮助？ 生成式人工智能 ( gen AI ) 已成为应对这些网络安全挑战的强大工具。 通过利用大型语言模型 ( LLM ) 在现有数据集的基础上生成新数据或模式，生成式人工智能可以在多个关键领域提供创新解决方案： 强化威胁检测和响应 生成式人工智能可用于模拟网络威胁，包括复杂的恶意软件和网络钓鱼攻击。 这些模拟有助于训练机器学习模型，以更准确地检测新的和不断演变的威胁。 此外，生成式人工智能可以帮助开发实时对威胁做出反应的自动响应系统。 虽然这永远不会消除对人工监督的需求，但可以减少人工干预和劳累，从而更快地缓解攻击。 例如，在适当的监督下，它可以自动为易受攻击的系统打补丁，或调整防火墙规则以阻止攻击载体。 这种自动快速反应能力对于减少零日漏洞尤为重要，因为从发现漏洞到攻击者利用漏洞之间的窗口很短。 从安全事件事后分析中汲取可操作的经验教训 在网络安全事件发生后，进行彻底的事后分析对于了解事件的经过、原因以及今后如何防止类似事件的发生至关重要。 在这一过程中，生成式人工智能可以综合和汇总多种来源的复杂数据（日志、网络流量和安全警报等），发挥关键作用。 通过分析这些数据，生成式人工智能可以识别可能导致安全漏洞的模式和异常，从而提供由于信息量和复杂性而可能被人类分析师忽视的见解。 此外，它还可以生成全面的报告，突出显示关键发现、诱发因素和潜在漏洞，从而简化事后分析过程。 这种能力不仅能加快恢复和学习过程，还能使组织实施更有效的补救策略，最终加强其网络安全态势。 生成用于深度模型训练的合成数据 用于培训网络安全系统的真实数据短缺，这是一个重大障碍。生成式人工智能可以创建真实的合成数据集，反映真实的网络流量和用户行为，而不会暴露敏感信息。 这种合成数据可用于训练检测系统，在不损害隐私或安全的情况下提高其准确性和有效性。 自动检测网络钓鱼 网络钓鱼仍然是最常见的攻击载体之一。 生成式人工智能可以分析网络钓鱼电子邮件和网站中的模式，生成能够高精度预测和检测网络钓鱼尝试的模型。 通过将这些模型集成到电子邮件系统和网络浏览器中，组织可以自动过滤掉网络钓鱼内容，保护用户免受潜在威胁。 综合考虑：机遇与风险 生成式人工智能有望实现复杂流程的自动化、加强威胁检测和响应、提供对网络威胁的更深入了解，从而改变网络安全实践。随着业界不断将生成式人工智能融入网络安全战略，我们必须对这项技术的道德使用和滥用潜力保持警惕。 尽管如此，它在加强数字防御方面所带来的好处是毋庸置疑的，因此成为应对网络威胁的持久战中的宝贵资产。 MongoDB 如何提供帮助？ 有了 MongoDB，您的开发团队就能以任何规模更快地构建和部署强大、正确和差异化的实时网络防御系统。 要了解 MongoDB 如何做到这一点，请考虑 AI 技术堆栈包含三层： 底层计算 (GPU) 和 LLM 微调模型的工具以及用于上下文学习和对训练模型进行推理的工具 人工智能应用程序和相关最终用户体验 MongoDB 在堆栈的第二层运行。 它使客户能够将自己的专有数据带到任何计算基础设施上运行的任何 LLM，以构建生成式人工智能驱动的网络安全应用程序。 为此，MongoDB 解决采用生成式人工智能保障网络安全时最棘手的问题。 MongoDB Atlas 将运营数据、非结构化数据和矢量数据安全地统一在一个完全托管的多云平台中，避免了在不同系统之间复制和同步数据的需要。 MongoDB 基于文档的架构还允许开发团队轻松地对应用程序数据和矢量嵌入之间的关系进行建模。 这样就可以更深入、更快速地分析和见解与安全相关的数据。 图 1： 在统一的 API 和开发者数据平台中，MongoDB Atlas 汇集了构建现代网络安全应用程序所需的所有数据服务。 MongoDB 的开放式架构与丰富的 AI 开发者框架、LLM 和嵌入式提供商的生态系统相集成。这与我们业界领先的多云功能相结合，使您的开发团队能够灵活快速地行动，避免在这个快速发展的领域中被任何特定的云提供商或 AI 技术限制。 请查看我们的 AI 资源页面，了解有关使用 MongoDB 构建 AI 驱动的应用的更多信息。 将生成式人工智能和 MongoDB 应用于现实世界的网络安全应用 威胁情报 ExTrac 利用 AI 驱动的分析技术和 MongoDB Atlas，通过分析数千个来源的数据来预测公共安全风险。该平台最初帮助西方政府预测冲突，现在正扩展到企业的声誉管理等方面。 MongoDB 的文档数据模型使 ExTrac 能够高效管理复杂数据，增强实时威胁识别。 Atlas Vector Search 有助于增强语言模型，并管理文本、图像和视频的矢量嵌入，从而加快功能开发。这种方法使 ExTrac 能够利用 MongoDB 的灵活性和强大功能，有效地为客户建立趋势模型、追踪不断变化的叙事和预测风险，从而处理任何形状和结构的数据。在 ExTrac 案例研究中了解更多信息。 网络安全评估 VISO TRUST 利用 AI 简化对第三方网络风险的评估，使复杂的供应商安全信息能够快速获取，以便做出明智的决策。 VISO TRUST 的平台利用 Amazon Bedrock 和 MongoDB Atlas，实现了供应商安全尽职调查的自动化，大大减少了安全团队的工作量。 其 AI 驱动的方法涉及人工智能，可对安全文档进行分类、检测组织并预测人工智能中的安全控制位置。 MongoDB Atlas 为密集检索系统提供文本嵌入，通过检索增强生成 ( RAG ) 提高 LLM 的准确性，提供即时、可操作的安全见解。 通过创新地使用技术，VISO TRUST 能够提供快速、可扩展的网络风险评估，为 InstaCart 和 Upwork 等企业大大减少了工作量和时间。 MongoDB 灵活的文档数据库和 Atlas Vector Search 在管理和查询海量数据方面发挥了关键作用，支持 VISO TRUST 提供全面网络风险情报的使命。 在 Viso Trust 案例研究中了解更多信息。 开始使用的步骤 由 LLM 驱动的生成式人工智能，辅以编码为矢量嵌入的操作数据，为网络安全领域带来了许多新的可能性。 如果您想进一步了解这项技术及其可能性，请查看我们的 Atlas Vector Search Learning Byte 。在短短 10 分钟内，您将大致了解不同的使用案例以及如何开始。 1 1 Hill, M. （2023 年 4 月 10 日）。 尽管进行了大规模的招聘活动，但网络安全劳动力缺口仍达 400 万。 CSO。

March 13, 2024
Next →

Hanabi Technologies Uses MongoDB to Power AI Assistant, Hana

For all the hype surrounding generative AI, cynics tend to view the few real-world implementations as little more than “fancy chatbots.” But for Abhinav Aggarwal, CEO of Hanabi Technologies, the idea of a generative AI-powered bot that is more than just an assistant was intriguing. “I’d been using ChatGPT since it launched,” said Aggarwal. “That got me thinking: How could we make a chatbot that was like a team member?” And with that concept, Hana was born. The problem with bots “Most generative AI chatbots do not act like people; they wait for a command and give a response,” said Aggarwal. “We wanted to create a human-like chatbot that would proactively help people based on what they wanted—automating reminders, for example, or fetching time zones from your calendar to correctly schedule meetings.” Hanabi’s flagship product, Hana, is an AI assistant designed to enhance team collaboration within Google Chat, working in concert with Google Workspace and its suite of products. “Our target customers are smaller companies of between 10 and 50 people. At this size you’re not going to build your own agent from scratch,” he said. Hana integrates with Google APIs to deliver a human-like assistant that chimes in with helpful interventions, such as automatically setting reminders and making sure meetings are booked in the right time zone for each participant. “Hana is designed to bring AI to smaller companies and help them collaborate in a space where they are already working—Google Workspace,” Aggarwal explained. The MongoDB Atlas solution For Hana to act like a member of the team, Hanabi needed to process massive amounts of data to support advanced features like retrieval-augmented generation (RAG) for better information retrieval across Google Docs and many other sources. And with a rapidly growing user base of over 600 organizations and 17,000+ installs, Hanabi also required a secure, scalable, and high-performing data storage solution. MongoDB Atlas provided a flexible document model, built-in vector database, and scalable cloud-based infrastructure, freeing Hanabi engineers to build new features for Hana rather than focusing on rote tasks like data extract, transform, and load processes or manual scaling and provisioning. Now, MongoDB Atlas handles a variety of responsibilities: Scalability and security: MongoDB Atlas’s auto-scaling and automatic backup features have enabled Hanabi to seamlessly grow its user base without the need for manual database management. RAG: MongoDB Atlas plays a critical role in Hana’s RAG functionality. The platform enables Hanabi to split Google Docs into small sections, create embeddings, and store these sections in Atlas’s vector database. Development Processes: According to Aggarwal, MongoDB’s flexibility in managing changing schemas has been essential to the company’s fast-paced development cycle. Data Visualization: Using MongoDB Atlas Charts has enabled Hanabi to create comprehensive dashboards for real-time data visualization. This has helped the team track usage, set reminders, and optimize performance without needing to build a manual dashboard. Impact and results With MongoDB Atlas, Hanabi can successfully scale Hana to meet the demands of its rapidly expanding user base. The integration is also enabling Hana to offer powerful features like automatic interactions with customers, advanced information retrieval from Google Docs, and manually added memory snippets, making it an essential tool for teams around the world. Next steps Hanabi plans to continue integrating more tools into Hana while expanding its reach to personal Gmail users. The company is also rolling out a new automatic-interaction feature, further enhancing Hana’s ability to proactively assist users without direct commands. MongoDB Atlas remains a key component of Hanabi’s stack, alongside Google Kubernetes Engine, NestJS, and LangChain, enabling Hanabi to focus on innovating to improve the customer experience. Tech Stack MongoDB Atlas Google Kubernetes Engine NestJS LangChain Are you building AI apps? Join the MongoDB AI Innovators Program today! Successful participants gain access to free MongoDB Atlas credits, technical enablement, and invaluable connections within the broader AI ecosystem. If your company is interested in being featured, we’d love to hear from you. Connect with us at ai_adopters@mongodb.com.

November 21, 2024