Choosing the Right Tool for the Job: Understanding the Analytics Spectrum

Mat Keep and Steve Jurczak
November 30, 2022 | Updated: December 7, 2023

Data-driven organizations share a common desire to get more value out of the data they're generating. To maximize that value, many of them are asking the same or similar questions:

How long does it take to get analytics and insights from our application data?
What would be the business impact if we could make that process faster? What new experiences could we create by having analytics integrated directly within our customer-facing apps?
How do our developers access the tools and APIs they need to build sophisticated analytics queries directly into their application code?
How do we make sense of voluminous streams of time-series data?

We believe the answer to these questions in today's digital economy is application-driven analytics.

What is Application-Driven Analytics?

Traditionally, there's been a separation at organizations between applications that run the business and analytics that manage the business. They're built by different teams, they serve different audiences, and the data itself is replicated and stored in different systems.

There are benefits to the traditional way of doing things and it's not going away. However, in today's digital economy, where the need to create competitive advantage and reduce costs and risk are paramount, organizations will continue to innovate upon the traditional model.

Today, those needs manifest themselves in the demand for smarter applications that drive better customer experiences and surface insights to initiate intelligent actions automatically. This all happens within the flow of the application on live, operational data in real time. Alongside those applications, the business also wants faster insights so it can see what's happening, when it's happening. This is known as business visibility, and the goal of it is to increase efficiency by enabling faster decisions on fresher data. In-app analytics and real-time visibility are enabled by what we call application-driven analytics.

You can find examples of application-driven analytics in multiple real-world industry use cases including:

Hyper-personalization in retail
Fraud prevention in financial services
Preventative maintenance in manufacturing
Single subscriber view in telecommunications
Fitness tracking in healthcare
A/B testing in gaming

Where Application-Driven Analytics fits in the Analytics Ecosystem

Application-driven analytics complements existing analytics processes where data is moved out of operational systems into centralized data warehouses and data lakes. In no way does it replace them. However, a broader spectrum of capabilities are now required to meet more demanding business requirements.

Contrasting the two approaches, application-driven analytics is designed to continuously query data in your operational systems. The freshest data comes in from the application serving many concurrent users at very low latency. It involves working on much smaller subsets of data compared to centralized analytics systems. Application-driven analytics is typically working with hundreds to possibly a few thousand records at a time. And it's running less complex queries against that data.

At the other end of the spectrum is centralized analytics. These systems are running much more complex queries across massive data sets — hundreds of thousands or maybe millions of records, and maybe at petabyte scale — that have been ingested from many different operational data sources across the organization.

Table 1 below identifies the required capabilities across the spectrum of different classes of analytics. These are designed to help MongoDB’s customers match appropriate technologies and skill sets to each business use case they are building for.

By mapping required capabilities to use cases, you can see how these different classes of analytics serve different purposes. If, for example, we're dealing with recommendations in an e-commerce platform, the centralized data warehouse or data lake will regularly analyze vast troves of first- and third-party customer data. This analysis is then blended with available inventory to create a set of potential customer offers.

These offers are then loaded back into operational systems where application-driven analytics is used to decide which offers are most relevant to the customer based on a set of real-time criteria, such as actual stock availability and which items a shopper might already have in their basket. This real-time decision-making is important because you wouldn't want to serve an offer on a product that can no longer be fulfilled or on an item a customer has already decided to buy. This example demonstrates why it is essential to choose the right tool for the job. Specifically, in order to build a portfolio of potential offers, the centralized data warehouse or data lake is an ideal fit. Such technologies can process hundreds of TBs of customer records and order data in a single query.

The same technologies, however, are completely inappropriate when it comes to serving those offers to customers in real time. Centralized analytics systems are not designed to serve thousands of concurrent user sessions. Nor can they access real-time inventory or basket data in order to make low latency decisions in milliseconds. Instead, for these scenarios, application-driven analytics served from an operational system is the right technology fit.

As we can see, application-driven analytics is complementary to traditional centralized analytics, and in no way competitive to it.

The benefits to organizations of using these complementary classes of analytics include:

Maximizing competitive advantage through smarter and more intelligent applications
Out-innovating and differentiating in the market
Improving customer experience and loyalty
Reducing cost by improving business visibility and efficiency

Through its design, MongoDB Atlas unifies the essential data services needed to deliver on application-driven analytics. It gives developers the tools, tech, and skills they need to infuse analytics into their apps. At the same time, Atlas provides business analysts, data scientists, and data engineers direct access to live data using their regular tools without impacting the app.

For more information about how to implement app-driven analytics and how the MongoDB developer data platform gives you the tools needed to succeed, download our white paper, Application-Driven Analytics: Defining the Next Wave of Modern Apps.

← Previous

Built With MongoDB: Inspirit Helps Kids Learn Science Through Immersive Technology

Immersive technologies like AR and VR have tremendous potential to transform learning outcomes for students. By representing complicated, often intangible concepts in an interactive, 3D platform and VR system, they encourage engagement and improve memory recall in STEM (science, technology, engineering, and mathematics) subjects. Palo Alto-based startup, Inspirit , has developed an immersive, interactive VR platform that combines the best of both 3D and VR instruction to allow students in middle school and high school to experience science and learning through virtual reality instead of reading. Inspirit is also part of the MongoDB for Startups program , which helps startups build faster and scale further with free MongoDB Atlas credits, one-on-one technical advice, co-marketing opportunities, and access to a vast partner network. Birth of a Startup Inspirit Co-founders Amrutha Vasan and Aditya Vishwanath were conducting research at Georgia Tech and trying to understand how to bring virtual reality into education when they had a critical insight. "A lot of students tend to lose sight or just lose interest in science by the time they hit high school," Vasan says. "What we realized very quickly was that you can give teachers the tools that they need in order to do things that are way too expensive or just simply impossible to do in the real world." As students get into higher order physics, calculus, biology, and chemistry, they struggle with visualizing difficult concepts. "What we aim to do is provide interactive 3D models and interactive simulations that actually teach them core science concepts so that they build that curiosity and actually engage with the material in order to keep them in STEM programs and moving forward," Vasan says. Company evolution Inspirit has evolved since its early days. The founders initially focused on VR headsets and 3D goggles that students would share in the classroom. Once the global COVID-19 pandemic hit, it was clear that students wouldn't be going back to the classroom anytime soon. "We had to very quickly pivot into building a web platform that students could use at home and teachers could use for hybrid and online teaching," Vasan says. "Through the pandemic, we learned very quickly exactly how the education system would actually be changing. And so through that adaptation, we've now built a cross-platform product." Building a cross-platform solution aligns with one of Inspirit's core values, which is to be accessible. Inspirit is now available on the web in addition to virtual reality headsets so students can access the tools on different platforms. Inspirit isn't just an extracurricular activity. One of the biggest differentiators between Inspirit VR classroom and other platforms is that it addresses core content. It's not something that students need to use independently. "You still need a really great teacher," Vasan says. "There is no way for you to show a student a eukaryotic cell other than by drawing it on a whiteboard or giving them a diagram or a video," she says. "You still need a great teacher in order to use our platform. But they can now use this to help their students visualize difficult science concepts." Inspirit helps students who love science immerse themselves in it more while at the same time helping kids who dislike science experience it in a new, more engaging way. Building with MongoDB Inspirit chose MongoDB over other platforms because of its support and flexibility. As a startup, the founders knew their database structure would be changing constantly. "MongoDB allows for a lot of flexibility," Vasan says, "which is really important because we're probably not going to have the perfect database going into a startup. Allowing us to consistently change it as we're scaling and growing has just been very helpful." Regarding MongoDB support, which is one of the benefits of the MongoDB for Startups program, Vasan says the company got the most support from MongoDB than any of the other databases they were looking at. "That's ultimately why we ended up choosing it for our company," she says. "The things that we love the most about MongoDB are Atlas , Charts , and search indexes . Atlas has been really cool for visualizing all the data and for helping us grow and scale quickly as well." "We've had a really good partnership with the MongoDB for Startups program in particular," Vasan says. Inspirit uses AWS, and Vasan cites how easily Atlas integrates with AWS as another key benefit that she appreciates. "AWS integrates very easily with MongoDB. So to be honest, it's been pretty easy scaling our backend." Compliance was another critical requirement. "We actually need to be COPPA compliant," Vasan says, referring to the Children's Online Privacy Protection Rule. "There are a lot of data regulations for what we're using since we have sensitive student, parent, and teacher data. And so both MongoDB and AWS actually make that a lot easier." If you're interested in learning more about Inspirit's immersive learning tools, visit their website . Are you part of a startup and interested in joining the MongoDB for Startups program? Apply now . For more startup content, check out our previous blog on Qubitro .

November 30, 2022

Next →

AI-Powered Call Centers: A New Era of Customer Service

Customer satisfaction is critical for insurance companies. Studies have shown that companies with superior customer experiences consistently outperform their peers. In fact, McKinsey found that life and property/casualty insurers with superior customer experiences saw a significant 20% and 65% increase in Total Shareholder Return , respectively, over five years. A satisfied customer is a loyal customer. They are 80% more likely to renew their policies, directly contributing to sustainable growth. However, one major challenge faced by many insurance companies is the inefficiency of their call centers. Agents often struggle to quickly locate and deliver accurate information to customers, leading to frustration and dissatisfaction. This article explores how Dataworkz and MongoDB can transform call center operations. By converting call recordings into searchable vectors (numerical representations of data points in a multi-dimensional space), businesses can quickly access relevant information and improve customer service. We'll dig into how the integration of Amazon Transcribe, Cohere, and MongoDB Atlas Vector Search—as well as Dataworkz's RAG-as-a-service platform— is achieving this transformation. From call recordings to vectors: A data-driven approach Customer service interactions are goldmines of valuable insights. By analyzing call recordings, we can identify successful resolution strategies and uncover frequently asked questions. In turn, by making this information—which is often buried in audio files— accessible to agents, they can give customers faster and more accurate assistance. However, the vast volume and unstructured nature of these audio files make it challenging to extract actionable information efficiently. To address this challenge, we propose a pipeline that leverages AI and analytics to transform raw audio recordings into vectors as shown in Figure 1: Storage of raw audio files: Past call recordings are stored in their original audio format Processing of the audio files with AI and analytics services (such as Amazon Transcribe Call Analytics ): speech-to-text conversion, summarization of content, and vectorization Storage of vectors and metadata: The generated vectors and associated metadata (e.g., call timestamps, agent information) are stored in an operational data store Figure 1: Customer service call insight extraction and vectorization flow Once the data is stored in vector format within the operational data store, it becomes accessible for real-time applications. This data can be consumed directly through vector search or integrated into a retrieval-augmented generation (RAG) architecture, a technique that combines the capabilities of large language models (LLMs) with external knowledge sources to generate more accurate and informative outputs. Introducing Dataworkz: Simplifying RAG implementation Building RAG pipelines can be cumbersome and time-consuming for developers who must learn yet another stack of technologies. Especially in this initial phase, where companies want to experiment and move fast, it is essential to leverage tools that allow us to abstract complexity and don’t require deep knowledge of each component in order to experiment with and realize the benefits of RAG quickly. Dataworkz offers a powerful and composable RAG-as-a-service platform that streamlines the process of building RAG applications for enterprises. To operationalize RAG effectively, organizations need to master five key capabilities: ETL for LLMs: Dataworkz connects with diverse data sources and formats, transforming the data to make it ready for consumption by generative AI applications. Indexing: The platform breaks down data into smaller chunks and creates embeddings that capture semantics, storing them in a vector database. Retrieval: Dataworkz ensures the retrieval of accurate information in response to user queries, a critical part of the RAG process. Synthesis: The retrieved information is then used to build the context for a foundational model, generating responses grounded in reality. Monitoring: With many moving parts in the RAG system, Dataworkz provides robust monitoring capabilities essential for production use cases. Dataworkz's intuitive point-and-click interface (as seen in Video 1) simplifies RAG implementation, allowing enterprises to quickly operationalize AI applications. The platform offers flexibility and choice in data connectors, embedding models, vector stores, and language models. Additionally, tools like A/B testing ensure the quality and reliability of generated responses. This combination of ease of use, optionality, and quality assurance is a key tenet of Dataworkz's "RAG as a Service" offering. Diving deeper: System architecture and functionalities Now that we’ve looked at the components of the pre-processing pipeline, let’s explore the proposed real-time system architecture in detail. It comprises the following modules and functions (see Figure 2): Amazon Transcribe , which receives the audio coming from the customer’s phone and converts it into text. Cohere ’s embedding model, served through Amazon Bedrock , vectorizes the text coming from Transcribe. MongoDB Atlas Vector Search receives the query vector and returns a document that contains the most semantically similar FAQ in the database. Figure 2: System architecture and modules Here are a couple of FAQs we used for the demo: Q: “Can you explain the different types of coverage available for my home insurance?” A: “Home insurance typically includes coverage for the structure of your home, your personal belongings, liability protection, and additional living expenses in case you need to temporarily relocate. I can provide more detailed information on each type if you'd like.” Q: “What is the process for adding a new driver to my auto insurance policy?" A: “To add a new driver to your auto insurance policy, I'll need some details about the driver, such as their name, date of birth, and driver's license number. We can add them to your policy over the phone, or you can do it through our online portal.” Note that the question is reported just for reference, and it’s not used for retrieval. The actual question is provided by the user through the voice interface and then matched in real-time with the answers in the database using Vector Search. This information is finally presented to the customer service operator in text form (see Fig. 3). The proposed architecture is simple but very powerful, easy to implement, and effective. Moreover, it can serve as a foundation for more advanced use cases that require complex interactions, such as agentic workflows , and iterative and multi-step processes that combine LLMs and hybrid search to complete sophisticated tasks. Figure 3: App interface, displaying what has been asked by the customer (left) and how the information is presented to the customer service operator (right) This solution not only impacts human operator workflows but can also underpin chatbots and voicebots, enabling them to provide more relevant and contextual customer responses. Building a better future for customer service By seamlessly integrating analytical and operational data streams, insurance companies can significantly enhance both operational efficiency and customer satisfaction. Our system empowers businesses to optimize staffing, accelerate inquiry resolution, and deliver superior customer service through data-driven, real-time insights. To embark on your own customer service transformation, explore our GitHub repository and take advantage of the Dataworkz free tier .

November 27, 2024