Three Major IoT Data-Related Challenges and How to Address Them

IoT has formed itself a crucial component for future-oriented solutions and holds a massive potential of economic value. McKinsey & Company estimates that by 2030, IoT (Internet of Things) will enable $5.5 trillion to $12.6 trillion in value worldwide, including the value captured by consumers and customers. For proof of its growing popularity and consumers’ dependency on it, you likely don't need to look any further than your own wrist. From fitness bands to connected vehicles, smart homes, and fleet-management solutions in manufacturing and retail, IoT already connects billions of devices worldwide, with many more to come.

As more IoT enabled devices come online, with increasingly sophisticated sensors, choosing the right underlying technology to make IoT solutions easier to implement and help companies seize new innovative opportunities is essential.

In this blog we will discuss how MongoDB has successfully addressed three major IoT data-related challenges across various industries, including Manufacturing, Retail, Telecommunications, and Healthcare. The challenges are the following:

  • Data Management

  • Real-Time Analytics

  • Supply Chain Optimization

FIgure 1: MongoDB Atlas for IoT

Let's dive right in!

Data management

Storing, transmitting, and processing the large amount of data that IoT devices produce is a significant challenge. Additionally, the data produced by IoT devices often comes in variable structures. This data must be carefully timestamped, indexed, and correlated with other data sources to make the context required for effective decision-making. This data volume and complexity combination makes it difficult to effectively and efficiently process data from IoT devices.

Bosch

Consider Bosch IoT Suite, a family of products and services in IoT device management, IoT data management, and IoT edge by Bosch Digital. These products and services hold over 250 international IoT projects and over 10 million connected devices.

Bosch implemented MongoDB to store, manage, and analyze data in real time. MongoDB’s ability to handle structured, semi-structured, and unstructured data, and efficient data modeling with JSON make it easy to map the information model of each device to its associated document in the database. In addition, dynamic schemas support agile development methodologies and make it simple to develop apps and software. Adding new devices, sensors, and assets is easy, which means the team can focus on creating better software.

ThingSpace

Another example is that of ThingSpace, Verizon’s market-leading IoT connectivity management platform, which provides the network access required to deliver various IoT products and services. Verizon works with companies that purchase network access from it to connect their devices, bundled together with their own solutions, which they sell to end users. ThingSpace’s customers each sell an IoT product that needs reliable connectivity to ensure the devices always work, which WiFi cannot offer.

Verizon’s monolithic RDBMS-based system would not be able to scale to handle both transactional and time-series workloads, so Verizon decided it needed a distributed database architecture going forward. MongoDB proved to be the only solution that scaled to meet Verizon’s requirements across different use cases and combinations of workload types. The immense processing needs resulting from the high number of devices and high velocity of messages coming in were only addressed by MongoDB’s highly available, scalable architecture. Native MongoDB Time Series allow for improved performance, through optimized storage with clustered indexes and optimized Time-Series query operators.

MongoDB's advanced capabilities, such as flexible data modeling, powerful indexing and Time Series provide an effective solution for managing the complex and diverse data generated by IoT devices.

Real-time analytics

Real-time data analytics, one of the most crucial parts of big data analytics today, brings value to businesses for making more data-driven real-time decisions. However, despite its importance, very few can respond to changes in data minute by minute or second by second. Many challenges arise when it comes to the implementation of real-time analytics for enterprises. Storing such a huge volume of data and analyzing it in real time is an entirely different story.

Thermo Fisher Cloud

Let’s consider the Thermo Fisher Cloud, one of the largest cloud platforms for the scientific community on AWS. MS Instrument Connect allows Thermo Fisher customers to see live experiment results from any mobile device or browser. Each experiment produced millions of "rows" of data, which led to suboptimal performance with existing databases. Internal developers needed a database that could easily handle a wide variety of fast-changing data. MongoDB's expressive query language and rich secondary indexes provided the flexibility to support both ad-hoc and predefined queries customers needed for their scientific experiments.

Anytime I can use a service like MongoDB Atlas, I’m going to take that so that we at Thermo Fisher can focus on what we’re good at, which is being the leader in serving science.

Joseph Fluckiger, Sr. Software Architect @Thermo Fisher

MongoDB Atlas scales seamlessly and is capable of ingesting enormous amounts of sensor and event data to support real-time analysis for catching any critical events or changes as they happen. That gives organizations new capabilities, including:

  • Capturing streaming or batch data of all types without excessive data mapping

  • Analyzing data easily and intuitively with a built-in aggregation framework

  • Delivering data insights rapidly and at scale with ease

With MongoDB organizations can optimize queries to quickly deliver results to improve operations and drive business growth.

Supply chain optimization

Items move through different locations in the supply chain, making it hard to maintain end-to-end visibility throughout their journey. The lack of control on any stage can dramatically harm the efficiency of planning, slow down the entire supply chain and ultimately result in lower return on investment. From optimizing warehouse space by sourcing raw materials as needed, to real-time supply chain insights, IoT-enabled supply chains can help significantly optimize these processes by eliminating blind spots and inefficiencies.

Longbow Advantage

Longbow Advantage delivers substantial business results by enabling clients to optimize their supply chains. Millions of shipments move through multiple warehouses every day, generating massive quantities of data throughout the day that must be analyzed for real-time visibility and reporting. Its flagship warehouse visibility platform, Rebus, combines real-time performance reporting with end-to-end warehouse visibility and intelligent labor management.

Longbow needed a database solution that could process quantities of that scale and deliver real-time warehouse visibility and reporting at the heart of Rebus, and it knew it could not rely on monolithic, time-consuming spreadsheets to do so. It became clear that MongoDB’s document database model was a good match and would allow Rebus to gather, store, and build visibility into disparate data in near real time.

Another key component of smart supply chain solutions is IoT-enabled mobile apps that provide real-time visibility and facilitate on-the-spot data-driven decisions. In such situations, the offline first paradigm becomes crucial, since staff need access to data in areas where connectivity is poor or nonexistent. Realm by MongoDB is a lightweight, object-oriented embedded database technology for resource constrained environments. It is an ideal solution for storing data on mobile devices. By utilizing MongoDB’s Realm SDKs, which wrap the Realm database, and Atlas Device Sync, which enables seamless data synchronization between MongoDB and Realm on your mobile phone with minimal developer efforts, businesses can rapidly develop mobile applications and drive innovation.

MongoDB provides a powerful solution for IoT-enabled supply chains that can optimize processes and eliminate inefficiencies, enabling organizations to make data-driven decisions and improve supply chain efficiency.

Conclusion

The IoT industry is rapidly evolving, and as the number of connected devices grows, so do the challenges faced by businesses leveraging these solutions.

Through a range of real-world use cases, we have seen how MongoDB has helped businesses deal with IoT data management, perform real-time analytics and optimize their supply chains, driving innovation in a variety of industries. With its unique features and capabilities, designed to manage the heavy lifting for you, MongoDB is well-positioned to continue playing a crucial role in the ongoing digital transformation of the IoT landscape.

Want to learn more or get started with MongoDB? Check out our IoT resources

← Previous

Bosch IoT and the Importance of Application-Driven Analytics

>> Announcement: Some features mentioned below will be deprecated on Sep. 30, 2025. Learn more . This post is also available in: 简体中文 For Kai Hackbarth and his team at Bosch, there exists the ability to bring real-time analytics into their applications that can handle lots and lots of data. “Without data, it’s just trial and error.” So says Kai Hackbarth . He’s a senior technology evangelist at Bosch Global Software with over 22 years of industry experience in the Internet of Things. “Be it automotive, be it industrial, be it smart-home, we’ve done it all,” Hackbarth said. Except for tires. That’s his latest focus and technology challenge. “Sounds maybe a bit simple,” Hackbarth said, “but if you think about it more deeply, [it’s really complex].” Because, as it turns out, tires can collect a lot of different pieces of data that can tell a system a lot about what’s going on with a lot of different things related to the car at any given moment. “Pressure, temperature, accelerometer,” Hackbarth said. “And then you also have other data from the car that’s critical for safety and sustainability.” But to be of any value, that data needs to be analyzed as close to the source as possible and in real time. Why? “It’s safety-critical,” Hackbarth said. “If you send all the raw data to the cloud this consumes a lot of costs.” Chief among those costs: Time. In order to react to an issue, that data cannot be historical. Because historical data about a tire that’s losing pressure or hydroplaning isn’t helpful to the applications inside the car that need to respond to these developments when they’re happening And thankfully for Hackbarth and his team at Bosch, there exists the ability to bring real-time analytics into their applications that can handle lots and lots of data. Smarter applications start with built-in analytics Traditionally, applications and analytics have existed as two separate classes of workloads with different requirements, for example: read and write access patterns, as well as concurrency and latency. As a result, businesses have usually deployed purpose-built data stores — including both databases for applications and data warehouses for analytics — and piping or duplicating the data between them. And that’s been fine when analytics don’t need to affect how an application responds in real time. But most customers expect applications to take intelligent actions in the moment, rather than after the fact. The same principle applies to Bosch’s tire project. The applications inside the car that can autonomously brake when approaching another vehicle too fast or slow down if the tire senses that it’s hydroplaning, need to also analyze all the data from all the sensors in real time. This process of building real-time analytics into applications is known as “ application-driven analytics .” And it’s how applications get smarter. Be they e-commerce apps on your phone, safety apps in your car or those that monitor rocket launches . The question for many development teams, though, is how do you build this capability into your applications easily? For a long time, that’s been a difficult question to answer. A platform for building real-time analytics into your apps “From my history [of doing this for] 22 years,” Hackbarth says, “we never had the capabilities to do this before.” Previously, teams everywhere — not just at Bosch — used to have to do a lot of custom engineering work to do real-time analytics close to the source, including: Stitching together multiple databases to handle different data structures (documents, tables, time series measurements, key values, graph), each accessed with its own unique query API. Building ETL data pipelines to transform data into required analytics formats and tier it from the live database to lower-cost object storage . Spinning up a federated query engine to work across each data tier, again using its own unique query API. Integrating serverless functions to react to real-time data changes. Standing up their own API layers to expose data to consuming applications. All of which result in a multitude of operational and security models to deal with, a ton of data integration work and lots of data duplication. But that’s no longer the case. There are now data platforms that bring together operational and analytical workloads into one, and that, in turn, allow you to bring live, operational data and real-time analysis together. MongoDB Atlas — the platform with which I am most familiar (since I work at MongoDB) — allows developers to do this by providing an integrated set of data and application services that fit their workflows. Developers can land data of any structure, index, query and analyze it in any way they want, and then archive it. And they can do all of this with a unified API and without having to build their own data pipelines or duplicate data. This is the platform on which the Bosch team continues to build its solutions. “Without data,” Hackbarth says, “it’s just trial and error.” But now with data and with a single platform to build real-time analytics into their applications, it’s something concrete, something responsive and something actionable. It’s something smart. Especially in the most critical of use cases that Bosch is working on. Like tires. If you’d like to learn more about how to build application-driven analytics into your applications, check out our three-part livestream demo showing how to build application-driven analytical applications in MongoDB Atlas . During this three-part series, we build real-time analytical capabilities for a simulated system for managing rocket launches. Part One covers the basics: Building complex analytical queries using MongoDB’s aggregation framework and building visualizations using charts. Part Two highlights some additional Atlas capabilities that are often invaluable when building app-driven analytics: Atlas search, triggers and embedding Charts generated visualizations into an application UI. Part Three focuses on how to use Atlas Data Federation, Atlas Data Lake and Atlas SQL Interface to perform analytics using large sets of historical data and federated queries across multiple data sources.

April 19, 2023
Next →

Hanabi Technologies Uses MongoDB to Power AI Assistant, Hana

For all the hype surrounding generative AI, cynics tend to view the few real-world implementations as little more than “fancy chatbots.” But for Abhinav Aggarwal, CEO of Hanabi Technologies, the idea of a generative AI-powered bot that is more than just an assistant was intriguing. “I’d been using ChatGPT since it launched,” said Aggarwal. “That got me thinking: How could we make a chatbot that was like a team member?” And with that concept, Hana was born. The problem with bots “Most generative AI chatbots do not act like people; they wait for a command and give a response,” said Aggarwal. “We wanted to create a human-like chatbot that would proactively help people based on what they wanted—automating reminders, for example, or fetching time zones from your calendar to correctly schedule meetings.” Hanabi’s flagship product, Hana, is an AI assistant designed to enhance team collaboration within Google Chat, working in concert with Google Workspace and its suite of products. “Our target customers are smaller companies of between 10 and 50 people. At this size you’re not going to build your own agent from scratch,” he said. Hana integrates with Google APIs to deliver a human-like assistant that chimes in with helpful interventions, such as automatically setting reminders and making sure meetings are booked in the right time zone for each participant. “Hana is designed to bring AI to smaller companies and help them collaborate in a space where they are already working—Google Workspace,” Aggarwal explained. The MongoDB Atlas solution For Hana to act like a member of the team, Hanabi needed to process massive amounts of data to support advanced features like retrieval-augmented generation (RAG) for better information retrieval across Google Docs and many other sources. And with a rapidly growing user base of over 600 organizations and 17,000+ installs, Hanabi also required a secure, scalable, and high-performing data storage solution. MongoDB Atlas provided a flexible document model, built-in vector database, and scalable cloud-based infrastructure, freeing Hanabi engineers to build new features for Hana rather than focusing on rote tasks like data extract, transform, and load processes or manual scaling and provisioning. Now, MongoDB Atlas handles a variety of responsibilities: Scalability and security: MongoDB Atlas’s auto-scaling and automatic backup features have enabled Hanabi to seamlessly grow its user base without the need for manual database management. RAG: MongoDB Atlas plays a critical role in Hana’s RAG functionality. The platform enables Hanabi to split Google Docs into small sections, create embeddings, and store these sections in Atlas’s vector database. Development Processes: According to Aggarwal, MongoDB’s flexibility in managing changing schemas has been essential to the company’s fast-paced development cycle. Data Visualization: Using MongoDB Atlas Charts has enabled Hanabi to create comprehensive dashboards for real-time data visualization. This has helped the team track usage, set reminders, and optimize performance without needing to build a manual dashboard. Impact and results With MongoDB Atlas, Hanabi can successfully scale Hana to meet the demands of its rapidly expanding user base. The integration is also enabling Hana to offer powerful features like automatic interactions with customers, advanced information retrieval from Google Docs, and manually added memory snippets, making it an essential tool for teams around the world. Next steps Hanabi plans to continue integrating more tools into Hana while expanding its reach to personal Gmail users. The company is also rolling out a new automatic-interaction feature, further enhancing Hana’s ability to proactively assist users without direct commands. MongoDB Atlas remains a key component of Hanabi’s stack, alongside Google Kubernetes Engine, NestJS, and LangChain, enabling Hanabi to focus on innovating to improve the customer experience. Tech Stack MongoDB Atlas Google Kubernetes Engine NestJS LangChain Are you building AI apps? Join the MongoDB AI Innovators Program today! Successful participants gain access to free MongoDB Atlas credits, technical enablement, and invaluable connections within the broader AI ecosystem. If your company is interested in being featured, we’d love to hear from you. Connect with us at ai_adopters@mongodb.com.

November 21, 2024