Real-Time Energy Monitoring for Smart Buildings with MongoDB and HiveMQ

The Internet of Things (IoT) has ushered in a new era of energy efficiency, enabling the deployment of energy-efficient sensors for energy conservation and resource utilization. With over 1.5 billion connected IoT devices already installed in commercial smart buildings in 2022 and a projected surge to 3.25 billion devices by 2028, the volume of data generated is staggering. To put it in perspective, an average home in 2020 would generate approximately 4.7 terabytes of data annually. However, managing and harnessing this immense amount of real-time streaming data poses a significant challenge for developers. In smart buildings, where a multitude of IoT sensors continuously gather event streaming data, developers often grapple with integrating disparate technologies and investing significant time into data streaming integration.

In this blog, we present a simple yet powerful solution to this challenge. We will demonstrate how you can effortlessly move IoT data using standard protocols such as MQTT to MongoDB Atlas using the HiveMQ Enterprise Extension for MongoDB. By optimizing smart buildings with real-time energy monitoring through the seamless integration of MongoDB and HiveMQ, we unlock the potential for efficient energy management and a sustainable future.

Let’s get started!

Dream team: HiveMQ + MongoDB

In a world where energy conservation and efficient resource utilization are essential, let’s go through Figure 1 and see how simple it is to use MongoDB, HiveMQ’s MQTT broker, and the Enterprise Extension for MongoDB to enable real-time energy monitoring for smart building.

Figure 1: Combining HiveMQ and MongoDB process data in real-time

Step 1: Data transmission

Using MQTT-based IoT devices deployed throughout the building, electricity consumption, temperature, and occupancy data is collected and sent to the HiveMQ MQTT broker. The MQTT broker acts as a central hub, efficiently and securely handling the communication between devices and backend systems. The HiveMQ MQTT broker also ensures reliable message delivery. It also provides MQTT-specific features like quality of service, session management, and topic-based message routing.

Step 2: Data ingestion

The HiveMQ MongoDB extension seamlessly integrates with MongoDB, allowing for persistent storage of the MQTT data in a highly scalable and flexible manner. The fully customizable templating system allows MQTT data to be stored according to the building’s specific operational requirements. MongoDB's document-based model accommodates the varying data formats and structures generated by different IoT devices.

Step 3: Data visualization and analytics

Once the MQTT data is securely stored in MongoDB, using its powerful in-app analytics, building managers can gain deep insights into energy consumption patterns, identify anomalies, and optimize energy usage. By leveraging MongoDB's rich query support and aggregation framework, building managers can make data-driven decisions promptly, reducing costs and enhancing sustainability. In cases where data needs to be exported to an ML/AI engine, MongoDB Spark and Kafka connectors can be used. Users of MongoDB Atlas can leverage Atlas Device Sync and Realm to send real-time alerts and messages to mobile devices. Data can be visualized using MongoDB Atlas Charts or through a third-party Business Intelligence (BI) tool connected via MongoDB BI connector or Atlas SQL interface.

Conclusion

By seamlessly integrating HiveMQ's MQTT broker with MongoDB, developers can efficiently handle data transmission, ingestion, and storage. This integration enables building managers to gain valuable insights into energy consumption patterns, make data-driven decisions, and optimize energy usage.

To learn more about MongoDB’s role in IoT, please visit our IoT webpage. You can also try the HiveMQ platform now with the Enterprise Extension for MongoDB for free.

Thank you Ainhoa Múgica for her contributions to this blog.

← Previous

How Telcos Drive Mission-Critical Innovation and Cost Savings Through Automation

Modern telecommunications customers don’t only expect flawless network performance, they demand it. For global enterprises that hold the key to human connection, the industry standard is nothing less than a fully integrated, customer-centric approach to service delivery. How are telecommunications firms tackling these customer expectations today? By embracing AI and machine learning capabilities, developing new decisioning models, and ensuring network security and optimization with a 99.995% uptime SLA. Why automation is vital in the telco modernization journey AI adoption is growing at a break-neck speed, and the telecommunications industry has a close eye on the way that automated decision-making can improve operational efficiency and reduce costs. In the latest TM Forum Digital Transformation Tracker 7 survey , 73% of respondents agreed that operational efficiency and cost reduction were very important drivers of CSPs’ digital transformation journeys, compared with 69% in 2022 and 65% in 2018. By eliminating manual tasks, and reducing errors introduced by manual intervention, automation is improving end-to-end performance and reducing handoffs and touchpoints. In the case of automated decision-making, it’s possible to leverage large volumes of data that already exist within telco organizations, alongside data science and machine learning techniques, to generate data-driven insights and inferences to better serve customers and develop cost savings. Three automation use cases in the telecommunications industry Here are three ways that MongoDB’s customers in the telecommunications industry can innovate with automation: Service assurance processes: Telcos can proactively identify issues impacting customers, or even predict them before they occur, utilizing automated processing of large amounts of diverse data. Network automation techniques can then step in to automatically remediate the situation, and output intent that can be processed by intent-based network automation tooling. Network automation: While service assurance processes can make decisions around what needs to be done, network automation tooling takes responsibility for effecting change. What’s more, the demands of 5G networks will force operators to open up traditional closed systems to third parties via network APIs. For example, these APIs can allow automated provisioning and configuration of 5G network slices. Customer expectations here will be that these operations are self-serve, and will happen in real time, making automation key. Customer issue management: We’ve all come across chatbots used in B2C customer service experiences. As chatbots become more sophisticated, potentially leveraging modern generative AI techniques, many more customer care issues will be automated. This change will not only reduce the cost of call centers, but speed up mean time to resolution for customer care issues. The future of AI/ML-based automated decision-making in telecommunications We’ve established the importance of automation for service assurance, network automation, and customer experiences. By utilizing the power of AI and data science, telcos have the opportunity to take these technologies further into the realm of network security and fraud mitigation. However, getting to automation with unstructured data is no simple task. Studies show that more than 50% of data scientists’ time is spent wrangling data, and more than 80% of all essential data is unstructured. Figure 1: Data processing operations for machine learning models. To build out a new AI/ML-based system, both data processing and ML capabilities must be in place. These two solutions are typically provided by different systems with clear integration points. Any AI/ML-based solution requires large amounts of historical data to train the model. The storage and feeding of this data is usually the job of a traditional data warehouse system. This gets complicated when decisions need to be made in real time with “live” data, such as in anti-fraud use cases. To achieve real-time results, it’s required to integrate an operational database into the architecture to stream real-time data and requests into the model and to persist the model output. In this hybrid system example, we have both operational and analytics data requirements co-existing; this interaction adds to the overall architectural complexity of the system. It’s important to note that raw data cannot be used by AI models “as it.” First the data must be cleaned, potentially deduplicated and turned into features. Standardized techniques are required to do this, including binning, normalization, standardization, and one-hot encoding. The aggregation pipeline provides powerful data processing capabilities that assist with this process. MongoDB Atlas, the developer data platform, is capable of handling each of the above requirements from a single platform. Its analytical nodes and Data Lake allow for massive amounts of historical data storage, and service to the model for training purposes. Figure 2: MongoDB's role in machine learning pipeline. Real-time data can also be ingested and served through MongoDB Atlas via change streams, triggers, and integrations. The powerful aggregation framework of MongoDB is capable of transforming raw data into usable features. Lastly, integration patterns based on Spark, Kafka, and HTTP are supported out of the box, which greatly reduces overall architectural complexity. Once decision-making models produce the new data output, it can be persisted back into the transactional database automatically actioned by additional automation tooling. Decision-making models will constantly evolve and will require new facets of data to be used. MongoDB’s document model and dynamic schema naturally supports the ingestion of new types and formats of data without the need for complex schema changes. > Join MongoDB, Hansen Technology and TM Forum for a live discussion on the future of AI in the telecommunications industry.

July 10, 2023
Next →

MongoDB and Partners: Building the AI Future, Together

If you’re like me, over the past year you’ve closely watched AI’s developments—and the world’s reactions to them. From infectious excitement about AI’s capabilities, to impatience with its cost and return on investment, every day has been filled with AI twists and turns. It’s been quite the roller coaster. During the ride, from time to time I’ve wondered where AI falls on the Gartner hype cycle, which gives "a view of how a technology or application will evolve over time." Have we hit the "peak of inflated expectations" only to fall into the "trough of disillusionment?" Or is the hype cycle an imperfect guide, as The Economist argues? The reality is that it takes time for any new technology—even transformative ones like AI—to take hold. And every advance, no matter how big, has had its detractors. A famous example is that of Picasso (!), who in 1968 said, “Computers are useless. They can only give you answers.” (!!) For our part, MongoDB is convinced that AI is a once-in-a-generation technology that will enhance every future application—a belief that has been reinforced by the incredible work our partners have shared at MongoDB’s 2024 events. Speeding AI development MongoDB is committed to helping organizations of all sizes succeed with AI, and one way we’re doing that is by collaborating with the MongoDB partner ecosystem to create powerful, user-friendly AI development tools and solutions. For example, Fireworks.ai —which is a member of the MongoDB AI Applications Program ecosystem —created an inference solution that hosts gen AI models and supports containerized deployments. This tool makes it easier for developers to build and deploy powerful applications with a range of easy-to-use tools and customization options. They can choose to use state-of-the-art, open-source language, image, and multimodal foundation models off the shelf, or they can customize and fine-tune models to their needs. Jointly, Fireworks.ai and MongoDB provide a solution for developers who want to leverage highly curated and optimized open-source models and combine these with their organization’s own proprietary data—and to do so with unparalleled speed and security. “MongoDB is one of the most sophisticated database providers, and it’s very easy to use,” said Benny Chen , cofounder of Fireworks.ai. "We want developers to be able to use these tools, and we want to work with providers who enable and empower developers." Nomic , another MAAP ecosystem member, also enables developers with best-in-class solutions across the entire unstructured data workflow. Their Embed offering, available through the Nomic API , allows users to vectorize large-scale datasets for use in text, image, and multimodal retrieval applications, including retrieval-augmented generation (RAG), using only their web browser. The Nomic-MongoDB solution is a highly efficient, open-weight model that developers can use to visualize the unstructured datasets they store in MongoDB Atlas . These insights help users quickly discover trends and articulate data-driven value propositions. Nomic also supported the recently announced vector quantization in MongoDB Atlas Vector Search , which reduces vector sizes while preserving performance. Last—but hardly least!—there’s our new reference architecture with MAAP partners AWS and Anthropic. Announced at MongoDB.local London , the reference architecture supports building memory-enhanced AI agents, and is designed to streamline complex processes and develop smarter, more responsive applications. For more—including a link to the code on Github— check out the MongoDB Developer Center . Making AI work for anyone and everyone The companies MongoDB partners with aren’t just making gen AI easier for developers—they’re building tools for everyone. For example, Capgemini has invested $2 billion in gen AI and is training 100,000 of its employees in the technology. GenYoda, a solution that helps insurance professionals with their daily work, is a product of this investment. GenYoda leverages MongoDB Atlas Vector Search to analyze large amounts of customer data, like policy statements, premiums, claims history, and health information. Using GenYoda, insurance professionals can quickly analyze underwriters’ reports to make informed decisions, create longitudinal health summaries, and streamline customer interactions to improve contact center efficiency. GenYoda can ingest 100,000 documents in just a few hours and respond to users’ queries in two to three seconds—a metric on par with the most widely used gen AI models. And it produces results: in one example, by using Capgemini’s solution an insurer was able to increase productivity by 15%, add new reports 25% faster (thus speeding decision-making), and reduce the manual effort of searching PDFs, increasing efficiency by 10%. Building the future of AI together So, what’s next? Honestly, I’m as curious as you are. But I’m also incredibly excited. At MongoDB, we’re active participants in the AI revolution, working to embrace the possibilities that lie ahead. The future of gen AI is bright, and I can’t wait to see what we’ll build together. To learn more about how MongoDB can accelerate your AI journey, explore the MongoDB AI Applications Program .

November 4, 2024