Skunkworks 2022: A Week of Building for MongoDB Engineers

MongoDB’s 2022 internal Skunkworks hackathon wrapped up in July, and it was a bustling time of hacking, building, and developing. For MongoDB engineers, Skunkworks is a week of no meetings and no interviews—engineers can have fun and work alongside co-workers from different teams to build side projects, proof of concepts, or anything else imaginable.

Many companies do one- or two-day hackathons, maybe once per year. We do one-week hackathons about every eight to ten months. Why? Hackathons are an important part of our engineering culture and embody our values of “Think Big, Go Far” and “Build Together.” We find that dedicating an entire week to the hackathon leads to significant innovation.

With more than 115 projects submitted, this year’s Star Trek-themed event was one of the best and largest hackathons yet. Building from some of the newest MongoDB 6.0 features, let’s look at a few winning projects and the people behind them.

Charts Slack Integration

Team: James Wang, Ryan Nguyen, Andrew McMenemy, and Muthukrishnan Krishnamurthy

Image containing the Slack and MongoDB Charts logos

We work on the Charts team and genuinely love our product; plus, it’s always fun taking complete ownership of it for a week. Reporting is a big ticket item for any data visualization tool. It’s something we’ve always had in the back of our minds, and users seem really keen on it. Variations of this request have been asked on our User Voice page since 2019, along with a request for a Slack integration with Charts since 2020.

This feature is built on top of our Embedded Charts SDK. We updated the URL used to fetch an embedded Chart/Dashboard to take on a new query parameter for screen grabbing. This new URL will trigger an AWS Lambda instance we wrote for the project, which will make use of our embedding SDK to embed and take a screenshot of the chart.

Image displaying charts hourly customer interaction utilizing different tables

Skunkworks is our favorite time of the year. Learning software through projects is so much easier said than done, especially once you start a full-time career in software. MongoDB’s hackathon improves us as engineers, provides a break from the standard work week, and allows us to work on whatever we want, whether it’s a solo project or helping to make someone's dream a reality. It inspires us to always be looking for the next best idea, along with it being a great mental health week.

MongoDB Carbon Footprint Calculation

Team: Nellie Spektor, Maya Raman, Cathy Wang, Rohan Chhaya, and Tiffany Feng

Our team was inspired by a previous Skunkworks hackathon project focused on sustainability within MongoDB Atlas. We decided to do a deep dive on carbon footprint measurement within MongoDB. We began by investigating the carbon efficiency of various parts of MongoDB, from drivers to Atlas.

For testing Atlas, we first set up multiple clusters, each varying in either location, cluster size, and sharding status. Then, using an atlas-co2 calculator script that a MongoDB Developer Advocate made, we were able to test how much carbon each cluster was emitting. Finally, we were able to display our findings about different cluster sizes and cloud providers and their carbon footprints.

While investigating drivers, we tested 100 insert/find/update/delete operations on 7 of our drivers and calculated the time taken and the wattage used, which was used to calculate carbon footprint and rank the drivers in terms of efficiency.

The biggest takeaway for us was the sheer difference in carbon emissions that a simple choice can make. For example, using the Rust driver instead of Java uses 144 times less electricity and therefore emissions. Simply shifting your cluster from one cloud provider in the Virginia region to a different cloud provider in the Iowa region saves over 3kg of carbon a week.

MongoDB is collaborating with a third-party vendor to get more robust carbon emission calculations, while tangentially understanding how we can provide a more efficient, sustainable product. Efforts like these help us to better refine our overall corporate emissions calculations.

The Skunkworks hackathon is a great initiative that shows engineers how much MongoDB values creativity and personal growth. It's amazing that we are actually encouraged to put aside our regular work and try out anything we’d like. While some people work on personal development projects, other people take the opportunity to tackle some tech debt or explore new features without the constraints of the normal product development process. Furthermore, it allows us to integrate our outside interests into our work, which makes us even more passionate and motivated.

Atlas Static Site Search

Team: Ben Perlmutter, Joon Young Lee, Shibi Balamurugan, Marcus Eagan, and Nick Larew

Our project was inspired by Algolia DocSearch. They’ve done a great job making it super easy to add search to a website. We wanted to make something similar with the MongoDB Atlas developer data platform, and we knew Atlas had the tools we needed to make this possible. It was just a question of writing some code to connect these services and creating a streamlined developer experience.

We called the project Atlas Static Site Search, and it has the following distinct components:

  • A website scraper that pulls site data and adds it to MongoDB built using Atlas Triggers.

  • A search index built with Atlas Search using the site data.

  • An Atlas Function that queries Atlas Search.

  • A React component that you can add to a website that uses the Realm Web SDK to call the Atlas Function that performs search.

  • A CLI that you can use to set up the whole backend (site scraper, search index, and search query function) with one command.

Image displaying the Atlas Static Site Search Architecture

The biggest challenge was getting all the different cloud services to work together well. Since it is a one week hackathon, we didn’t have time to write proper integration tests to validate that things were working as expected before deploying them. There was a lot of deploying code, praying it would work, it not working, and hotfixing.

The Skunkworks hackathon is one of our favorite parts of working at MongoDB. Whether you are an intern or an experienced engineer, you’re given the freedom to work on a passion project or learn something new. It speaks to the respect that MongoDB leadership has for the engineering and product teams. A lot of great ideas and innovative products have come out of hackathons in the past. We’ll see what happens with Atlas Static Site Search!

Simulating Common Customer Workloads

Team: Xiaochen Wu, Kyle Suarez, and Nishith Atreya

Our team recognized how replicating customer workloads has a myriad of benefits and can directly or indirectly help build a better testing environment, empower our support team, and identify potential improvement opportunities in our own product portfolio.

We tried to replicate two different workloads—one transactional and one in-app analytical. After identifying these workloads, we brainstormed how to use MongoDB features and products to support them. Then, we identified important characteristics of each workload and began replicating them using available datasets. Following this, we monitored how each workload performed in the MongoDB platform and collected insights and recommendations for our internal teams.

One of the biggest challenges was trying to figure out the important characteristics of each workload we were trying to replicate. For example, it was more difficult than expected to think of the most commonly used queries, search, and recommendation patterns that would appropriately represent the transactional customer workload.

After the completion of our project, we created a recommendation for MongoDB to build a workload suite consisting of workloads that cover a variety of customer industries. This would allow our engineering organization to test major upgrades, perform product research, and identify improvement opportunities in our platform.

$semanticSearch Aggregation Stage

Team: Thomas Rueckstiess and Steve Liu

At MongoDB Labs, we're always exploring how new technologies can be integrated with MongoDB. We read a paper that was published by a few researchers from Meta describing a novel architecture for semantic search and thought this could be a cool week-long project. We made four key changes:

  1. Built a web service that was the API interface to the model

  2. Introduced an aggregation pipeline called $semanticSearch that communicated with the API

  3. Deployed the web service on AWS

  4. Built a Star Trek themed front end using React

The Skunkworks hackathon helps us explore the creativity of MongoDB engineers. Every project submitted looked well polished and innovative. We walked away inspired by the talent that's evident in the business. Hackathons provide a creative outlet for engineers away from the day-to-day tickets and helps build a meritocratic culture where any project can receive recognition and reward.

Join us for the next hackathon: We’re actively hiring and looking for more talented, creative, and passionate engineers who want to build the next generation of MongoDB products and features!

← Previous

Built With MongoDB: Thunkable Brings the Power of App Development to Non-Developers

Great ideas can come from anywhere. But, if you have a great idea for a mobile app, you won't get far without developer talent. In today's app market, there are so many ideas in various stages of development that developer talent has become scarce and costly. To fill the gap, low-code and no-code solutions have emerged. Thunkable is a no-code platform that makes it easy to build custom native mobile apps without any advanced software engineering knowledge or certifications. The platform has seen tremendous growth, recently expanding to more than three million users. MongoDB has been pivotal to that growth, enabling the company to offer its services at scale without having to worry about managing the database. Thunkable is also part of the MongoDB for Startups program, which has helped the company solve some of the technical hurdles involved with scaling to millions of users. App creation for everyone Just because you don't have a computer science degree doesn't mean you can't come up with a great idea for an app. Thunkable co-founder and CTO, Wei Li, says the company is on a mission to democratize mobile app development. "We are currently empowering more than three million users across the globe that can come to our platform to build and publish their apps and do it without writing a single piece of code," Li says. Thunkable uses a simple drag-and-drop design canvas and powerful logic blocks to give innovators the tools they need to breathe life into their app designs. "It's very exciting seeing people from all different backgrounds trying to build solutions," says Jose Dominguez, engineer at Thunkable. "The part that I'm most proud about," says Li, "is our global users. Every day, I hear wonderful stories. For example, we recently have had people using our platform to build mobile apps to coordinate relief efforts in Ukraine. And since the pandemic, we have seen more users coming to our platform to build an app that addresses their needs for work, family, and community." Mobile phones are a transformational technology, but at the same time, there's untapped potential waiting to be exploited. Li believes Thunkable can help unleash the latent power lurking in our back pockets. "There are so many needs people could solve by using their smartphones," Li says. "But because they cannot program their phone, they become passive consumers. We want to empower them to become active creators." Modern app development with MongoDB Thunkable chose MongoDB early on because it mapped to its existing architecture; it stuck with MongoDB because it scaled when it became critical to do so. "We decided to build with MongoDB because it fits our data very naturally," Dominguez says. "We abstract our users' apps as documents, so it's a natural fit. We do a lot of writes so we needed a system to handle those kinds of loads and MongoDB was the perfect fit." Like a lot of startups, Thunkable has had to figure out how to achieve its goals with limited resources. Its engineering team consists of about four to six developers. "The engineering team has always been focused on building the product," Dominguez says. "So not having to worry about the database was a great win for us. It allowed us to iterate very fast and build new versions of the platform without having to worry about scaling the database or backups." After scaling to three million users, the Thunkable engineering team needed to rethink some of its design decisions about data. So, they talked with MongoDB engineers, courtesy of the startup program. Since then, their data storage needs have decreased while performance has improved. "Our partnership with MongoDB has been fundamental to our growth," Dominguez says. Li concurs: "As we scale, supporting more enterprise customers, we don't have to worry about database management issues. We know MongoDB will do a great job helping us as we scale." Building better together In addition to free MongoDB Atlas credits and one-on-one technical support, participants in the MongoDB for Startups program enjoy co-marketing opportunities and access to our partner network. "It's wonderful to be part of the MongoDB for Startups program," Li says. "We have all the support we need, from database management, MongoDB upgrades, and maintenance. You just go to one portal, one website. It's wonderful." MongoDB technical support has also been a life-saver for Thunkable, says Dominguez. "We had an initial call with an engineer and went through our logs. The engineer spotted a few things that we could fix right away. We were having issues with our op log — the tool that MongoDB uses to replicate to other servers — and the engineers helped get us out of those issues." The MongoDB dashboard is another tool Dominguez cites as being especially helpful. "For us, being able to log in to the dashboard and have all the functionality already there is a life saver," he says. "Not only being able to monitor the cluster, but digging into our collections and seeing if our indices are performing properly gives us tremendous insight. The integration of MongoDB and Google Cloud makes our infrastructure much easier to maintain." Future integrations Although they've scaled to millions of users, the team at Thunkable isn't finished. "This upcoming year will be a big one for Thunkable," Li says. "We're going to use our recent funding to scale up our support for enterprise, mainly for team support, collaboration, and enterprise design integration. For example, we recently integrated with Figma so you can import your Figma design into the Thunkable platform and have a functional app." "We're also currently hiring," Li adds. "We're planning to double our team within the next six months." To learn more about Thunkable or if you want to get started building your own mobile apps, check out Thunkable to get started for free . Are you part of a startup and interested in joining the MongoDB for Startups program? Apply now .

October 12, 2022
Next →

MongoDB Database Observability: Integrating with Monitoring Tools

This post is the final in a three-part series on leveraging database observability. Welcome back to our series on Leveraging Database Observability! Our previous post showcased a real-world use case highlighting how MongoDB Atlas’s observability tools effectively tackle database performance challenges. Whether you’re a developer, DBA, or DevOps engineer, our mission is to empower you to harness the full potential of your data through our observability suite . Integrating Atlas metrics with your central enterprise observability tools can simplify your operations. By seamlessly working with popular observability tools, our approach helps teams streamline workflows and enhance visibility across systems. Integrating MongoDB Atlas with third-party monitoring tools MongoDB’s developer data platform combines all essential data services for building modern applications within a unified experience. Our purpose-built observability tools for Atlas environments offer automatic monitoring and optimization, guiding diagnostics tailored specifically for MongoDB. Additionally, we extend Atlas metrics into your existing enterprise observability stack, enabling seamless integration without replacing your current tools. This creates a consolidated, single-pane view that unifies Atlas telemetry with other tech and application metrics, ensuring comprehensive visibility into both database and full-stack performance. This integration empowers you to monitor, receive alerts, and make data-driven decisions within your existing workflows, driving greater efficiency. Below is a quick guide to modifying integration settings through the Atlas UI and the popular integrations we support: Navigate to the Project Integrations page in Atlas. Choose the organization and project you want to configure from the navigation bar. On the Project Integrations page, select the third-party services you’d like to integrate. Configure the chosen services with the required API keys and regions. Critical integrations for your observability platform With Atlas’s Datadog and Prometheus integrations, you can send critical MongoDB metrics to these platforms, empowering detailed, real-time monitoring. Through Datadog , you can track database operation counts, query efficiency, and resource usage, ideal for pinpointing bottlenecks and managing resources. Similarly, Prometheus enables you to monitor essential metrics like query times, connection rates, and memory usage, supporting flexible tracking of database health and performance. Both integrations facilitate proactive detection of issues, alert configuration for resource thresholds, and a cohesive view of Atlas data when visualized in Grafana. Atlas’s integration with PagerDuty streamlines incident management by sending metrics like performance alerts, billing anomalies, and security events directly to PagerDuty. This integration records incidents automatically, notifies teams upon alerts, and supports two-way syncing, ensuring resolved alerts in Atlas are reflected in PagerDuty. It enables efficient incident response and resource allocation to maintain system stability. With Atlas integrations for Microsoft Teams and Slack, you can route key metrics—such as query latency, disk usage, and throughput—to these channels for timely updates. Teams can use these insights for real-time performance monitoring, incident response, and collaboration. Notifications through these platforms ensure your team stays informed on database performance, storage health, and user activity changes as they occur. Use case: Centralized observability with MongoDB Atlas, Datadog, and Slack Let’s walk through a hypothetical scenario for ShopSmart, an e-commerce company that leverages MongoDB Atlas to manage its product catalog and customer data. As traffic surges, the DevOps team faces challenges in monitoring application performance and database health effectively. To tackle these challenges, the team leverages MongoDB Atlas’ integration with Datadog and Slack, creating a powerful observability ecosystem. Integrating MongoDB Atlas with Datadog: The team pushes key MongoDB Atlas metrics into Datadog, such as query performance, connection counts, and Atlas Vector Search metrics. With Datadog, they can visualize these metrics and correlate overall MongoDB performance with their other applications. Out-of-the-box monitors and dedicated dashboards allow the team to track metrics like throughput, average read/write latency, and current connections. This visibility helps pinpoint bottlenecks in real time, ensuring optimal database performance and improving overall application responsiveness. Setting up alerts in Datadog: The team configures alerts for critical metrics like high query latency and increased error rates. When thresholds are breached, Datadog instantly notifies the team. This proactive approach allows the team to address potential performance issues before they impact customers. Integrating Datadog with Slack: To ensure fast communication, alerts are sent directly to the dedicated Slack channel, “ShopSmart-Alerts.” This integration fosters seamless collaboration, enabling the team to discuss and resolve issues in real-time. With these integrations, ShopSmart’s engineering team can monitor performance quickly and address issues efficiently. The unified observability approach enhances operational efficiency, improves the customer experience, and supports ShopSmart’s competitive edge in the e-commerce industry. By leveraging MongoDB Atlas, Datadog, and Slack, the team ensures scalable performance and drives continuous innovation. Conclusion MongoDB Atlas empowers developers and organizations to achieve unparalleled observability and control over their database environments. By seamlessly integrating with central enterprise observability tools, Atlas enhances your ability to monitor performance metrics and ensures you can do so within your existing workflows. This means you can focus on building modern applications confidently, knowing you have the insights and alerts necessary to maintain optimal performance. Embrace the power of MongoDB Atlas and transform your approach to database management—because your applications can thrive when your data is observable. And that wraps up our Leveraging Database Observability series! We hope you learned something new and found value in these discussions. Sign up for MongoDB Atlas , our cloud database service, to see database observability in action. To dive deeper and expand your knowledge, check out this learning byte for more insights on the MongoDB observability suite and how it can enhance your database performance.

November 14, 2024