Pipeline

Why I Wrote the New MongoDB Aggregations Book

In early May 2021, I published my book, Practical MongoDB Aggregations, which I released electronically and free for anyone to read . I love the MongoDB database and the uniqueness and power of its aggregation framework to analyse and manipulate massive amounts of data intuitively and efficiently. The opportunity to share this passion with others spurred me to write the book, with which I aim to support developers, architects, data analysts, data engineers, and data scientists to better understand how to maximise their productivity and effectiveness when building aggregation pipelines, as well as how to optimise these pipelines. Like many people over the past year during the pandemic, I’ve struggled to keep myself occupied when not busy doing my day job. Hence, my book was born not just from a desire to improve people’s knowledge but as my pandemic project, written over many weekends, to stave off the boredom. I believe aggregation pipelines provide a powerful domain-specific language for data processing in a way I’ve not seen before in other data-oriented tools, languages, or standards. SQL is a good data query language that caters to some analytical use cases via “group-by/having” statements. However, it typically has to be paired with a procedural language (e.g., Oracle’s PL/SQL ) to encompass an ordered set of complex data transformation rules. In the big data world of Hadoop , I find the MapReduce approach is too complex to develop with efficiently. Higher-level tools like Spark help alleviate some of this. However, by the necessity of still having to be general-purpose and versatile, the amount of Spark code required to process data sitting in any type of database is still too high for my liking. Many ETL tools provide proprietary data transformation capabilities, but these have to cater to the lowest common denominator capabilities across all the different types of databases they interact with. For these reasons and from experience, I consider MongoDB Aggregations to be the best tool for processing large data sets because it combines performance with productivity. Nevertheless, I sense the aggregation framework is shrouded in mystery for many people, hence my desire to demystify it with this book. I believe I identified a knowledge gap that many users wanted to be filled. MongoDB Inc. provides excellent reference documentation about aggregations in the MongoDB Manual , and MongoDB University provides a tremendous free online training course on aggregations . What I felt was still to be addressed was an opinionated yet informed perspective on how best to assemble aggregation pipelines from the well-documented parts—something that points the way to achieve optimal productivity and performance, accompanied by fully formed example pipelines to help put these approaches into practice. I hope readers of my book will learn some new things of value and enjoy reading it. A good test of the relevance of my book, in time, will be if people come back to it repeatedly as they continue with their journey of developing aggregations. Read the book for free now!