Elasticsearch vs MongoDB Atlas Search

Architecture

Elasticsearch: A Bolt-On approach

If a database’s internal search features are not adequate to satisfy the desired user experience, then another option is to bolt-on a dedicated search engine, such as Elasticsearch, alongside the database.

This provides the search features demanded by customers, but it does so while imposing additional constraints on developers and ops teams while driving up data duplication and technology sprawl.

A bolt-on specialized search engine alongside your database mandates synchronizing data between the two systems. While users get the rich search experience they expect, this comes at a significant cost. The application stack gets more complex and unwieldy. All of this translates to reduced developer velocity, compromised customer experience, and escalating costs.

How search works with a bolt-on solution:

Synchronization Overhead

To surface relevant and up-to-date search results, the database and search engine need to be kept synchronized, duplicating data between systems.

This means engineering teams need to create a synchronization mechanism that replicates data from the database to the search engine. Typically they will create a data pipeline with custom filtering and transformation logic built on top of messaging systems such as Apache Kafka, or using packaged connectors from specialized providers. Whether building or buying, the process takes time and adds ongoing costs. The synchronization mechanism also has to be deployed onto its own nodes, creating additional hardware sprawl.

Once the synchronization mechanism has been deployed, it needs to be monitored and managed, adding more engineering overhead.

It is important that replication to the search engine keeps pace with database writes so that search results do not excessively lag the database and break application SLAs. Monitoring the replication process is necessary to identify and remediate synchronization issues. This becomes especially complex if the search index falls so far behind the database that it has to be resynced from scratch, causing potential application downtime. It is not uncommon to find that 10% of engineering cycles are lost to manually recovering synchronization failures.

New application features that necessitate changes to the database’s schema often need both the synchronization logic and the search engine schema to also be updated at the same time. This creates more dependencies that slow down the pace of rolling new features to production.

How MongoDB Atlas Search is different

Atlas Search is built on top of MongoDB, the most popular and widely used modern database in the market. MongoDB has become so popular because engineering teams can build and ship applications faster than other data platforms. You can get started with both MongoDB Atlas and Atlas Search in minutes on a fully managed service that handles operations for you — on any cloud you choose.

By embedding an Apache Lucene search index directly alongside the database, data is automatically synchronized between the two, developers work with a single API, there is no separate system to run and pay for, and everything is fully managed for you, relieving operational burden. The MongoDB developer data platform radically simplifies your data architecture, enabling you to gain a competitive advantage by innovating faster while reducing cost, risk, and complexity.

With a distributed architecture, your database and search engine is resilient and globally scalable. Replication with self-healing recovery keeps your applications highly available while giving you the ability to isolate operational and search workloads on separate nodes within a single cluster. Native sharding provides elastic and application-transparent horizontal scale-out to accommodate your workload’s growth, along with geographic distribution for data residency controls. These controls ensure that data is kept close to users for low latency and to comply with data sovereignty mandated by modern privacy regulations.

Atlas Search is part of MongoDB Atlas, the multi-cloud developer data platform that combines transactional processing, relevance-based search, real-time analytics, mobile edge computing with cloud sync, and a federated query engine in an elegant and integrated data architecture. Through a flexible document data model and unified query interface, Atlas provides a first-class developer experience to power almost any class of application. At the same time, it meets the most demanding requirements for resilience, scale, and data privacy.

Why not just use a Search Engine as a Database?

With search engines storing and querying data, some engineering teams may consider eliminating the database altogether and just using the search engine for data persistence. At first glance, this would address many of the constraints discussed above, presenting a single system to develop against and to operationalize, while eliminating the overhead of data synchronization.

But as noted earlier, databases and search engines are different technologies designed to do different things.

Beyond serving application queries, databases are designed around a core set of data persistence and processing capabilities. These demand data integrity, consistency, and durability; balanced performance across reads and writes; concurrency; availability; security; disaster recovery; and more.

With a specialized architecture and indexing focused on fast, relevance-based information retrieval, dedicated search engines have a different set of design goals that compromise many of the capabilities that make databases so essential.

Elastic themselves caution against using a search engine as a database both in press articles and in core product documentation.

As discussed above, Elasticsearch is a capable search engine technology. However, its core system architecture is built around Lucene indexes in a way that forces compromises in many core database capabilities in order to meet its primary design goal as a scalable search engine.

DevOps

The Impact on Developer Productivity

It is critical in today’s digital economy for developers to build and evolve applications at speed. Introducing a separate search engine, like Elasticsearch, alongside the database means developers now have two separate systems they need to work with, which slows them down.

With this approach, developers have to learn how to work with two entirely different query languages to access the database and the search engine. This increases their learning curve and means frequent context switching when building application functionality, both of which impact their productivity while complicating testing and ongoing maintenance.

Because this approach requires two different APIs/drivers, application dependencies become much more complex, reducing the pace and frequency of releasing applications to production.

The DevOps Burden

Doubling up with a database and separate search engine such as Elasticsearch also adds time, cost, and complexity to operations and site reliability engineering (SRE) teams.

Now they have an additional system in their technology stack that needs constant care and feeding: It has to be provisioned, secured, monitored, scaled, patched, and backed up with its own tooling and APIs. It also means working across multiple vendors, making issue resolution more complex. Every new project means another dataset living in its own silo, adding to data sprawl and governance overhead.

How MongoDB Atlas Search is Different..

..for the Developer

The document data model is intuitive and flexible. Documents map directly to the objects in your code so they are much easier and more natural to work with. You can store, index, and search data of any structure and modify your schema at any time as you add new features to your applications.

You work with data as code. The MongoDB Query API and drivers are idiomatic to your programming language. Ad hoc queries, indexing, full-text search, and real-time aggregations provide powerful ways for accessing, grouping, transforming, searching, and analyzing your data to support any class of workload.

..for IT Operations

By embedding an Apache Lucene search index directly alongside the database, data is automatically synchronized between the two. This means engineers and administrators work with a single API, there is no separate system to run and pay for, and everything is fully managed for you, relieving operational burden. The MongoDB developer data platform radically simplifies your data architecture, enabling you to gain a competitive advantage by innovating faster while reducing cost, risk, and complexity.