Mongodb vs elasticsearch query performance

The Challenges of Application Search

Search is ubiquitous in application experiences. Whether we are shopping for groceries or buying a new home, browsing the web to find answers to our burning questions, servicing our customers, looking for our next job, or seeking suggestions for our next vacation, the search bar helps us navigate and discover the most relevant information — all in a way that seemingly interprets our natural language. People now expect these same intuitive search experiences in every application they use, whether at home or at work.

However, building these experiences is hard. In many cases, developers have to ‘bolt-on’ a search engine like Elasticsearch to their database and create a replication mechanism to keep the two systems synchronized. This approach introduces a huge amount of complexity to the application stack, reducing developer velocity while driving up risk, complexity, and cost.

Elasticsearch is a distributed search and analytics engine built on top of Apache Lucene and developed by Elastic. It extends Lucene’s indexing and search functionalities using RESTful APIs, and it achieves the distribution of data on multiple servers using the index and shards concept. Elasticsearch is based on JSON and is suitable for search use cases against time series data, structured or unstructured text, numerical data, or geospatial data.

Deployment options include self-hosting Elasticsearch, where the user is self-managing their instance, and Elastic’s cloud-hosted variant included in Elastic Cloud (which includes the rest of the ELK stack).

What MongoDB Atlas Search offers over Elasticsearch

A closer look at the respective approaches of Elasticsearch and MongoDB Atlas Search

Architecture

Elasticsearch: A Bolt On approach

If a database’s internal search features are not adequate to satisfy the desired user experience, then another option is to bolt on a dedicated search engine, such as Elasticsearch, alongside the database.

This provides the search features demanded by customers, but it does so while imposing additional constraints on developers and ops teams while driving up data duplication and technology sprawl.

A bolt-on specialized search engine alongside your database mandates synchronizing data between the two systems. While users get the rich search experience they expect, this comes at a significant cost. The application stack gets more complex and unwieldy. All of this translates to reduced developer velocity, compromised customer experience, and escalating costs.

How search works with a bolt-on solution:

To surface relevant and up-to-date search results, the database and search engine need to be kept synchronized, duplicating data between systems.

This means engineering teams need to create a synchronization mechanism that replicates data from the database to the search engine. Typically they will create a data pipeline with custom filtering and transformation logic built on top of messaging systems such as Apache Kafka, or using packaged connectors from specialized providers. Whether building or buying, the process takes time and adds ongoing costs. The synchronization mechanism also has to be deployed onto its own nodes, creating additional hardware sprawl.

Once the synchronization mechanism has been deployed, it needs to be monitored and managed, adding more engineering overhead.

It is important that replication to the search engine keeps pace with database writes so that search results do not excessively lag the database and break application SLAs. Monitoring the replication process is necessary to identify and remediate synchronization issues. This becomes especially complex if the search index falls so far behind the database that it has to be resynced from scratch, causing potential application downtime. It is not uncommon to find that 10% of engineering cycles are lost to manually recovering synchronization failures.

New application features that necessitate changes to the database’s schema often need both the synchronization logic and the search engine schema to also be updated at the same time. This creates more dependencies that slow down the pace of rolling new features to production.

How MongoDB Atlas Search is different

Atlas Search is built on top of MongoDB, the most popular and widely used modern database in the market. MongoDB has become so popular because engineering teams can build and ship applications faster than other data platforms. You can get started with both MongoDB Atlas and Atlas Search in minutes on a fully managed service that handles operations for you — on any cloud you choose.

By embedding an Apache Lucene search index directly alongside the database, data is automatically synchronized between the two, developers work with a single API, there is no separate system to run and pay for, and everything is fully managed for you, relieving operational burden. The MongoDB developer data platform radically simplifies your data architecture, enabling you to gain a competitive advantage by innovating faster while reducing cost, risk, and complexity.

With a distributed architecture, your database and search engine is resilient and globally scalable. Replication with self-healing recovery keeps your applications highly available while giving you the ability to isolate operational and search workloads on separate nodes within a single cluster. Native sharding provides elastic and application-transparent horizontal scale-out to accommodate your workload’s growth, along with geographic distribution for data residency controls. These controls ensure that data is kept close to users for low latency and to comply with data sovereignty mandated by modern privacy regulations.

Atlas Search is part of MongoDB Atlas, the multi-cloud developer data platform that combines transactional processing, relevance-based search, real-time analytics, mobile edge computing with cloud sync, and a cloud data lake in an elegant and integrated data architecture. Through a flexible document data model and unified query interface, Atlas provides a first-class developer experience to power almost any class of application. At the same time, it meets the most demanding requirements for resilience, scale, and data privacy.

Why not just use a Search Engine as a Database?

With search engines storing and querying data, some engineering teams may consider eliminating the database altogether and just using the search engine for data persistence. At first glance, this would address many of the constraints discussed above, presenting a single system to develop against and to operationalize, while eliminating the overhead of data synchronization.

But as noted earlier, databases and search engines are different technologies designed to do different things.

Beyond serving application queries, databases are designed around a core set of data persistence and processing capabilities. These demand data integrity, consistency, and durability; balanced performance across reads and writes; concurrency; availability; security; disaster recovery; and more.

With a specialized architecture and indexing focused on fast, relevance-based information retrieval, dedicated search engines have a different set of design goals that compromise many of the capabilities that make databases so essential.

Elastic themselves caution against using a search engine as a database both in press articles and in core product documentation.

As discussed above, Elasticsearch is a capable search engine technology. However, its core system architecture is built around Lucene indexes in a way that forces compromises in many core database capabilities in order to meet its primary design goal as a scalable search engine.

User Reviews

What are the pain points users are experiencing when going down the path of a Bolt On approach

DevOps

The Impact on Developer Productivity

It is critical in today’s digital economy for developers to build and evolve applications at speed. Introducing a separate search engine, like Elasticsearch, alongside the database means developers now have two separate systems they need to work with, which slows them down.

With this approach, developers have to learn how to work with two entirely different query languages to access the database and the search engine. This increases their learning curve and means frequent context switching when building application functionality, both of which impact their productivity while complicating testing and ongoing maintenance.

Because this approach requires two different APIs/drivers, application dependencies become much more complex, reducing the pace and frequency of releasing applications to production.

The DevOps Burden

Doubling up with a database and separate search engine such as Elasticsearch also adds time, cost, and complexity to operations and site reliability engineering (SRE) teams.

Now they have an additional system in their technology stack that needs constant care and feeding: It has to be provisioned, secured, monitored, scaled, patched, and backed up with its own tooling and APIs. It also means working across multiple vendors, making issue resolution more complex. Every new project means another dataset living in its own silo, adding to data sprawl and governance overhead.

How MongoDB Atlas Search is Different..

..for the Developer

The document data model is intuitive and flexible. Documents map directly to the objects in your code so they are much easier and more natural to work with. You can store, index, and search data of any structure and modify your schema at any time as you add new features to your applications.

You work with data as code. The MongoDB Query API and drivers are idiomatic to your programming language. Ad hoc queries, indexing, full-text search, and real-time aggregations provide powerful ways for accessing, grouping, transforming, searching, and analyzing your data to support any class of workload.

..for IT Operations

By embedding an Apache Lucene search index directly alongside the database, data is automatically synchronized between the two. This means engineers and administrators work with a single API, there is no separate system to run and pay for, and everything is fully managed for you, relieving operational burden. The MongoDB developer data platform radically simplifies your data architecture, enabling you to gain a competitive advantage by innovating faster while reducing cost, risk, and complexity.

Why choose MongoDB Atlas Search?

MongoDB Atlas Search is optimized for:

Is Atlas Search Always the Right Solution?

The above examples demonstrate how Atlas Search is designed for application search use cases. By design, it is tightly integrated with the MongoDB Atlas platform. Therefore all data has to first be stored in MongoDB database collections in order to then create the required search indexes against it.

Atlas Search is not currently designed for log analytics typically used in DevOps observability or security and threat hunting applications. Atlas Search is also not suitable for enterprise-wide search systems. In these scenarios, Elasticsearch provides built-in connectors and agents to crawl and extract data from multiple internal source systems, index them, and then make data and analytics searchable with bespoke tools.

For these use cases, it can be better to use MongoDB as one of your data sources alongside your existing Elasticsearch search engine.

Getting Started With Atlas Search

MongoDB Atlas offers a forever-free tier for development. Once deployed, simply click a button to add search to your application. With unlimited time to explore, see for yourself how a fully managed search engine integration helps your team build applications faster

Atlas Search is available with all Atlas clusters — including free clusters — so you can evaluate it at no cost.

Our Getting Started tutorial steps you through the process. Atlas Search documentation provides a complete reference on how to configure, manage, and query search indexes, along with performance recommendations. The MongoDB Developer Hub and MongoDB YouTube channel provide a wealth of articles and tutorials for beginners through to expert users.

Support Throughout Your Application Lifecycle

Frequently Asked Questions

MongoDB Atlas Search addresses a broad spectrum of search requirements without burdening your DevOps teams with undifferentiated work.

What is full-text search?

How is full-text search different from a database?

Is Elasticsearch like MongoDB?

How do I add data to Elasticsearch from MongoDB?

Is Atlas Search available for free tier?

What is the pricing for Atlas Search?

Where can customers find instructions or support for using Atlas Search?

Does Atlas Search support partial string matching?

Does Atlas Search support case-insensitive searches?

Is Atlas Search available for non-Atlas MongoDB deployments?

How is Atlas Search different from MongoDB’s legacy text search?

Related Resources

More than 31,000 customers have chosen to build the applications of today and tomorrow on MongoDB.

→

Learn more about MongoDB by exploring our latest webinars, video presentations, white papers, and more.

→

Build your MongoDB skills and advance your career with courses and certification. Learn to build better, faster applications for free.

→

Guides, samples, and references you need to use the database, visualize data, and build applications on the MongoDB data platform.

→

Join the MongoDB community at an event near you, or sign up for one of our upcoming webinars.

→

Trusted by:

Ready to get started?

Launch a new cluster or migrate to MongoDB Atlas with zero downtime.

Which is better MongoDB or Elasticsearch?

When should I use Elasticsearch and MongoDB?

Is MongoDB good for query?

How is Elasticsearch similar to MongoDB in terms of its data storage and usage?