eBay: Building Mission-Critical Multi-Data Center Applications with MongoDB

Mat Keep
September 26, 2017 | Updated: November 9, 2017
#Customer Stories

As a top 10 global retail brand with 170+ million active buyers and 1 billion live listings across 190 markets around the world, eBay cannot afford systems downtime. This is why the company relies on MongoDB as one of its core enterprise data platform standards, powering multiple, customer-facing applications that run ebay.com.

At this year’s MongoDB World conference, Feng Qu, eBay’s lead NoSQL DBA, presented Practical Design Patterns for Resilient Applications – a set of architectural blueprints his team has developed to support enterprise-class MongoDB deployments.

Mr. Qu began his session discussing how the concept of availability has changed over the years. In the past, it was acceptable for sites to take scheduled downtime for weekly maintenance events. With the global nature of today’s services, neither users, or the business, are quite so accepting! In addition, most organizations now build out their services on commodity hardware platforms, rather than the exotic Sun Solaris / Sparc servers of yesteryear. While commodity hardware is much less costly, it also fails much more regularly. Both of these factors radically alter how engineering teams consider availability, and has led eBay to create its “Resiliency Design Patterns” to institute database best practices that maximize Mean Time To Failure (MTTF) and minimize Mean Time To Recovery (MTTR).

To build their apps, eBay developers can choose from five corporate-approved database standards. Alongside MongoDB, teams also have the option to use Oracle or MySQL relational databases, and two NoSQL options. Mr. Qu’s DBA team provide guidance on the appropriate database choice, qualifying the selection against the application’s data access patterns, user load, data types, and more.

eBay currently runs over 3,000 non-relational database instances powering a range of applications, managing multiple petabytes of data between them. In the past Oracle was the System of Record, while the non-relational databases handled transient data used in “systems of engagement”. However, the non-relational database landscape has matured. With consistent, point-in-time backup and recovery, MongoDB now also serves System of Record use cases at eBay.

While all of eBay’s non-relational database choices offer built in resilience to failure, they make different design tradeoffs that can impact application behavior. The DBA team assesses these differences across six dimensions: availability, consistency, durability, recoverability, scalability, and performance. For example, those NoSQL databases using peer-to-peer, masterless designs have expensive data repair and rebalancing processes that must be initiated following a node failure. This rebalancing process impacts both application throughput and latency, and can cause connection stacking as clients wait for recovery, which can lead to application downtime. To mitigate these affects, eBay has had to layer an application-level sharding solution, originally developed for its Oracle estate, on top of those masterless databases. This approach enables the DBA team to divide larger clusters into a series of sub-clusters, which isolates rebalancing overhead to a smaller set of nodes, impacting just a subset of queries. It is against these different types of database behaviors that the eBay DBA team builds its Resiliency Design Patterns.

Mr. Qu presented eBay’s standard “MongoDB Resilience Design Pattern”, as shown in Figure 1 below.

eBay design pattern of it's MongoDB Resilience Architecture

Figure 1: eBay design pattern for its MongoDB Resilience Architecture. (Image courtesy of eBay’s MongoDB World presentation).

In this design pattern, a 7-node MongoDB replica set is distributed across eBay’s three US data centers. This pattern ensures that in the event of the primary data center failing, the database cluster can maintain availability by establishing a quorum between remaining data centers. MongoDB’s replica set members can be assigned election priorities that control which secondary members are considered as candidates for promotion in the event of a primary failure. For example, the nodes local to DC1 are prioritized for election if the primary replica set member fails. Only if the entire DC1 suffers an outage are the replica set members in DC2 considered for election, with the new primary member selected on the basis of which node has committed the most recent write operations. This design pattern can be extended by using MongoDB’s majority write concern to enable writes that are durable across data centers.

The standard MongoDB design pattern is used as the basis for eBay’s “Read Intensive / Highly Available Read Pattern” discussed in the presentation, which is used to power the eBay product catalog. For the catalog workload, the MongoDB replica set is scaled out to 50 members, providing massive data distribution for both read scalability and resilience.

For more write-intensive workloads, eBay has developed its “Extreme High Read / Write Pattern”, which distributes a sharded MongoDB cluster across its US data centers.

Figure 2: eBay design pattern for the MongoDB Extreme High Read / Write Pattern. (Image courtesy of eBay’s MongoDB World presentation).

Again, eBay developers can configure this design pattern with specific MongoDB write and read concerns to tune the levels of durability and consistency that best meet the needs of different applications.

Mr. Qu noted that with recent product enhancements, MongoDB is being deployed to serve a greater range of application needs:

The addition of zone sharding to MongoDB 3.4 now enables eBay to serve applications that demand distributed, always-on write availability across multiple data centers.
Retryable writes, targeted for the forthcoming MongoDB 3.6 release, will allow eBay to reduce application-side exception handling code.

Review the recording of Feng Qu’s presentation at MongoDB World to learn more about eBay’s Design Patterns.

Download the MongoDB Multi-Data Center Deployments guide to get deeper insight into enabling active/active data center deployments and global data distribution with MongoDB.

← Previous

GDPR: Impact to Your Data Management Landscape: Part 4

Welcome to the final installment of our 4-part blog series. In part 1 , we provided a primer into the GDPR – covering its rationale, and key measures In part 2 , we explored what the GDPR means for your data platform In part 3 , we discussed how MongoDB’s products and services can support you in your path to compliance Finally, in this part 4, we’re going to examine how the GDPR can help in customer experience, and provide a couple of case studies. If you can’t wait for all 4 parts of the series, but would rather get started now, download the complete GDPR: Impact to Your Data Management Landscape whitepaper today. Using the GDPR for Customer Experience Transformation As discussed in parts 1 and 2 of this blog series, to comply with the GDPR, organizations will need to identify all personal data within their systems. Forward-looking companies can leverage the regulations for personal data discovery processes to transform interactions with their customers. Marketing and sales groups have long seen the value in aggregating data from multiple, disconnected systems into a single, holistic, real-time representation of their customer. This single view can help in enhancing customer insight and intelligence – with the ability to better understand and predict customer preferences, behaviors, and needs. However, for many organizations, delivering a single view of the customer to the business has been elusive. Technology has been one limitation – for example, the rigid, tabular data model imposed by traditional relational databases inhibits the schema flexibility necessary to accommodate the diverse customer data sets contained in multiple source systems. But limitations extend beyond just the technology to include the business processes needed to deliver and maintain a single view. MongoDB has been used in many single view projects across enterprises of all sizes and industries. Through the best practices observed and institutionalized over the years, MongoDB has developed a repeatable, 10-step methodology to successfully delivering a single view. Figure 1:10-step methodology to building a single customer view with MongoDB You can learn more by downloading the MongoDB single view whitepaper , covering: The 10-step methodology to delivering a single view The required technology capabilities and tools to accelerate project delivery Case studies from customers who have built transformational single view applications on MongoDB Case Studies MongoDB has been downloaded over 30 million times and counts 50% of the Fortune 100 as commercial customers of MongoDB’s products and services. Among the Fortune 500 and Global 500, MongoDB customers include: 40 of the top financial services institutions 15 of the top retailers 15 of the top telcos 15 of the top healthcare companies 10 of the top media and entertainment companies MongoDB is used by enterprises of all sizes, and from all industries to build modern applications, often as part of digital transformation initiatives in the cloud. An example of such a company is Estates Gazette, the UK’s leading commercial property data service. Estates Gazette (EG) The company’s business was built on print media, with the Estates Gazette journal serving as the authoritative source on commercial property across the UK for well over a century. Back in the 1990s, the company was quick to identify the disruptive potential of the Internet, embracing it as a new channel for information distribution. Pairing its rich catalog of property data and market intelligence with new data sources from mobile and location services – and the ability to run sophisticated analytics across all of it in the cloud – the company is now accelerating its move into enriched market insights, complemented with decision support systems. To power its digital transformation, Estates Gazette migrated from legacy relational databases to MongoDB, running an event-driven architecture and microservices, deployed to the Amazon Web Services (AWS) cloud. The company is also using MongoDB Enterprise Advanced with the Encrypted storage engine to extend its security profile, and prepare for the EU GDPR. You can learn more by reading the Estates Gazette case study . Leading European Retailer As part of its ongoing digital transformation that extends customer engagement beyond brick and mortar stores to mobile channels, the retailer with over 50,000 employees and €4.5bn in annual sales, was building a new mobile app offering opt-in marketing services to collect customer data, storing it in MongoDB. As part of its GDPR readiness, the retailer employed MongoDB Global Consulting Services to advise on data protection best practices, taking advantage of the MongoDB Enterprise Advanced access controls, encryption, and auditing framework. By using MongoDB consultants in the design phase of the project, the retailer has been able to adopt a “security by design and by default” approach, while enhancing its security posture. Wrapping Up our 4-Part Blog Series That wraps up the final part of our 4-part blog series. If you want to read the entire series in one place, download the complete GDPR: Impact to Your Data Management Landscape whitepaper today. It’s worth remembering that it takes much more than security controls of a database to achieve GDPR compliance. However, MongoDB is offering a holistic vision of how database customers can accelerate a path to meeting regulations scheduled for enforcement from May 2018. Using the advanced security features available in MongoDB Enterprise Advanced and the MongoDB Atlas managed database service, organizations have extensive capabilities to implement the data discovery, defense, and detection requirements demanded by the GDPR. Methodologies used in successfully delivering customer single view projects can be used to support data discovery, and used to innovate in delivering a differentiated customer experience. Disclaimer For a full description of the GDPR’s regulations, roles, and responsibilities, it is recommended that readers refer to the text of the GDPR (Regulation (EU) 2016/679), available from the Official Journal of the European Union , and refer to legal counsel for the interpretation of how the regulations apply to their organization. Further, in order to effectively achieve the functionality described in this blog series, it is critical to ensure that the database is implemented according to the specifications and instructions detailed in the MongoDB security documentation . Readers should consider engaging MongoDB Global Consulting Services to assist with implementation.

September 18, 2017

Next →

AI-Powered Call Centers: A New Era of Customer Service

Customer satisfaction is critical for insurance companies. Studies have shown that companies with superior customer experiences consistently outperform their peers. In fact, McKinsey found that life and property/casualty insurers with superior customer experiences saw a significant 20% and 65% increase in Total Shareholder Return , respectively, over five years. A satisfied customer is a loyal customer. They are 80% more likely to renew their policies, directly contributing to sustainable growth. However, one major challenge faced by many insurance companies is the inefficiency of their call centers. Agents often struggle to quickly locate and deliver accurate information to customers, leading to frustration and dissatisfaction. This article explores how Dataworkz and MongoDB can transform call center operations. By converting call recordings into searchable vectors (numerical representations of data points in a multi-dimensional space), businesses can quickly access relevant information and improve customer service. We'll dig into how the integration of Amazon Transcribe, Cohere, and MongoDB Atlas Vector Search—as well as Dataworkz's RAG-as-a-service platform— is achieving this transformation. From call recordings to vectors: A data-driven approach Customer service interactions are goldmines of valuable insights. By analyzing call recordings, we can identify successful resolution strategies and uncover frequently asked questions. In turn, by making this information—which is often buried in audio files— accessible to agents, they can give customers faster and more accurate assistance. However, the vast volume and unstructured nature of these audio files make it challenging to extract actionable information efficiently. To address this challenge, we propose a pipeline that leverages AI and analytics to transform raw audio recordings into vectors as shown in Figure 1: Storage of raw audio files: Past call recordings are stored in their original audio format Processing of the audio files with AI and analytics services (such as Amazon Transcribe Call Analytics ): speech-to-text conversion, summarization of content, and vectorization Storage of vectors and metadata: The generated vectors and associated metadata (e.g., call timestamps, agent information) are stored in an operational data store Figure 1: Customer service call insight extraction and vectorization flow Once the data is stored in vector format within the operational data store, it becomes accessible for real-time applications. This data can be consumed directly through vector search or integrated into a retrieval-augmented generation (RAG) architecture, a technique that combines the capabilities of large language models (LLMs) with external knowledge sources to generate more accurate and informative outputs. Introducing Dataworkz: Simplifying RAG implementation Building RAG pipelines can be cumbersome and time-consuming for developers who must learn yet another stack of technologies. Especially in this initial phase, where companies want to experiment and move fast, it is essential to leverage tools that allow us to abstract complexity and don’t require deep knowledge of each component in order to experiment with and realize the benefits of RAG quickly. Dataworkz offers a powerful and composable RAG-as-a-service platform that streamlines the process of building RAG applications for enterprises. To operationalize RAG effectively, organizations need to master five key capabilities: ETL for LLMs: Dataworkz connects with diverse data sources and formats, transforming the data to make it ready for consumption by generative AI applications. Indexing: The platform breaks down data into smaller chunks and creates embeddings that capture semantics, storing them in a vector database. Retrieval: Dataworkz ensures the retrieval of accurate information in response to user queries, a critical part of the RAG process. Synthesis: The retrieved information is then used to build the context for a foundational model, generating responses grounded in reality. Monitoring: With many moving parts in the RAG system, Dataworkz provides robust monitoring capabilities essential for production use cases. Dataworkz's intuitive point-and-click interface (as seen in Video 1) simplifies RAG implementation, allowing enterprises to quickly operationalize AI applications. The platform offers flexibility and choice in data connectors, embedding models, vector stores, and language models. Additionally, tools like A/B testing ensure the quality and reliability of generated responses. This combination of ease of use, optionality, and quality assurance is a key tenet of Dataworkz's "RAG as a Service" offering. Diving deeper: System architecture and functionalities Now that we’ve looked at the components of the pre-processing pipeline, let’s explore the proposed real-time system architecture in detail. It comprises the following modules and functions (see Figure 2): Amazon Transcribe , which receives the audio coming from the customer’s phone and converts it into text. Cohere ’s embedding model, served through Amazon Bedrock , vectorizes the text coming from Transcribe. MongoDB Atlas Vector Search receives the query vector and returns a document that contains the most semantically similar FAQ in the database. Figure 2: System architecture and modules Here are a couple of FAQs we used for the demo: Q: “Can you explain the different types of coverage available for my home insurance?” A: “Home insurance typically includes coverage for the structure of your home, your personal belongings, liability protection, and additional living expenses in case you need to temporarily relocate. I can provide more detailed information on each type if you'd like.” Q: “What is the process for adding a new driver to my auto insurance policy?" A: “To add a new driver to your auto insurance policy, I'll need some details about the driver, such as their name, date of birth, and driver's license number. We can add them to your policy over the phone, or you can do it through our online portal.” Note that the question is reported just for reference, and it’s not used for retrieval. The actual question is provided by the user through the voice interface and then matched in real-time with the answers in the database using Vector Search. This information is finally presented to the customer service operator in text form (see Fig. 3). The proposed architecture is simple but very powerful, easy to implement, and effective. Moreover, it can serve as a foundation for more advanced use cases that require complex interactions, such as agentic workflows , and iterative and multi-step processes that combine LLMs and hybrid search to complete sophisticated tasks. Figure 3: App interface, displaying what has been asked by the customer (left) and how the information is presented to the customer service operator (right) This solution not only impacts human operator workflows but can also underpin chatbots and voicebots, enabling them to provide more relevant and contextual customer responses. Building a better future for customer service By seamlessly integrating analytical and operational data streams, insurance companies can significantly enhance both operational efficiency and customer satisfaction. Our system empowers businesses to optimize staffing, accelerate inquiry resolution, and deliver superior customer service through data-driven, real-time insights. To embark on your own customer service transformation, explore our GitHub repository and take advantage of the Dataworkz free tier .

November 27, 2024