Elastic serverless: Stateless architecture for speed and scale (COP336)
Elastic Cloud Serverless: Reimagining Elasticsearch for the Cloud
Introduction
Elastic Cloud Serverless is now generally available on AWS, providing hassle-free operations at scale with an architecture optimized for real-time applications.
The presentation explores how Elastic Cloud Serverless was developed to address the challenges of ever-increasing data volume and complexity.
The Evolving Data Landscape
Data is messy, unstructured, and growing exponentially, with the magnitude of data being generated daily approaching a zetabyte.
Existing data architectures are struggling to keep up with the new demands of workloads like real-time analytics, vector search, and semantic search.
The value of unstructured data in organizations has increased tenfold in just a couple of years, requiring a new architecture to support these growing needs.
Limitations of the Traditional Elasticsearch Architecture
The traditional Elasticsearch architecture, with its stateful data nodes, leads to inevitable contention between indexing and search operations, making it challenging to optimize hardware for both workloads.
The need for primary and replica shards to ensure data availability increases storage costs and further exacerbates resource contention.
The introduction of data tiering and searchable snapshots helped, but the underlying challenges remained.
Towards a Stateless Elasticsearch Architecture
The team at Elastic recognized the need for a cloud-native architecture that could fully leverage the capabilities of cloud object storage.
The goal was to rebuild Elasticsearch from the ground up, separating compute from storage and indexing from search, to achieve true scalability and cost-efficiency.
This involved redesigning core Elasticsearch components, such as the translog and cluster metadata, to be stateless and optimized for cloud object storage.
The Elastic Cloud Serverless Architecture
Elastic Cloud Serverless leverages Kubernetes and Amazon EKS to provide a global control plane and regional data planes for provisioning, updating, and auto-scaling the infrastructure.
Indexing is performed by stateless pods that write directly to S3, eliminating the need for primary and replica shards.
Search operations are also scaled independently, with search pods querying data directly from S3.
This stateless, decoupled architecture allows Elastic Cloud Serverless to automatically respond to changes in usage, scaling compute and storage resources as needed.
Performance Demonstration
The presentation showcases the results of scale and performance tests conducted on Elastic Cloud Serverless.
The indexing test demonstrates the ability to ingest nearly 4 billion documents in a short timeframe, with the serverless architecture automatically scaling to handle the increasing load.
The search test shows Elastic Cloud Serverless handling over 3,000 queries per second, with a median response time of 36 milliseconds, across 5 terabytes of data.
Elastic Cloud Serverless Solutions
Elastic Cloud Serverless provides a range of out-of-the-box solutions for search, observability, and security, combining the benefits of a data lake with the speed and capabilities of Elasticsearch.
These solutions are designed to power a variety of use cases, from real-time applications to vector search and generative AI workloads.
Conclusion
Elastic Cloud Serverless is the result of a complete reimagining of Elasticsearch for the cloud, delivering simplified operations, cost-efficiency, and the ability to handle the demands of modern data workloads.
Customers can focus on building their applications and delivering value, while Elastic handles the underlying infrastructure and operations.
Elastic Cloud Serverless is available for trial, and users can sign up and get started in minutes, without the need for a credit card.
These cookies are used to collect information about how you interact with this website and allow us to remember you. We use this information to improve and customize your browsing experience, as well as for analytics.
If you decline, your information won’t be tracked when you visit this website. A single cookie will be used in your browser to remember your preference.