Indexless observability: How to decrease your observability costs (MAM206)

Here is a detailed summary of the video transcription in Markdown format, divided into sections for better readability:

Introduction and Assumptions

• The speaker is making assumptions about the attendees' roles, likely being Site Reliability Engineers (SREs), DevOps, Network Admins, CIS Admins, and Software Engineers.
• The talk is about observability, an architectural pattern, a data management process, and a philosophy on how to handle data more efficiently.
• The speaker is Chris, a Principal Engineer at Coralogix, a full-stack observability platform.

The Problem with Traditional Indexing

• Traditional databases use indexing to optimize query performance, where specific columns are indexed to enable faster lookups.
• In the world of observability, the approach has been to index everything, which has led to several issues:
  • The scale of data has increased due to factors like microservices, the cloud, and compliance requirements.
  • Indexing everything has become rigid, expensive, complex, and painful, leading to performance issues and cost problems.

The Alternative to Traditional Indexing

The speaker proposes the following principles for an index-less observability approach:

1. Treat Logs and Traces like a Database: Approach logs and traces as you would a database, considering how the data will be queried and accessed.
2. Everything is Queryable in Seconds: Ensure that all data, not just the indexed data, is accessible within seconds, as observability use cases require real-time access.
3. Can't Compensate with Rapid Storage: Relying on fast storage alone is not a sustainable solution, and the focus should be on managing the data effectively.
4. Ingestion is Only the Beginning: Transform the raw data into more efficient formats, such as metrics, to reduce storage and computational requirements.
5. Own Your Data: Maintain ownership and control over your data, storing it in cost-effective solutions like Amazon S3 for long-term retention and easy access.

Applying the Principles in Practice

The speaker discusses a well-known architecture and approach to implementing this index-less observability:

1. Open Telemetry: An open-source telemetry collector that provides an abstraction layer, allowing for easier vendor migration.
2. Apache Kafka: A high-throughput message broker used to ingest the data.
3. Amazon S3: A scalable and cost-effective storage solution for the raw data.
4. Amazon Athena: A serverless query service that allows for ad-hoc querying of the data in S3.
5. Prometheus/Thanos: Used for efficiently storing and querying metrics.

The speaker also discusses the drawbacks and challenges of this approach, such as the cost implications of Athena, the complexity of managing Kafka at scale, and the potential performance bottlenecks in the data transformation logic.

Coralogix: An Index-less Observability Solution

The speaker introduces Coralogix, a product built around the index-less observability principles. Coralogix provides a turnkey solution that abstracts away the complexity of the underlying architecture, allowing customers to benefit from the cost savings and performance improvements of this approach.

The speaker highlights Coralogix's support for various data types, pricing model, and upcoming feature announcements, emphasizing the company's experience and expertise in delivering this index-less observability solution.