An insider’s look into architecture choices for Amazon DynamoDB (DAT419)

Here is a detailed summary of the video transcription in markdown format:

Scaling Dynamo DB: Architectural Choices for High Availability and Consistent Performance

Introduction

  • Joseph Idzior, Director of Product Management for non-relational databases at AWS, and Amrith Kumar, Senior Principal Engineer on DynamoDB, are presenting.
  • They will discuss the architectural choices made with DynamoDB to serve customers and provide explanations for the "why" and "how" behind these choices.

The DynamoDB Use Case

  • Customers using DynamoDB have high throughput applications that perform the same simple operations repeatedly (e.g., put an item in a shopping cart, get a taxi location, start a session).
  • DynamoDB was designed as an optimization, not just a slightly better relational database, focused on scalability, consistent performance, and high availability.

The Origins of DynamoDB

  • 20 years ago, Amazon faced a massive failure on their website during the holiday season, leading them to write a "Correction of Error" (COE) document.
  • The COE revealed that 90% of their database access patterns were simple key lookups, not needing the full capabilities of a relational database.
  • This led to the creation of the Dynamo paper and the initial internal Dynamo system at Amazon.
  • In 2012, Amazon launched the fully managed DynamoDB service, which has grown to over 1 million customers.

DynamoDB Architectural Choices

  1. Partitioning and Scaling: DynamoDB is designed as a distributed hash table, partitioning data across storage nodes. This allows for near-unlimited scale by splitting partitions as needed.
  2. Data Modeling: In a distributed system, data modeling is crucial to how data is accessed and placed. DynamoDB focuses on partition key lookups and range scans within partitions.
  3. Scalability at Every Layer: Every component of the DynamoDB service, from the transaction manager to the request routers, is engineered to scale horizontally.
  4. Multi-Tenancy and Resource Governance: DynamoDB is a multi-tenant service, with storage nodes hosting partitions for multiple customers. This allows efficient capacity management and performance isolation.
  5. Warm Throughput: DynamoDB recently launched the "warm throughput" feature, which exposes the cumulative throughput capacity of a table's partitions, enabling customers to understand and scale their workloads.

Engineering for High Availability

  • DynamoDB has had zero planned maintenance outages, as the team has focused on building a highly available system from the ground up.
  • Principles for building highly available systems:
    • Assume anything can fail, and engineer the system to handle it.
    • Build simple, modular components with well-defined failure domains.
    • Automate deployments and rollbacks to minimize the impact of failures.
    • Design for compatible, progressive deployments to avoid client-server version lock-step.

Engineering for Consistent Performance

  • Customers demand predictable latency, not just low latency.
    • DynamoDB focuses on eliminating performance variance by design choices, such as:
      • Routing requests to the same Availability Zone when possible
      • Implementing single-shot transactions
      • Using parallelism and hedging to reduce tail latency
  • Adjusting timeouts is a dangerous way to address performance issues; instead, engineer the system for consistent behavior.

DynamoDB Updates and Customer Highlights

  • Recent DynamoDB updates include:
    • 50% reduction in on-demand pricing
    • Warm Throughput feature
    • Reduced pricing and synchronous replication for Global Tables
    • Security and integration enhancements
  • Customer examples of building applications on DynamoDB, such as Duolingo, JP Morgan, and Roblox, are available on YouTube.

Resources

  • Additional DynamoDB architectural talks and white papers are recommended for further learning.

Your Digital Journey deserves a great story.

Build one with us.

Cookies Icon

These cookies are used to collect information about how you interact with this website and allow us to remember you. We use this information to improve and customize your browsing experience, as well as for analytics.

If you decline, your information won’t be tracked when you visit this website. A single cookie will be used in your browser to remember your preference.

Talk to us