Beyond 11 9s of durability: Data protection with Amazon S3 (STG339)

Here is a detailed summary of the video transcription in markdown format, broken down into sections for better readability:

Background on 11 9s of Durability in Amazon S3

  • Ian McGarry, Software Development Manager for Amazon S3, discusses the internal mechanisms and checks that Amazon S3 uses to achieve 11 9s of durability.
  • S3 operates at a massive scale, with millions of drives and over 1 PB of data transfer per second.
  • S3 has three core pillars to ensure durability: device failure, zone failure, and operational changes.
  • Device failure is handled through integrity checks, data redundancy, and periodic durability auditing.
  • Zone failure is handled by storing data across multiple availability zones, allowing S3 to tolerate the loss of an entire zone.
  • S3 uses a robust DNS-based routing system to route around zone failures quickly, ensuring availability.
  • S3 also has strong controls and guardrails in place, such as deployment restrictions and shadow mode testing, to mitigate the impact of operational changes.
  • S3 started as a globally distributed service but later shifted to a regionally isolated architecture to improve performance and durability.

Beyond 11 9s: Data Protection in Amazon S3

  • Data protection objectives: Protect against accidental deletes, overwrites, and lifecycle expirations; build resiliency against bad actors and regional issues; and meet governance and compliance requirements.
  • Data protection and security are closely related, with features like IAM, MFA, and encryption providing benefits for both.
  • The continuum of data protection includes:
    • Versioning: Protects against overwrites and accidental deletes.
    • MFA Delete: Adds an additional layer of protection for versioning.
    • Object Lock: Enforces retention periods and legal holds for data.
    • Cross-bucket, cross-account, and multi-region protection:
      • S3 Replication: Replicates data across regions or accounts.
      • Batch Operations: Enables bulk replication of existing data.
      • Backup Solutions: Isolated, managed backups of S3 data.
  • Auditing and monitoring features:
    • Trusted Advisor: Identifies buckets without versioning enabled.
    • AWS Config: Monitors changes to bucket configurations.
    • S3 Storage Lens: Provides metrics and visibility into data protection settings.

Protecting Against Accidental Deletes, Overwrites, and Lifecycle Expirations

  • Versioning protects against overwrites by creating a new version for each update, and delete markers instead of object deletions.
  • Lifecycle rules can manage the retention of current and non-current versions.
  • Conditional checks like if-match can prevent accidental overwrites.
  • Cloud Trail can be used to monitor and audit changes to bucket configurations.

Cross-Region and Cross-Account Protection

  • S3 Replication enables replication of data across regions or accounts, with options for delete marker replication, storage class selection, and cross-account replication.
  • Replication provides fault tolerance and allows for quick recovery in the event of a regional outage.
  • Batch Operations can be used to retroactively replicate existing data.
  • Replication Time Control provides SLA-backed replication with event notifications.
  • Backup Solutions, both AWS-managed and partner solutions, provide isolated and managed backups of S3 data.

Recap and Resources

  • Key takeaways:
    • Choose the right data protection features and make them the default.
    • Audit and monitor for unexpected changes.
    • Regularly review new features and updates.
  • Additional resources:
    • Skill Builder for training and learning opportunities.
    • Customer surveys to provide feedback and help shape future improvements.

Your Digital Journey deserves a great story.

Build one with us.

Cookies Icon

These cookies are used to collect information about how you interact with this website and allow us to remember you. We use this information to improve and customize your browsing experience, as well as for analytics.

If you decline, your information won’t be tracked when you visit this website. A single cookie will be used in your browser to remember your preference.

Talk to us