[NEW LAUNCH] Unlock the power of your data with Amazon S3 Metadata (STG366-NEW)
S3 Metadata: Accelerating Data Discovery
Data Discovery Challenges
Customers have seen their data sets, particularly unstructured data in S3, growing at a tremendous rate. Currently, there are over 400 trillion objects in S3.
This data is coming from many different sources and used for many different use cases, including:
AI/ML model training
Customizing pre-trained models for specific use cases
Customers are facing challenges in finding the right data sets to train their models at scale and in a timely fashion.
S3 Metadata Overview
S3 Metadata is a new feature that automatically generates rich and robust object metadata, including:
S3 system metadata (e.g., storage class, encryption type, size)
Object tags and user-defined metadata
The metadata table is stored in S3 Tables, a new flavor of S3 buckets optimized for analytical workloads.
The metadata is generated and updated in near real-time as changes occur in the source S3 bucket.
Key Features
Automatic Metadata Generation: S3 will automatically generate the metadata, including system metadata and custom metadata (object tags and user-defined metadata), without the need for customers to build and maintain their own metadata stores.
S3 Tables Integration: The metadata table is stored in S3 Tables, which provide optimized performance and reduced management overhead for analytical workloads.
Near Real-Time Updates: The metadata table is updated in near real-time as changes occur in the source S3 bucket, ensuring the data is up-to-date.
Analytics Integration: The metadata table integrates with AWS analytics services, such as Amazon Athena, as well as open-source tools, allowing customers to easily query and analyze the metadata.
Fine-Grained Access Control: Customers can use AWS Lake Formation to enforce fine-grained access control, including column and row-level access, on the metadata table.
Custom Metadata and Advanced Use Cases
In addition to the automatically generated metadata, customers can also use object tags and user-defined metadata to annotate objects with information specific to their business context.
These custom metadata can be used to:
Easily identify AI-generated data and track its lineage
Implement cost attribution by department or business unit
Build custom metadata pipelines to extract and store metadata based on specific requirements
Wrap-up and Resources
S3 Metadata provides a simple and automatic way to extract and manage metadata for objects stored in S3, helping customers accelerate data discovery and extract value from their data.
These cookies are used to collect information about how you interact with this website and allow us to remember you. We use this information to improve and customize your browsing experience, as well as for analytics.
If you decline, your information won’t be tracked when you visit this website. A single cookie will be used in your browser to remember your preference.