AWS re:Invent 2025 - Simplify Data Lake Access with IAM Identity Center and trusted identity...

Simplifying Data Lake Access with IAM Identity Center and Trusted Identity Propagation

Overview

• The presentation discusses how to simplify data lake access using AWS IAM Identity Center and the concept of trusted identity propagation.
• The key challenges addressed are the complexity of managing multiple IAM roles and policies as access patterns grow, as well as the difficulty in tracking user actions when they assume different roles.

Data Lake Ingestion and Transformation

• Data is ingested into the data lake using AWS Glue, which connects to external data sources and automatically establishes the data catalog.
• Data transformation, cleansing, and augmentation can then be performed before the data is ready for consumption by end-users and applications.

Traditional IAM Role-Based Access Control

• IAM roles are commonly used to grant access to data in the data lake.
• However, as the number of access patterns grows, the management of IAM roles and policies becomes increasingly complex.
• This can lead to issues such as overly permissive roles or unused permissions.
• From the end-user perspective, the lack of a global identity and unified access control model makes it difficult to track user actions across different roles.

Trusted Identity Propagation with IAM Identity Center

• IAM Identity Center provides a solution to these challenges by enabling administrators to assign permissions based on the user's identity or the groups they belong to, rather than managing individual IAM roles.
• The process involves:
  1. Integrating IAM Identity Center with the organization's identity provider (e.g., Okta, Azure AD, Active Directory).
  2. Registering AWS applications (e.g., Amazon Redshift, Amazon Athena) with IAM Identity Center.
  3. Assigning user or group-based access permissions to these applications.
• When a user (e.g., Alice) authenticates with IAM Identity Center and accesses an application like Amazon Redshift, the application can query IAM Identity Center to determine the user's identity and group memberships, and then authorize access accordingly.

Benefits of Trusted Identity Propagation

• Eliminates the need to manage a large number of IAM roles and policies, as permissions are now based on user or group identities.
• Provides a unified access control model across the AWS ecosystem, where the user's identity is recognized by all integrated applications.
• Enables granular, user-specific permissions, rather than relying on broad role-based access.
• Improves visibility and auditability, as user actions can be tracked across all downstream services using the unique user identity.
• Allows the integration of custom applications with the same identity-based access control model, extending the benefits beyond just AWS services.

Technical Details and Examples

• AWS services like Amazon Redshift, Amazon Athena, and AWS Lake Formation can integrate with IAM Identity Center to leverage the user's identity and group information for access control.
• When a user (e.g., Alice) accesses an application like Amazon Quicksight, the application assumes a service role and enhances it with the user's identity context. This context is then propagated to downstream services like Amazon Athena, allowing them to authorize access based on the user's identity.
• The CloudTrail logs capture the "on behalf of" user identity, enabling comprehensive tracking of user actions across the entire AWS ecosystem.

Business Impact and Use Cases

• Simplifies the management of data lake access by shifting the focus from role-based to identity-based permissions.
• Improves security and compliance by providing a centralized, auditable view of user actions across the data lake and other AWS services.
• Enables more granular access control, allowing organizations to better enforce data governance policies and protect sensitive information.
• Facilitates the integration of custom applications with the same identity-based access control model, extending the benefits beyond just AWS services.
• Enhances the user experience by eliminating the need for end-users to assume different roles to access data, streamlining their workflow.

Conclusion

The presentation highlights how IAM Identity Center and trusted identity propagation can significantly simplify data lake access management, improve security and compliance, and provide a more seamless user experience. By shifting the focus from role-based to identity-based permissions, organizations can better manage the growing complexity of data lake access patterns and ensure that users have the appropriate level of access to the data they need.