AWS Storage Gateway: Bridging the On-Premises and Cloud Divide

Key Takeaways

1. Hybrid Cloud Challenges: Customers face challenges in working with hybrid cloud environments, having complex on-premises applications and databases, but wanting to leverage the cloud's scalability and resources.
2. AWS Storage Gateway: This service can bridge the gap between on-premises and cloud, enabling customers to access their files locally while also leveraging cloud management, edge computing capabilities, and cloud-based analytics.
3. Data Transfer Options: AWS provides various options for moving data to the cloud, including AWS DataSync for online data transfers, AWS Transfer Family for external file transfers, and AWS Storage Gateway for hybrid access.
4. Storage Gateway Use Cases: The S3 File Gateway is the most commonly used gateway type, enabling customers to build data lakes, store database backups, and archive cold data. It provides a fast and easy way to access the cloud, with up to 64TB of cache, security, compliance, and cost-effective pricing.
5. Customer Success Story: Apollo Tires, a multinational tire manufacturer, implemented the S3 File Gateway and migrated 160TB of data, saving up to 90% in backup costs and modernizing their manufacturing process.
6. Deep Dive: The Storage Gateway is deployed as a virtual machine, with a local cache disk for optimizing data transfers. It stores files as individual objects, enabling data workflows and integration with other AWS services.
7. Event-Driven Workflows: Storage Gateway provides file upload event notifications and working file set notifications, allowing customers to build event-driven architectures using AWS services like EventBridge, Lambda, Step Functions, and more.
8. Analytics and AI/ML Integration: Customers can leverage the data stored in S3 through Storage Gateway to unlock data-driven innovation, utilizing AWS analytics, machine learning, and generative AI services like SageMaker, Amazon Bedrock, and Amazon Q.

Demonstration: Building an Event-Driven Hybrid Architecture

The demonstration showcased how to build an event-driven hybrid architecture using AWS Storage Gateway, EventBridge, Lambda, and Step Functions. Key steps included:

1. Setting up an S3 File Gateway and mapping an SMB share to a Windows host.
2. Configuring file upload event notifications in EventBridge to trigger a Lambda function.
3. Leveraging the "Notify when uploaded" API to trigger a working file set event notification when all files are successfully uploaded to S3.
4. Using the working file set event to trigger a Step Functions state machine that sorts the files in the S3 bucket based on file metadata.
5. Sending a notification to an Amazon Chime group upon the completion of the file sorting process.

The demonstration highlighted how Storage Gateway can be integrated into an event-driven hybrid architecture to automate data workflows and leverage cloud-based analytics and AI/ML services.

Additional Resources

• Blogs and use cases on the AWS Storage Gateway product page
• Upcoming re:Invent sessions:
  • MGM STG349 - Multi-Cloud Data Transfer with AWS DataSync
  • STG318 - Hands-on Workshop: Automating Data Workflows Using AWS Storage Services
  • STG350 - Petabyte-Scale Migrations with AWS DataSync
  • STG204 - Hybrid Cloud Architecture with AWS DataSync
  • STG352 - Chalk Talk: Hybrid Cloud Data Management at Scale with AWS DataSync and Storage Gateway