Powering the grid: GE’s 600 TB migration to Amazon Keyspaces (DAT318)
Key Takeaways:
GE Vernova: Purpose-Built for Energy Transition
GE Vernova is a new purpose-built company focused on driving the energy transition and decarbonization.
As an industrial company with equipment, data, and a key role in power generation, transmission, and distribution, GE Vernova is uniquely positioned to lead the energy transition.
The company works with a "GE Vernova way" focused on innovation, customer-centricity, lean practices, teamwork, and accountability for decarbonization.
Database Migration from Cassandra to Amazon KeySpaces
Challenges
600TB of critical industrial data across 300 SaaS enterprise customers
Need for 99.999% availability for thousands of users
6-month timeline to complete migration
Migration Approach
Experiment: Started with a small experiment to send live data to Amazon KeySpaces alongside the existing Cassandra system.
Observe and Optimize: Analyzed the 5x increase in data size and adjusted hashing algorithms and data storage approaches to optimize performance.
Historic Data Migration: Developed a scalable process using S3, Glue jobs, and a caching mechanism to migrate historic data from Cassandra to KeySpaces in parallel.
Incremental Cutover: Gradually migrated customers from Cassandra to KeySpaces, maintaining full availability throughout the process.
Key Learnings
Importance of daily management and continuous monitoring to address issues quickly
Value of strong partnership and collaboration between the GE team, AWS, and ProServe
Need for early validation and testing to identify and resolve issues proactively
Significance of building trust and maintaining team morale during a complex transformation
Benefits of Amazon KeySpaces
Improved flexibility (e.g., configurable TTL)
Better observability and cost optimization
Ability to focus on customer needs rather than database maintenance
Availability of features like point-in-time recovery
Conclusion
The database migration was a critical step in GE Vernova's journey to lead the energy transition, enabled by the partnership and capabilities of Amazon KeySpaces.
The team's focus on lean practices, continuous improvement, and strong collaboration were key to the successful migration.
The learnings and best practices shared can help others undertake similar large-scale data transformations.
These cookies are used to collect information about how you interact with this website and allow us to remember you. We use this information to improve and customize your browsing experience, as well as for analytics.
If you decline, your information won’t be tracked when you visit this website. A single cookie will be used in your browser to remember your preference.