Powering the grid: GE’s 600 TB migration to Amazon Keyspaces (DAT318)

Key Takeaways:

GE Vernova: Purpose-Built for Energy Transition

  • GE Vernova is a new purpose-built company focused on driving the energy transition and decarbonization.
  • As an industrial company with equipment, data, and a key role in power generation, transmission, and distribution, GE Vernova is uniquely positioned to lead the energy transition.
  • The company works with a "GE Vernova way" focused on innovation, customer-centricity, lean practices, teamwork, and accountability for decarbonization.

Database Migration from Cassandra to Amazon KeySpaces

Challenges

  • 600TB of critical industrial data across 300 SaaS enterprise customers
  • Need for 99.999% availability for thousands of users
  • 6-month timeline to complete migration

Migration Approach

  1. Experiment: Started with a small experiment to send live data to Amazon KeySpaces alongside the existing Cassandra system.
  2. Observe and Optimize: Analyzed the 5x increase in data size and adjusted hashing algorithms and data storage approaches to optimize performance.
  3. Historic Data Migration: Developed a scalable process using S3, Glue jobs, and a caching mechanism to migrate historic data from Cassandra to KeySpaces in parallel.
  4. Incremental Cutover: Gradually migrated customers from Cassandra to KeySpaces, maintaining full availability throughout the process.

Key Learnings

  • Importance of daily management and continuous monitoring to address issues quickly
  • Value of strong partnership and collaboration between the GE team, AWS, and ProServe
  • Need for early validation and testing to identify and resolve issues proactively
  • Significance of building trust and maintaining team morale during a complex transformation

Benefits of Amazon KeySpaces

  • Improved flexibility (e.g., configurable TTL)
  • Better observability and cost optimization
  • Ability to focus on customer needs rather than database maintenance
  • Availability of features like point-in-time recovery

Conclusion

  • The database migration was a critical step in GE Vernova's journey to lead the energy transition, enabled by the partnership and capabilities of Amazon KeySpaces.
  • The team's focus on lean practices, continuous improvement, and strong collaboration were key to the successful migration.
  • The learnings and best practices shared can help others undertake similar large-scale data transformations.

Your Digital Journey deserves a great story.

Build one with us.

Cookies Icon

These cookies are used to collect information about how you interact with this website and allow us to remember you. We use this information to improve and customize your browsing experience, as well as for analytics.

If you decline, your information won’t be tracked when you visit this website. A single cookie will be used in your browser to remember your preference.

Talk to us