Monitor and manage data quality (ANT343)

The key takeaways from the presentation can be summarized as follows:

Importance of Data Quality

  • Data quality is critical for virtually every data process and business decision in an organization.
  • Poor data quality can lead to significant issues, including compliance failures, unhappy clients, and wasted time fixing data problems.

Vanguard's Data Ecosystem

  • Vanguard has a federated, mesh-style data ecosystem with dozens of data products managed by different teams.
  • Each data product has its own definition of data quality that needs to be enforced.
  • Data quality checks are performed at multiple stages of the data workflow, including consistency, accuracy, completeness, and timeliness checks.

Vanguard's Custom Data Platform

  • Vanguard built a custom data platform called G4 to manage data quality across their ecosystem.
  • G4 is powered by AWS Glue Data Quality, leveraging its scalability, self-service capabilities, and integration with other AWS services.
  • The platform has three main layers:
    1. Interaction layer: Where data stewards define data quality rules using DQL.
    2. Orchestration layer: Manages the execution of data quality checks at scale.
    3. Execution layer: Where the actual data quality checks are performed using AWS Glue.

Key Considerations

  1. Platforming data quality: Integrating data quality checks into operational processes and workflows, not just the technical implementation.
  2. Handling dynamic data: Splitting data into homogeneous sets to simplify rule authoring and maintenance.
  3. Detection vs. Prevention: Vanguard chose to prevent low-quality data from entering their data hubs, rather than just detecting issues after the fact.

Overall, the presentation showcases how Vanguard leveraged AWS Glue Data Quality as part of a custom data quality platform to address the complex data quality challenges in their federated, mesh-style data ecosystem.

Your Digital Journey deserves a great story.

Build one with us.

Cookies Icon

These cookies are used to collect information about how you interact with this website and allow us to remember you. We use this information to improve and customize your browsing experience, as well as for analytics.

If you decline, your information won’t be tracked when you visit this website. A single cookie will be used in your browser to remember your preference.

Talk to us