Monitor and manage data quality (ANT343)

The key takeaways from the presentation can be summarized as follows:

Importance of Data Quality

• Data quality is critical for virtually every data process and business decision in an organization.
• Poor data quality can lead to significant issues, including compliance failures, unhappy clients, and wasted time fixing data problems.

Vanguard's Data Ecosystem

• Vanguard has a federated, mesh-style data ecosystem with dozens of data products managed by different teams.
• Each data product has its own definition of data quality that needs to be enforced.
• Data quality checks are performed at multiple stages of the data workflow, including consistency, accuracy, completeness, and timeliness checks.

Vanguard's Custom Data Platform

• Vanguard built a custom data platform called G4 to manage data quality across their ecosystem.
• G4 is powered by AWS Glue Data Quality, leveraging its scalability, self-service capabilities, and integration with other AWS services.
• The platform has three main layers:
  1. Interaction layer: Where data stewards define data quality rules using DQL.
  2. Orchestration layer: Manages the execution of data quality checks at scale.
  3. Execution layer: Where the actual data quality checks are performed using AWS Glue.

Key Considerations

1. Platforming data quality: Integrating data quality checks into operational processes and workflows, not just the technical implementation.
2. Handling dynamic data: Splitting data into homogeneous sets to simplify rule authoring and maintenance.
3. Detection vs. Prevention: Vanguard chose to prevent low-quality data from entering their data hubs, rather than just detecting issues after the fact.

Overall, the presentation showcases how Vanguard leveraged AWS Glue Data Quality as part of a custom data quality platform to address the complex data quality challenges in their federated, mesh-style data ecosystem.