Investing in data quality: Nasdaq’s journey to data reliability (BIZ207)
NASDAQ's Journey with Data Observability and Monte Carlo
Introduction to NASDAQ
NASDAQ is an exchange operator that manages over 30 different marketplaces globally, primarily focused in North America and the Nordics.
NASDAQ is also a major provider of financial technology to other market operators and financial institutions, with over 130 marketplaces and 2,200 financial institutions as customers.
NASDAQ's Data Challenges
NASDAQ uses its data for various purposes, including regulatory reporting, market surveillance, client reporting, and driving key business insights.
Challenges in data management and monetization include:
Ensuring data quality and trustworthiness, even if the data is accessible.
Bridging the gap between the technical capabilities of the data platform and the business needs for data access and monetization.
NASDAQ's Intelligence Platform
NASDAQ has developed the Intelligence Platform, a cloud-native data solution that has been in development for over a decade.
The Intelligence Platform has helped NASDAQ achieve a 75% reduction in time to market with new reports and insights, with roughly 2,200 users and 6,000-10,000 reports generated per day.
Data Observability and Monte Carlo
Data observability is the process of monitoring data end-to-end in a distributed environment, including understanding data quality, lineage, and the financial impact of data issues.
NASDAQ has partnered with Monte Carlo, a data observability platform, to address its data quality and monitoring challenges.
The key aspects of data observability that NASDAQ has implemented include:
Detecting data freshness, volume, schema changes, and field health issues.
Triaging data issues by understanding the impact, assigning ownership, and communicating the problem.
Resolving data issues quickly and measuring the effectiveness of the solutions.
Deploying Monte Carlo at NASDAQ
NASDAQ has a centralized data lake setup, with a shared ingestion account and separate query accounts for different applications.
NASDAQ has implemented a multi-layered approach to deploying Monte Carlo:
Shared monitors for common metrics like schema detection and volume.
Application-specific monitors for monitoring specific data pipelines and models.
Monitors for the critical billing endpoint to ensure accuracy and regulatory compliance.
Benefits of Data Observability with Monte Carlo
NASDAQ has been able to detect and resolve issues more quickly, such as:
Intraday load failures, saving 8 hours of development and operations time.
Duplicate data, preventing incorrect billing and calculations.
Missing data in the billing endpoint, which was detected and resolved in minutes.
Conclusion
NASDAQ encourages attendees to visit the Monte Carlo booth (667) and reach out with any further questions.
These cookies are used to collect information about how you interact with this website and allow us to remember you. We use this information to improve and customize your browsing experience, as well as for analytics.
If you decline, your information won’t be tracked when you visit this website. A single cookie will be used in your browser to remember your preference.