AWS re:Invent 2025 - Observability & Security unite: Unify your data in Amazon CloudWatch (COP361)

Unifying Security and Observability Data in Amazon CloudWatch

Challenges with Data Fragmentation

• Organizations often have separate teams and tools for security, observability, and compliance/audit, leading to data silos
• The same underlying data (e.g. AWS service logs, application logs, third-party data) is often duplicated across these different teams and tools
• This results in:
  • Lack of comprehensive insights and delayed detection/response to issues
  • Operational overhead managing complex data pipelines and ETL processes
  • Increased costs from data duplication across multiple stores

Introducing CloudWatch Unified Store

To address these challenges, AWS introduced new capabilities in Amazon CloudWatch:

Data Collection

• Support for 65+ AWS services and 10 third-party data sources (e.g. Crowdstrike, Okta) out-of-the-box
• Organizational-level enablement for logs like CloudTrail and VPC Flow Logs

Data Curation

• Out-of-the-box transformers for common log formats (OCSF, OTel)
• Custom data pipelines using Grok processors for parsing and enrichment
• Automatic source and type metadata tagging

Centralized Data Storage

• Cross-account, cross-region log centralization with flexible retention policies
• Separate storage optimized for security vs observability use cases
• Open access to data via Amazon S3 tables integration

Advanced Analytics

• Facets for interactive exploration of log data without writing queries
• Amazon S3 table integration for connecting to any analytics engine (Athena, Redshift, etc.)

S&P Global's Journey

• S&P Global had requirements for:
  • Raw, immutable logs in a central archive
  • Curated logs for security tooling
  • Local log access for troubleshooting
• Partnered with AWS to simplify their log management architecture using CloudWatch capabilities
• Achieved a federated model with centralized storage, independent curation, and reduced operational overhead
• Expected 20-25% cost savings compared to their previous approach

Key Takeaways

1. Data fragmentation is a common challenge that can be addressed by a unified data store
2. Meeting users "where they are" is crucial - avoid disrupting existing workflows
3. Start small, learn, and scale iteratively - CloudWatch provides building blocks to enable this
4. Specific features like facets, S3 table integration, and centralized storage help reduce complexity and cost

Resources

• AWS News Blog: Unify Your Security and Observability Data in Amazon CloudWatch
• AWS Observability Resources