AWS re:Invent 2025 - Secure Amazon ECS observability with CDK and Grafana (DEV338)

Secure Amazon ECS Observability with CDK and Grafana

Overview

• The speaker, Chiku, was tasked by his CTO to find a solution that would enable non-technical users to easily monitor and understand the activity within their healthcare startup's applications.
• The key requirements were to provide real-time insights, beautiful interactive dashboards, vendor independence, and open-source flexibility.
• Chiku decided to use Loki and Grafana to complement AWS CloudWatch and provide the desired observability capabilities.

Observability and the AWS Approach

• Observability is the ability to understand the internal state of a system by examining its external outputs, such as logs, metrics, and traces.
• The speaker explains the three core pillars of observability: logs, metrics, and traces.
• AWS provides the AWS Distro for OpenTelemetry (ADOT), a secure, production-ready distribution of OpenTelemetry supported by AWS. ADOT simplifies the deployment of instrumentation to collect traces, application metrics, and correlate them to AWS services like X-Ray and Amazon Managed Prometheus.

Architecture and Security

• The architecture consists of a public-facing application that interacts with users, and a private subnet where the observability components are deployed.
• Security is a key focus, with the following measures implemented:
  • AWS Client VPN for secure access to the private subnet
  • Separation of public and private subnets with security groups
  • IAM roles to control access to AWS services
  • Secure communication between the application and the observability components using the local host

AWS CDK and GitHub Integration

• The speaker used AWS CDK to define the infrastructure as code, which improved developer productivity and simplified resource management.
• CDK also enabled dependency sharing and avoided circular dependencies.
• GitHub was used as the source code repository, with the recommendation to use OIDC (OpenID Connect) for authentication instead of access keys.
• Automated CI/CD pipelines were set up to run security scans on code pushes.

Observability in Action

• The speaker demonstrated the instrumentation of the application using OpenTelemetry, capturing custom metrics and traces.
• The Grafana dashboard was used to visualize the metrics and traces, including the ability to drill down into specific errors and understand the root cause.
• A simulated 404 error was introduced, and the observability tools were used to quickly identify the issue and the underlying cause.

Key Takeaways

1. Start with security in mind from the beginning of the project, rather than treating it as an afterthought.
2. Leverage open-source tools like OpenTelemetry to achieve vendor-independent observability and benefit from a large, active community.
3. Automate infrastructure and deployment using tools like AWS CDK and GitHub to improve developer productivity and ensure consistent, secure deployments.
4. Combine logs, metrics, and traces to gain a comprehensive understanding of application behavior and quickly identify and resolve issues.

Resources

• GitHub repository for the project: [link]
• Two-part Medium article on the application: [link]