Building Event-Driven Architectures using Amazon ECS with AWS Fargate
What is Event-Driven Architecture?
- An event is a JSON object that represents a change in system state
- Events cannot be changed, but can be modified by creating new events
- Events are a contract between the producer and the consumer
Event-Driven Architecture is:
- Something happens, and we react
- This allows for decoupling of applications
Integration Patterns in Event-Driven Architecture
Synchronous Request-Response
- Advantages: Low latency, simple, fail fast
- Disadvantages: Potential for throttling, added complexity, resiliency issues
Asynchronous Communication
- Introduces a broker that acknowledges the request but does not do the work
- Consumers pull the data from the broker when ready
Event Routers
- Route one event at a time
- Target multiple endpoints and protocols
- Filter at the router level
Event Bus
- Reduces location coupling
- Efficient for the sender
- Routing logic handled at the rule level
Event Topics
- Broadcast to multiple consumers
- Filter the consumer at the topic level
- Support multiple protocols (email, SMS, Lambda, etc.)
Event Stores
- Handle multiple events
- Ensure order is enforced (FIFO)
- Support dead-letter queues
SQS and Amazon MQ
- Fully managed message queues
- Scale almost infinitely
Event Store Streams
- All services get all events
- Filtering must be done at the consumer
- Order is enforced based on different parameters
Amazon Kinesis Data Streams and Amazon MSK (Kafka)
- Provide high-throughput streaming capabilities
AWS Fargate for Event-Driven Architectures
- Fargate is a serverless compute engine for containers
- Allows you to run containers without managing the underlying infrastructure
- Inherent qualities that make it well-suited for event-driven architectures:
- Scalable
- Supports microservices
- Integrates with event-based services like EventBridge and Step Functions
Patterns for AWS Fargate
Standalone Tasks
- Run tasks based on events (e.g., video processing, scheduled processing)
- Considerations:
- Use when Lambda is not a good fit
- Manage task failures separately
- Burst can cause scaling issues
Long-Running Services
- Run services that need to be up 24/7
- Scale dynamically based on metrics (e.g., queue depth, backlog per task)
Push-Based Patterns
- Use EventBridge to send events to public or private APIs
- API Destination feature allows sending events to APIs with rate limiting and retry handling
Pull-Based Patterns
- Process messages from queues (e.g., SQS)
- Scale task instances dynamically based on queue depth or custom metrics
Orchestration Patterns
- Use AWS Step Functions to orchestrate workflows
- Async pattern: Fire-and-forget
- Sync pattern: Wait for task completion
- Callback pattern: Get task results
- Step Functions Activity pattern: Decouple workflow and task processing
Resources
- Slides and recordings of related sessions: s12d.com/svs339
- Serverless learning resources: s12d.com/serverless.learning