Building Event-Driven Architectures using Amazon ECS with AWS Fargate

What is Event-Driven Architecture?

• An event is a JSON object that represents a change in system state
• Events cannot be changed, but can be modified by creating new events
• Events are a contract between the producer and the consumer

Event-Driven Architecture is:

• Something happens, and we react
• This allows for decoupling of applications

Integration Patterns in Event-Driven Architecture

Synchronous Request-Response

• Advantages: Low latency, simple, fail fast
• Disadvantages: Potential for throttling, added complexity, resiliency issues

Asynchronous Communication

• Introduces a broker that acknowledges the request but does not do the work
• Consumers pull the data from the broker when ready

Event Routers

• Route one event at a time
• Target multiple endpoints and protocols
• Filter at the router level

Event Bus

• Reduces location coupling
• Efficient for the sender
• Routing logic handled at the rule level

Event Topics

• Broadcast to multiple consumers
• Filter the consumer at the topic level
• Support multiple protocols (email, SMS, Lambda, etc.)

Event Stores

• Handle multiple events
• Ensure order is enforced (FIFO)
• Support dead-letter queues

SQS and Amazon MQ

• Fully managed message queues
• Scale almost infinitely

Event Store Streams

• All services get all events
• Filtering must be done at the consumer
• Order is enforced based on different parameters

Amazon Kinesis Data Streams and Amazon MSK (Kafka)

• Provide high-throughput streaming capabilities

AWS Fargate for Event-Driven Architectures

• Fargate is a serverless compute engine for containers
• Allows you to run containers without managing the underlying infrastructure
• Inherent qualities that make it well-suited for event-driven architectures:
  • Scalable
  • Supports microservices
  • Integrates with event-based services like EventBridge and Step Functions

Patterns for AWS Fargate

Standalone Tasks

• Run tasks based on events (e.g., video processing, scheduled processing)
• Considerations:
  • Use when Lambda is not a good fit
  • Manage task failures separately
  • Burst can cause scaling issues

Long-Running Services

• Run services that need to be up 24/7
• Scale dynamically based on metrics (e.g., queue depth, backlog per task)

Push-Based Patterns

• Use EventBridge to send events to public or private APIs
• API Destination feature allows sending events to APIs with rate limiting and retry handling

Pull-Based Patterns

• Process messages from queues (e.g., SQS)
• Scale task instances dynamically based on queue depth or custom metrics

Orchestration Patterns

• Use AWS Step Functions to orchestrate workflows
  • Async pattern: Fire-and-forget
  • Sync pattern: Wait for task completion
  • Callback pattern: Get task results
• Step Functions Activity pattern: Decouple workflow and task processing

Resources

• Slides and recordings of related sessions: s12d.com/svs339
• Serverless learning resources: s12d.com/serverless.learning