This article is based on reporting originally published on InfoQ, attributed to Hrishikesh Barua.
What Is AWSSSMChaosRunner?
AWS engineers open-sourced a chaos engineering tool called AWSSSMChaosRunner, which they used to test fault injection in Prime Video. The library is built on top of AWS Systems Manager (SSM) and enables teams to remotely execute fault-injection commands against targeted EC2 instances. By simulating real-world failure conditions, teams can validate whether their countermeasures — such as timeouts, retries, and circuit breakers — actually work under stress.
The tool was described by Varun Jewalikar, Software Engineer at Prime Video, and Adrian Hornsby, Principal Developer Advocate (Architecture) at AWS. Their core argument: typical chaos engineering scenarios like resource exhaustion or a slow network are rarely tested with confidence using unit or integration tests alone.
How It Works: AWS Systems Manager as the Foundation
AWS Systems Manager is an operational management service that can execute tasks across AWS resources. Its key component for AWSSSMChaosRunner is the SSM Agent, which:
- Comes pre-installed by default on many AWS-provided Windows and Linux AMIs.
- Supports SSM Documents — declarative runbooks that define sequences of commands to execute.
- Can run arbitrary shell scripts, which AWSSSMChaosRunner leverages for fault injection.
The SendCommand API in SSM allows commands to be dispatched to multiple instances simultaneously, with filtering by AWS resource tags. This makes it straightforward to target a specific tier of your infrastructure — for example, all instances tagged as role=video-streaming. Logs from all targeted instances can be aggregated in a single view using Amazon CloudWatch.
Faults are specified declaratively as a collection of commands. The SSM Agent handles security concerns such as creating a dedicated execution user on the EC2 instance, reducing the operational overhead for the team running the experiment.
Supported Fault Injection Types
AWSSSMChaosRunner supports failure injections at the infrastructure layer and the AWS service layer. Examples of what it can simulate include:
- Network latency: Introducing artificial delay on a specific network interface.
- Packet loss / TCP drop: Silently dropping all outgoing TCP traffic on a specific port.
- CPU hogging: Consuming CPU resources to simulate a resource-exhausted host.
It is important to note that AWSSSMChaosRunner works against EC2-based workloads and also supports Amazon ECS. However, it cannot be used with AWS Lambda because Lambda is a fully managed, serverless service where the underlying infrastructure is not accessible. For Lambda fault injection, other approaches and tools exist.
How Prime Video Used It
Prime Video, which relies heavily on AWS services internally, used AWSSSMChaosRunner to test how its systems performed when dependent services experienced high latency. The chaos experiments, combined with a load generation component that simulated real-world traffic, exposed issues that would be difficult to reproduce in a standard testing environment.
One concrete outcome: the team identified and fixed a bug in their Elasticache timeout configuration. Without chaos engineering, this misconfiguration might only have surfaced during an actual production incident.
Architecture Overview: From SSM Documents to Chaos Runner
AWSSSMChaosRunner originated as a set of raw SSM Documents written to inject faults into AWS resources. The library abstracts this further:
- Define fault scenarios declaratively as a collection of SSM commands or documents.
- Execute the document via standard SSM Agent APIs, targeting instances by tags.
- Run the load generation component in parallel to simulate realistic traffic against the application under test.
- Observe results in CloudWatch logs to determine whether the system behaved as expected under the injected fault.
How AWSSSMChaosRunner Compares to Other Chaos Engineering Tools
Chaos engineering as a discipline has matured significantly, with several tools and frameworks available:
- Netflix Chaos Monkey: One of the earliest chaos engineering tools, designed to randomly terminate EC2 instances to test resilience.
- LinkedIn Project Waterbear: LinkedIn’s internal chaos engineering framework.
- Twitter’s Python library: A Python-based framework developed by Twitter for their own fault injection needs.
- Gremlin: A commercial fault injection platform offered as a managed service.
- AWSSSMChaosRunner: AWS-native, open source, and tightly integrated with SSM, EC2, and ECS — making it a natural fit for teams already operating in the AWS ecosystem.
The primary advantage of AWSSSMChaosRunner for AWS-centric teams is its deep integration with existing AWS tooling — no additional agent installation is required on instances that already have the SSM Agent, and access control can be managed through standard IAM policies.
Why Chaos Engineering Matters for Production Reliability
As Jewalikar and Hornsby noted, most engineering teams have countermeasures for failure scenarios — timeouts, retries, fallback logic — but these are rarely validated with high confidence before a real incident. Unit and integration tests operate in controlled environments that cannot replicate the complexity of degraded network conditions or resource contention at scale.
Chaos engineering bridges this gap by deliberately introducing failure in a controlled way, allowing teams to observe system behavior, identify misconfigured settings (like the Elasticache timeout Prime Video discovered), and build confidence that their systems will degrade gracefully under real-world stress.
Video: AWS Chaos Engineering Playlist
Source: InfoQ. Original reporting by Hrishikesh Barua. Authors of the referenced AWS blog post: Varun Jewalikar and Adrian Hornsby.