Chaos engineering tests the behavior of a system when a controlled failure occurs. Instead of assuming that redundancy, monitoring, and recovery work because a design diagram says they should, an operator formulates a hypothesis and measures the response to a carefully bounded disruption. AWS Fault Injection Service (AWS FIS) provides experiment templates and preconfigured fault actions for supported AWS resources. It is a real operational tool: FIS runs real actions against real resources, so it must be treated as a potentially disruptive change rather than a harmless simulation.<\/p>\n
The objective is not to break as many components as possible. A well-designed exercise answers one reliability question, such as whether an application remains usable after a test instance becomes unavailable or whether the support team detects an important failure before users are affected. The experiment requires a known steady state, a narrow target, permission to run, stop conditions, observability, and a recovery procedure. Begin in a representative nonproduction environment and document what remains untested before considering any production experiment.<\/p>\n
Choose an outcome the business can recognize. A hypothesis might be that a web application continues serving successful requests when one disposable instance in a multi-instance target group becomes unavailable, while a CloudWatch alarm reports the event and the application recovers within an agreed period. Another might test whether a worker processes messages after a temporary downstream interruption without losing or duplicating important data. The hypothesis should identify what success would look like and which signals could disprove it.<\/p>\n
Define steady state with observable metrics. Useful measures may include successful request rate, latency, error rate, queue depth, replica health, or time to restore normal processing. Establish an ordinary baseline before introducing a fault. An alarm threshold that fires during normal operation is a poor safety guard, while a metric that is not being collected cannot show whether the experiment became unsafe. Start with a small set of reliable signals rather than an attractive dashboard of unrelated numbers.<\/p>\n
The broader AWS reliability and operations model<\/a> is useful for deciding what the experiment actually tests. High availability, fault tolerance, disaster recovery, and backup recovery are different promises. Removing one instance may test an availability mechanism without proving that the system can recover deleted data, survive regional disruption, or handle compromised credentials. Write the boundaries down so a successful test is not reported as proof of universal resilience.<\/p>\n An AWS FIS experiment template defines actions, targets, stop conditions, an experiment IAM role, and optional reporting or logging configuration. Actions describe the supported disruption, while targets identify the resources eligible for that action. A template can sequence actions or run permitted actions concurrently. The experiment role grants AWS FIS the required permissions; it is distinct from the engineer’s own authorization to create or start experiments. Review both sides of the access model.<\/p>\n Target selection deserves special care. FIS can select resource identifiers directly or use supported criteria such as tags and resource state. A broad tag or an unreviewed resource group can expand the blast radius beyond what the author intended. Before running anything, inspect or preview the resolved targets using the available console and API capabilities, and confirm their account, region, environment, and business owner. Prefer a tiny dedicated set of disposable resources during initial learning.<\/p>\n A template is not an execution, and creating one does not mean the fault has occurred. Starting an experiment runs its configured actions on the resolved targets. A finished experiment might complete successfully, stop because of an alarm, fail due to permissions or target selection, or be stopped manually. AWS does not let the operator resume a stopped or failed experiment; a subsequent attempt starts a new experiment from an appropriate template. That behavior should be reflected in the change procedure and post-incident record.<\/p>\n AWS FIS stop conditions are tied to Amazon CloudWatch alarms that indicate when an application has reached an unacceptable threshold. If a configured stop alarm triggers while the experiment is running, FIS stops the experiment. That protects against some out-of-bounds results, but stopping an action does not guarantee immediate restoration of every affected service or reverse every consequence. Application-specific recovery actions, data integrity, and ongoing health still need validation after the stop event.<\/p>\n Use an alarm connected to the outcome you care about, not simply any metric the service exposes. For a user-facing application, an elevated error rate or excessive latency may be more relevant than the CPU utilization of a single host. In a queue-processing service, delayed work or a growing backlog may be important. Review evaluation periods and missing-data behavior so the alarm can actually fire during the planned experiment window. The CloudWatch metrics and alarm model<\/a> helps explain why signal quality and alert routing matter to fault-injection safety.<\/p>\n Prepare independent safeguards as well. Limit the duration, obtain authorization from the service owner, schedule the test when an appropriate on-call team is available, and identify a person who can manually stop the experiment. Check that recovery access does not depend solely on the resource being disrupted. AWS FIS also has a safety-lever mechanism; operators should understand its current behavior and permission requirements where applicable. Do not assume that the presence of one alarm makes an experiment automatically safe to run against production.<\/p>\n A useful introductory scenario involves a small, nonproduction application with redundant disposable compute instances behind a load balancer. Before the experiment, confirm that both instances are healthy, the application responds, monitoring is working, and the test account does not contain production data. Set a hypothesis about response continuity and recovery time, and select an AWS FIS action appropriate to the resource type. Verify that the experiment role has only the permissions required for the selected operation and targets.<\/p>\n During the exercise, observe the resolved target, action state, service requests, health checks, and stop alarm. Record when the fault began, what users or test clients experienced, whether automation reacted, and when healthy operation returned. Compare the measured result against the hypothesis. A successful load balancer health check does not prove the business transaction succeeded; use an application-level check where practical. A failed experiment may reflect the injection setup rather than the application architecture, so distinguish permission errors from reliability failures.<\/p>\n For systems that use scaling, examine the relationship between health checks, desired capacity, replacement behavior, and traffic distribution. AWS Auto Scaling behavior<\/a> can help restore capacity under supported conditions, but the application may still have startup delays, data dependencies, or session-handling limitations. The point of the experiment is to expose those operational assumptions before a real incident, not to claim that adding Auto Scaling eliminates downtime.<\/p>\n Some AWS FIS scenarios use supported Systems Manager documents or other service-specific actions. Those integrations may require managed-node prerequisites, relevant role permissions, agent health, and the correct target operating environment. If a test command cannot execute, diagnose the management channel rather than widening the IAM role indiscriminately. The Systems Manager remote-command workflow<\/a> is useful background for understanding how a fleet action can be authorized, dispatched, observed, and audited.<\/p>\n Enable the appropriate AWS FIS experiment logging where available. AWS documents that experiment logs may be sent to CloudWatch Logs or Amazon S3 through template configuration and are disabled by default. Logs can show experiment progress and action information, while application logs, metrics, and AWS CloudTrail may answer different questions about workload behavior and API changes. Plan where the evidence will be retained and who may read it, especially when resource identifiers or operational details are sensitive.<\/p>\n Build a timeline from several evidence sources. Note template version, selected targets, experiment start, action states, relevant alarms, user-facing impact, manual interventions, and verified recovery. Avoid writing a report that says only \u201cFIS completed,\u201d because an action can complete while the application remains impaired. A well-supported conclusion identifies both what the experiment established and what was not measured, such as database consistency or performance under peak load.<\/p>\n Chaos experiments and disaster-recovery drills have related but distinct purposes. Disabling one component can test redundant capacity, while restoring a corrupted or deleted dataset requires a separate data-recovery exercise. If the workload stores important state, confirm whether the fault could affect open transactions, message delivery, or consistency. Use synthetic information for initial tests and avoid destructive operations until an appropriate recovery procedure has been independently validated.<\/p>\nUnderstand the AWS FIS experiment template<\/h2>\n
Build stop conditions and independent safety controls<\/h2>\n
Run a small preproduction exercise with observable results<\/h2>\n
Integrate Systems Manager, logging, and incident evidence<\/h2>\n
Connect controlled failures to recovery and data safety<\/h2>\n