Originally based on content from opensource.com.
What Are Transient Faults?
In electrical engineering, a transient fault is an error condition that disappears once power is disconnected and restored — the same principle behind the instinctive fix of turning a frozen device off and on again. In cloud computing, the equivalent faults are far harder to contain.
Distributed systems face increased complexity: known unknowns, unknown unknowns, infrastructure you will never physically touch, and technology that evolves faster than any single team can track. Today’s users have zero tolerance for unresponsive or unreliable services. With competitors a single click away, any measurable downtime risks permanent loss of users.
Two real-world incidents illustrate how quickly transient faults can become serious outages:
- A Rough Patch (Azure DevOps): Microsoft’s corporate VP of cloud developer services Brian Harry described weeks of sleepless nights tracing a storm of requests to the access control service (ACS) that exhausted Source Network Address Translation (SNAT) ports, blocked authentications, and impacted thousands of users.
- 503 Errors (Azure Functions): Cellenza’s Mikael Krief documented on the ALM DevOps Rangers blog how a refactored extension spawned a storm of connections and threads, overwhelming an Azure service and returning 503 Service Unavailable errors to end users. The incident reinforced the critical importance of monitoring from the very beginning of any implementation.
While failure and disaster recovery strategies can minimize the impact of infrastructure outages, solutions that rely on remote resources or services also need built-in sensitivity to transient faults. Well-designed systems detect these faults and attempt to self-correct before escalating an alert — or worse, becoming completely unresponsive.
Three patterns address the majority of transient fault scenarios in distributed systems: retry, throttling, and circuit breaker.
1. Retry Pattern
The retry pattern is the simplest of the three and mirrors what humans do instinctively — try again. It is effective for transient faults caused by brief network latency spikes, momentary service overloads, or short-lived connectivity interruptions across distributed networks.
How It Works
The core logic uses an incrementally increasing delay (exponential back-off) between successive attempts, and a maximum retry limit to prevent infinite loops:
Set failure_count = 0
CALL the [micro] service
If (fail) failure_count++
If (failure_count > retry_limit) OR (not a transient failure) → FAIL
DELAY (delay_time)
Increase delay_time by a factor of failure_count
RETRY from step 2Code language: JavaScript (javascript)The back-off delay is critical: retrying too aggressively can compound the original problem and push a struggling service into the throttling or circuit-breaker territory described below.
When to Use It
- Short-lived, self-resolving network errors
- Temporary service overloads expected to clear within seconds
- Idempotent operations where repeating the call is safe
Open-source implementations to explore include java-design-patterns and Microsoft’s Transient Fault Handling Application Block.
2. Throttling Pattern
Throttling protects your services from clients that are overutilising resources — whether due to a logic bug, a misconfigured consumer, or a sudden traffic surge. Think of it as a four-lane tunnel managing traffic from a six-lane freeway: the tunnel sets the maximum throughput, and excess flow must be managed at the entry point.
How It Works
Increment request_count
// limit = maximum requests permitted within a time interval
// degrade = return a "slow down" error or temporarily pause the operation
If (request_count > limit) → DEGRADE service
CALL the [micro] serviceCode language: JavaScript (javascript)When to Use It
- Enforcing service-level agreements (SLAs) on throughput
- Preventing a single consumer from monopolising shared resources
- Smoothing bursts of unexpected traffic
- Complementing the retry pattern — back-off delays in retries help avoid accidentally triggering throttle limits
Open-source implementations include WebApiThrottle and Core.Throttling.
3. Circuit Breaker Pattern
Like a circuit breaker in a home’s electrical panel, this pattern is your last line of defence. Where the retry pattern handles brief, self-resolving faults, the circuit breaker is designed for faults that persist over a longer period. Continuously retrying a failing service operation under these conditions can worsen the situation, cascade failures across dependent services, and ultimately bring down the entire system.
The core hypothesis: a failed service call is more likely to succeed if retried only after a significant delay — giving the downstream service time to recover before new load arrives.
How It Works
The circuit breaker maintains a state — open (normal operation) or closed (tripped). When failures exceed a threshold, the breaker trips and subsequent calls are blocked until a timer expires and a probe call succeeds.
// Circuit breaker has NOT tripped (open state = normal flow)
If (circuit_state == open)
CALL the [micro] service
If (fail) fail_count++
If (fail_count > limit) circuit_state = closed // trip the breaker
// Circuit breaker IS tripped (closed state = blocking calls)
Else
If (circuit_state == closed) Start recovery Timer
// Probe call when timer fires
On Timer timeout
CALL the [micro] service
If (success) circuit_state = open // reset the breakerCode language: PHP (php)States at a Glance
- Open: Calls flow normally; failures are counted.
- Closed (tripped): Calls are blocked; a timer runs to allow the downstream service to recover.
- Half-open (implied in most implementations): A single probe call tests whether the service has recovered before fully reopening.
Open-source implementations include Netflix’s Hystrix, circuit-breaker libraries, and Polly for .NET.
Combining the Patterns: A Microservices Example
In practice, these three patterns work together. Consider a microservices architecture where service #7 fails under load:
- Upstream callers first attempt retries with exponential back-off.
- If traffic to healthy services spikes as a result, throttling prevents resource exhaustion across the mesh.
- If service #7 remains unavailable beyond the retry limit, circuit breakers trip, halting cascading failures while the service recovers.
The complexity of managing these interactions grows rapidly — especially when feature flags are involved. A single flag flip can alter traffic patterns, expose previously hidden failure modes, and invalidate assumptions baked into your retry and throttle configurations.
Testing Your Fault-Handling Logic
Implementing these patterns is only half the work. You must validate them:
- Unit tests should verify that your fault-handling logic triggers correctly — confirm that retries fire the expected number of times, throttle limits return the right error codes, and circuit breakers transition state as designed.
- Integration tests should simulate resilient faults across your collective services to confirm that the system as a whole responds appropriately.
- Service virtualisation tools such as Hoverfly can simulate transient faults, slow responses, and degrading services in a controlled environment without requiring a real downstream dependency to misbehave on cue.
Stakeholders will not forgive fault-handling logic that itself fails under pressure. Treat resilience patterns as first-class features and test them accordingly.
Faults Are a Feature, Not a Failure
In a blameless DevOps culture, faults — like failures — are learning opportunities rather than indictments. The goal is not to eliminate faults entirely, which is impossible in distributed systems, but to build systems that detect, contain, and self-correct them before users are impacted.
Applying the retry, throttling, and circuit breaker patterns is a concrete, measurable step toward raising the quality bar of your infrastructure — continuously delivering business value and maintaining the trust of your stakeholders even when the unexpected happens.