What happens when your dependencies fail in production
Your WebAPI controller is perfect.
DTOs protect your boundaries. Validation catches bad requests before they hit the database. Status codes communicate clearly. Async operations scale under load. Everything from Part 1 is implemented.
You shipped it six months ago. It's been rock solid.
Then Black Friday hits. The payment API goes down. And your perfect controller — the one that passed every code review, the one with 94% test coverage — takes the entire site with it.
Think about the breaker panel in your house. A storm surge hits, or a shorted appliance starts pulling too much current — and the breaker trips. You get a dark kitchen instead of a house fire. That panel exists because somebody understood the alternative to failing fast is burning down. Your API needs the same hardware. It just doesn't come pre-installed.
Your controller was correct. The problem was you didn't design for failure.
💥 The Failure Cascade
Your "perfect" controller:
[ApiController]
[Route("api/v{version:apiVersion}/orders")]
public class OrdersController : ControllerBase
{
private readonly IOrderService _orderService;
public OrdersController(IOrderService orderService)
{
_orderService = orderService;
}
[HttpPost]
public async Task<IActionResult> CreateOrder(
CreateOrderRequest request,
CancellationToken cancellationToken)
{
var result = await _orderService.CreateOrderAsync(request, cancellationToken);
if (!result.IsSuccess)
return BadRequest(result.Error);
return CreatedAtAction(
nameof(GetOrder),
new { id = result.Value.Id },
result.Value);
}
// GetOrder action omitted for brevity
}
Your service layer:
public class OrderService : IOrderService
{
private readonly IPaymentClient _paymentClient;
private readonly IOrderRepository _orderRepository;
public OrderService(
IPaymentClient paymentClient,
IOrderRepository orderRepository)
{
_paymentClient = paymentClient;
_orderRepository = orderRepository;
}
public async Task<Result<OrderDto>> CreateOrderAsync(
CreateOrderRequest request,
CancellationToken cancellationToken)
{
// Charge customer through external payment API
var paymentResult = await _paymentClient.ChargeCustomerAsync(
request.CustomerId,
request.Total,
cancellationToken);
if (!paymentResult.IsSuccess)
return Result<OrderDto>.Failure(paymentResult.Error);
// Save order
var order = await _orderRepository.CreateAsync(request, cancellationToken);
return Result<OrderDto>.Success(order);
}
}
Note: Result<T> is a custom type for handling errors without exceptions — we build it from scratch in Exception Handling That Survives Production, including where the teaching version falls short. For now, just know .IsSuccess tells you if the operation worked, and .Value or .Error give you the outcome.
Looks good, right?
DTOs at the boundary. Proper status codes. Error handling with Result<T>. Async with CancellationToken.
What happens on Black Friday when the payment API goes down:
11:47 AM - Payment API starts responding slowly (5 seconds per request)
11:49 AM - Payment API stops responding entirely
11:50 AM - Hundreds of orders in flight, every one hanging on the payment call — waiting out the 100-second default HttpClient timeout
11:51 AM - First order finally fails; thousands more are stacked up behind it, each holding memory, a socket, and a slot in the outbound connection pool
11:52 AM - Requests pile up faster than they clear; Kestrel's request queue grows
11:53 AM - Health checks slow to a crawl, then start failing
11:54 AM - Site stops responding
11:55 AM - Load balancer removes your instances (health checks failing)
11:56 AM - Black Friday revenue: $0
11:57 AM - You're explaining to the CEO why the site is down and customers are shopping at competitors
Your controller was perfect. Your dependency wasn't.
🎯 The Shift
From: "My code is correct, dependencies should work"
To: "Dependencies will fail, my code should handle it"
I've debugged production APIs at scale — payment processors, claims pipelines, systems where every minute of downtime has a dollar figure attached. The pattern repeats:
- Payment gateway has an outage
- Database primary fails over to replica
- Third-party API rate-limits you
- Network partition between services
- DNS resolution fails temporarily
- Cloud provider has a regional issue
Your code doesn't control when dependencies fail. But you control what happens when they do.
🛡️ Pattern 1: Timeout (Fail Fast)
The trap: Waiting forever for a dead dependency.
What happens:
// HttpClient default timeout: 100 seconds
var response = await _httpClient.PostAsync(url, content, cancellationToken);
When the payment API hangs:
- Every in-flight order hangs for 100 seconds before timing out
- Each hanging request holds memory, a socket, and a slot in the outbound connection pool
- Under load: requests pile up faster than they clear
- Kestrel's request queue grows
- Health checks slow down, then fail
- Load balancer pulls you — site goes down
And if anything in the stack blocks on a result — one .Result, one .Wait() — the thread pool drains too. But you don't need that bug to go down; the pile-up alone will do it.
The fix: Fail fast. Don't wait 100 seconds for a dead API.
First, install the package. Microsoft.Extensions.Http.Resilience is a NuGet package, not part of the framework — it ships with the .NET 8 extensions wave but works on .NET 6 and later:
dotnet add package Microsoft.Extensions.Http.Resilience
Then wire it up:
// Program.cs
builder.Services.AddHttpClient<IPaymentClient, PaymentClient>()
.ConfigureHttpClient(client =>
{
client.BaseAddress = new Uri("https://payment-api.example.com");
})
.AddStandardResilienceHandler(options =>
{
// Total timeout for the entire request including retries
options.TotalRequestTimeout.Timeout = TimeSpan.FromSeconds(10);
// Timeout per individual attempt
options.AttemptTimeout.Timeout = TimeSpan.FromSeconds(3);
});
What this does:
- Each attempt times out in 3 seconds (not 100)
- Total request (including retries) times out in 10 seconds
- Each failed attempt is cut off at 3 seconds — the caller is never left hanging for 100
- Application stays responsive even when dependency is down
Healthcare example:
Patient at pharmacy counter. Prescription pricing API is down.
Without timeout: Patient waits 100 seconds while you stare at loading spinner. Awkward. They leave without medication.
With 3-second timeout: System fails in 3 seconds, shows cached pricing or manual override. Patient gets medication. Revenue protected.
🔄 Pattern 2: Retry (Handle Transient Failures)
Not every failure is permanent. Sometimes a packet drops. Sometimes a load balancer hiccups. Sometimes a container is mid-restart when your request arrives. These are transient failures — they fix themselves in seconds, but your user still gets an error page.
A single network blip shouldn't cost you an order.
The Standard Resilience Handler includes retry automatically:
builder.Services.AddHttpClient<IPaymentClient, PaymentClient>()
.AddStandardResilienceHandler(options =>
{
// Retry configuration
options.Retry.MaxRetryAttempts = 3;
options.Retry.BackoffType = DelayBackoffType.Exponential;
options.Retry.Delay = TimeSpan.FromSeconds(1);
options.Retry.UseJitter = true;
});
What this does:
- Request fails with 503 Service Unavailable
- Wait 1 second (with jitter), retry
- Fails again → wait ~2 seconds (exponential backoff + jitter), retry
- Fails third time → wait ~4 seconds (exponential backoff + jitter), retry
- Still failing → propagate error
Jitter prevents thundering herd: If 1,000 requests fail at the same moment, they don't all retry at the same moment (which would DDoS the recovering service). Jitter adds randomness to retry delays.
Note: No custom ShouldHandle predicate here, and that's deliberate. The default predicate already retries HttpRequestException (network failures where you never get a response — DNS timeouts, connection resets, TLS handshake failures), TimeoutRejectedException (the exception Polly's attempt timeout throws), and 408/429/5xx responses. If you override it, you replace all of that — and if you forget to re-include TimeoutRejectedException, your 3-second attempt timeouts are never retried and never trip the breaker. The defaults are good. Leave them alone.
⚠️ Idempotency matters more than the retry settings. Retries are only safe when the operation is idempotent — running it twice produces the same result as running it once. GET, PUT, and DELETE are idempotent by HTTP convention. POST is not. Retrying POST /payments can double-charge a customer. The fix is an idempotency key in the request — a client-generated unique ID per logical operation. The payment provider stores the result against that key; a retry with the same key returns the original result instead of charging again. Most modern payment APIs (Stripe, Square, etc.) require one. And here's the part that should make you sit up: the standard handler retries all HTTP methods by default — including POST. Out of the box, that retry policy will happily replay your payment request.
Use the idempotency key. It is the only defence that holds regardless of what your HTTP stack does, and it protects you against retries you didn't write — a load balancer, a client, an impatient user hitting the button twice. There is also options.Retry.DisableForUnsafeHttpMethods(), which turns off retries for POST, PATCH, PUT, DELETE, and CONNECT. Treat it as a second layer rather than the answer: it appears in the newer Microsoft.Extensions.Http.Resilience packages, and there's an open bug where POST requests are still retried despite it being configured. If you rely on it, write a test that proves it — don't take the call site on faith. Sort this out before you ship, not after the first double-charge.
Healthcare example:
Claims processing. Pharmacy benefit manager API hiccups.
Without retry: Claim fails. Patient's medication shows "not covered." Pharmacy calls insurance. 20-minute hold time. Patient leaves.
With retry: Transient failure retries automatically. Claim processes. Patient gets medication. Nobody notices the hiccup.
⚡ Pattern 3: Circuit Breaker (Stop the Bleeding)
Timeouts and retries handle the symptoms. Circuit breaker handles the disease.
This is the breaker panel from the top of the post — and it's not a metaphor anymore. The pattern is literally named after the thing in your basement, and it works the same way: too many failures flow through, the circuit opens, and the damage stops spreading.
Here's the scenario: payment API goes down completely. You've got timeouts — great, each request only wastes 3 seconds instead of 100. You've got retries — great, each request wastes 3 seconds four times. You're still burning time, sockets, and goodwill on a dependency you already know is dead.
- Request 1: Tries payment API, times out after 3 seconds
- Request 2: Tries payment API, times out after 3 seconds
- Request 3: Tries payment API, times out after 3 seconds
- ... (repeat for every order)
- 1,000 orders: 1,000 × 3 seconds = 3,000 seconds of cumulative customer wait
- Payment API gets hammered with requests it can't fulfill
- Your requests and resources are wasted on calls you know will fail
The fix: After N failures, stop trying.
The Standard Resilience Handler includes circuit breaker:
builder.Services.AddHttpClient<IPaymentClient, PaymentClient>()
.AddStandardResilienceHandler(options =>
{
// Circuit breaker configuration
options.CircuitBreaker.FailureRatio = 0.5; // Open if 50% of requests fail
options.CircuitBreaker.MinimumThroughput = 10; // Need at least 10 requests to evaluate
options.CircuitBreaker.SamplingDuration = TimeSpan.FromSeconds(30);
options.CircuitBreaker.BreakDuration = TimeSpan.FromSeconds(30);
});
Same rule as retry: no custom ShouldHandle. The default predicate already counts network failures, attempt timeouts, and 408/429/5xx responses as failures — exactly what you want feeding the breaker.
How circuit breaker works:
Closed State (Normal Operation)
- Requests go through normally
- Successes and failures are tracked
Open State (Dependency is Down)
- After 50% failure rate within 30-second window (with minimum 10 requests)
- Circuit opens
- All requests fail immediately without calling payment API
- Throws
BrokenCircuitExceptionimmediately - Stays open for 30 seconds
Half-Open State (Testing Recovery)
- After 30 seconds, circuit moves to half-open
- Allows a single probe request through
- If the probe succeeds: circuit closes (back to normal)
- If the probe fails: circuit re-opens for another 30 seconds
What this does:
Payment API goes down. After 5 failures out of 10 requests:
- Circuit opens
- Next 1,000 orders fail immediately (no 3-second timeout wasted)
- Payment API is not hammered with requests it can't handle
- After 30 seconds, try one request to see if API recovered
- If recovered: resume normal operation
- If still down: fail fast for another 30 seconds
Healthcare example:
Prescription pricing API goes down. You're processing 500 claims per minute.
Without circuit breaker:
- 500 claims/min × 3-second timeouts = 25 minutes of cumulative customer wait piling up every minute of wall-clock. The waits overlap — but every one of them is a patient standing at a counter
- Prescription API gets hammered with 500 requests/min it can't answer
- Your system is unusable
With circuit breaker:
- First 5 failures out of 10 → circuit opens
- Next 495 claims fail immediately (in milliseconds, not 3 seconds)
- System shows cached pricing or routes to manual review
- Prescription API gets a break (not hammered with requests)
- After 30 seconds, test if API recovered
- Claims keep processing (degraded mode, but operational)
🏗️ Combining All Three Patterns
In production, you use all three together:
// Program.cs
builder.Services.AddHttpClient<IPaymentClient, PaymentClient>()
.ConfigureHttpClient(client =>
{
client.BaseAddress = new Uri(
builder.Configuration["PaymentApi:BaseUrl"]
?? throw new InvalidOperationException("PaymentApi:BaseUrl not configured"));
})
.AddStandardResilienceHandler(options =>
{
// Timeout: Fail fast
options.TotalRequestTimeout.Timeout = TimeSpan.FromSeconds(10);
options.AttemptTimeout.Timeout = TimeSpan.FromSeconds(3);
// Retry: Handle transient failures
options.Retry.MaxRetryAttempts = 3;
options.Retry.BackoffType = DelayBackoffType.Exponential;
options.Retry.Delay = TimeSpan.FromSeconds(1);
options.Retry.UseJitter = true;
// Circuit Breaker: Stop hammering dead dependencies
options.CircuitBreaker.FailureRatio = 0.5;
options.CircuitBreaker.MinimumThroughput = 10;
options.CircuitBreaker.SamplingDuration = TimeSpan.FromSeconds(30);
options.CircuitBreaker.BreakDuration = TimeSpan.FromSeconds(30);
});
How they work together:
- Request fails (payment API slow)
- Timeout kicks in after 3 seconds (don't wait forever)
- Retry waits 1 second, tries again
- Timeout applies to retry too (3 seconds max)
- After 3 retry attempts, if still failing → propagate error
- Circuit breaker watches: if 50% of requests fail → open circuit
- Once open, all requests fail immediately (no timeout, no retry)
- After 30 seconds, circuit allows one test request
- If successful → back to normal
- If fails → stay open for 30 more seconds
Do the math on the worst case, though: 3-second attempts plus 1/2/4-second backoff delays blows right past a 10-second total. The total timeout will usually cut the retry sequence short — and that's its job. The total timeout is the hard promise you make to your caller; everything inside it is best-effort.
Order matters in the pipeline:
AddStandardResilienceHandler() actually configures five strategies, outermost to innermost:
- Rate Limiter (outermost - caps concurrent requests at 1,000 by default)
- Total Request Timeout (absolute limit across all attempts)
- Retry (retries the circuit-breaker-wrapped call)
- Circuit Breaker (trips when enough individual calls fail)
- Attempt Timeout (innermost - per individual call)
This is the correct order. Retry wraps the circuit breaker, which wraps the attempt timeout. When the circuit opens, the breaker throws BrokenCircuitException — which the retry predicate doesn't match, so the failure propagates on the very first attempt. No retries, no backoff delays. You fail in microseconds instead of seconds against a dependency you already know is down.
🔧 Custom Resilience Pipeline with Monitoring
Standard resilience handler is great for most cases. But sometimes you need custom behavior with logging to monitor what's happening in production:
using Polly.Timeout; // for TimeoutRejectedException
builder.Services.AddHttpClient<IPaymentClient, PaymentClient>()
.ConfigureHttpClient(client =>
{
client.BaseAddress = new Uri(
builder.Configuration["PaymentApi:BaseUrl"]
?? throw new InvalidOperationException("PaymentApi:BaseUrl not configured"));
})
.AddResilienceHandler("payment-pipeline", (pipelineBuilder, context) =>
{
var logger = context.ServiceProvider
.GetRequiredService<ILogger<PaymentClient>>();
// Outermost: Total timeout across all attempts
pipelineBuilder.AddTimeout(TimeSpan.FromSeconds(10));
// Retry with logging
pipelineBuilder.AddRetry(new HttpRetryStrategyOptions
{
MaxRetryAttempts = 3,
Delay = TimeSpan.FromSeconds(1),
BackoffType = DelayBackoffType.Exponential,
UseJitter = true,
ShouldHandle = new PredicateBuilder<HttpResponseMessage>()
.Handle<HttpRequestException>()
.Handle<TimeoutRejectedException>() // the attempt timeout below throws this — without it, timeouts are never retried
.HandleResult(response =>
response.StatusCode >= HttpStatusCode.InternalServerError ||
response.StatusCode == HttpStatusCode.RequestTimeout ||
response.StatusCode == HttpStatusCode.TooManyRequests),
OnRetry = args =>
{
// args.AttemptNumber is zero-based — the attempt that just failed
logger.LogWarning(
"Payment API retry after attempt {AttemptNumber} (Status: {Status}). Next retry in {Delay}ms.",
args.AttemptNumber,
args.Outcome.Result?.StatusCode,
args.RetryDelay.TotalMilliseconds);
return ValueTask.CompletedTask;
}
});
// Circuit breaker with logging
pipelineBuilder.AddCircuitBreaker(new HttpCircuitBreakerStrategyOptions
{
FailureRatio = 0.5,
MinimumThroughput = 10,
SamplingDuration = TimeSpan.FromSeconds(30),
BreakDuration = TimeSpan.FromSeconds(30),
ShouldHandle = new PredicateBuilder<HttpResponseMessage>()
.Handle<HttpRequestException>()
.Handle<TimeoutRejectedException>() // attempt timeouts must count as failures, or the breaker never trips
.HandleResult(response =>
response.StatusCode >= HttpStatusCode.InternalServerError ||
response.StatusCode == HttpStatusCode.RequestTimeout ||
response.StatusCode == HttpStatusCode.TooManyRequests),
OnOpened = args =>
{
logger.LogError(
"Payment API circuit breaker OPENED. Break duration: {Duration}s",
args.BreakDuration.TotalSeconds);
return ValueTask.CompletedTask;
},
OnClosed = args =>
{
logger.LogInformation(
"Payment API circuit breaker CLOSED. Normal operation resumed.");
return ValueTask.CompletedTask;
},
OnHalfOpened = args =>
{
logger.LogInformation(
"Payment API circuit breaker HALF-OPEN (testing recovery)");
return ValueTask.CompletedTask;
}
});
// Innermost: Timeout per attempt
pipelineBuilder.AddTimeout(TimeSpan.FromSeconds(3));
});
Critical detail: In Polly v8, the first strategy added is the outermost. The order above mirrors the standard handler's pipeline (minus the rate limiter) — the full breakdown, and why it matters when the circuit opens, is in the previous section.
When to use custom pipelines:
- Need logging on retry/circuit breaker events for production monitoring
- Different retry strategies for different error types
- More complex fallback logic
- Integration with monitoring/alerting systems
What to monitor in production:
- Retry attempt count (spike = dependency issues)
- Circuit breaker state changes (opened = dependency down)
- Timeout frequency (increase = dependency slow)
- Request success/failure ratio
- Total request duration (including retries)
Alert on:
- Circuit breaker opens (dependency is down)
- Retry rate >20% (dependency degraded)
- Timeout rate >10% (dependency slow)
⚠️ Common Gotchas
1. No Timeout = Request Pile-Up
The trap:
// No timeout configured
builder.Services.AddHttpClient<IPaymentClient, PaymentClient>();
What happens:
Default HttpClient timeout is 100 seconds. Dependency hangs. Every in-flight request hangs with it — each one holding memory, a socket, a connection-pool slot — for 100 full seconds. Requests pile up faster than they clear. Health checks fail. Site crashes.
The fix: Always configure timeout. 3-5 seconds for most external APIs.
2. Retry Without Backoff = DDoS Yourself
The trap:
// Immediate retry (no delay)
options.Retry.Delay = TimeSpan.Zero;
options.Retry.BackoffType = DelayBackoffType.Constant;
What happens:
Dependency has brief outage. 1,000 requests fail simultaneously. All 1,000 retry immediately. Dependency gets hammered with 3,000 requests in 1 second. You DDoS the service you're trying to use.
The fix: Exponential backoff with jitter. Standard resilience handler does this by default.
3. Circuit Breaker Too Sensitive
The trap:
// Hair-trigger configuration
options.CircuitBreaker.FailureRatio = 0.01;
options.CircuitBreaker.MinimumThroughput = 2; // Two requests, one failure — circuit opens
What happens:
Two requests, one transient hiccup. Circuit opens. All requests fail fast. Dependency recovers immediately but circuit stays open for 30 seconds. False positive outage. (Polly won't even let you go lower — MinimumThroughput is validated to be at least 2, precisely so a single failure can never trip a breaker.)
The fix: Reasonable thresholds. 50% failure ratio with a minimum of 10 requests is a good starting point — note that it's not what the handler ships with. The defaults are a 10% failure ratio over a minimum of 100 requests, which is deliberately hard to trip on a busy client and may never trip at all on a quiet one.
4. No Fallback Strategy
The trap:
var paymentResult = await _paymentClient.ChargeCustomerAsync(...);
if (!paymentResult.IsSuccess)
return Result<OrderDto>.Failure("Payment failed"); // Dead end
What happens:
Payment API is down. Circuit breaker is open. Every order fails with "Payment failed." Revenue: $0.
The fix: Fallback strategy.
var paymentResult = await _paymentClient.ChargeCustomerAsync(...);
if (!paymentResult.IsSuccess)
{
// Fallback: Save the order, queue payment for later
var pendingOrder = await _orderRepository.CreatePendingAsync(request);
await _paymentQueue.EnqueueAsync(pendingOrder.Id); // _paymentQueue: injected IPaymentQueue
return Result<OrderDto>.Success(new OrderDto
{
Id = pendingOrder.Id,
Status = "Pending Payment", // Background job will retry
Message = "Order received. Payment processing..."
});
}
Healthcare example:
Prescription pricing API down. Don't tell patient "system unavailable, go away."
Fallback options:
- Use cached pricing from last successful call
- Use average pricing for this medication
- Flag for manual pharmacist review
- Process claim, adjust pricing later if needed
Patient gets medication. Revenue protected. System operational.
🎓 Why These Patterns Matter
Every cascade failure I've helped untangle followed the same script:
Dependency fails → No resilience patterns → Cascade failure → Site down → Revenue lost
It doesn't matter if the dependency is a payment gateway, a claims processor, or a pricing API. The cascade is always the same. And every time, someone says "but the code was fine." The code was fine. The architecture wasn't.
Picture two e-commerce sites on Black Friday. Same payment gateway. Same 15-minute outage.
Site A has no resilience patterns. 100-second timeouts let requests pile up by the thousands inside two minutes. The site goes dark for the entire outage plus recovery time. Customer support lines light up. Social media does the rest.
Site B has the patterns from this post. 3-second timeouts keep requests short. Circuit breaker trips after the first wave of failures. Orders queue for later processing. The site stays up — degraded, not dead. When the gateway recovers, queued orders process automatically. Most customers never notice.
Same outage. One site lost its peak revenue hour. The other lost nothing.
Your controller is the last line of defense. When dependencies fail — and they will — your API can either cascade or degrade gracefully. These three patterns are the difference.
🧭 Key Takeaways
- Dependencies will fail in production — design for it
- Timeout: Fail fast (3-5 seconds), don't wait forever
- Retry: Handle transient failures with exponential backoff + jitter
- Circuit Breaker: Stop hammering dead dependencies
- Combine them: Use
AddStandardResilienceHandler()from theMicrosoft.Extensions.Http.ResilienceNuGet package (works on .NET 6+) - Monitor: Log when patterns activate, alert on circuit breaker opens
- Fallback: Don't just fail — queue, cache, or degrade gracefully
🚀 Next Steps
Review your HTTP clients:
- Do you have timeouts? Default 100 seconds will bury you in piled-up requests
- Do you retry transient failures? Network blips shouldn't fail orders
- Do you have circuit breakers? Stop hammering dependencies that are down
- Can you monitor resilience? Log retry attempts, circuit breaker state
- What's your fallback? When payment API is down, what happens to orders?
Start with your most critical dependency (payment, auth, inventory). Add resilience. Test it (disable the dependency, watch your circuit breaker work).
The APIs that survive Black Friday implement these patterns. The ones that crash don't.
Your controller is correct. Now install the breaker panel.
Related Posts:
- Building Professional WebAPI Controllers (Part 1) - Boundaries, DTOs, validation, status codes
- Exception Handling That Survives Production - Where the
Result<T>used throughout this post comes from - Dependency Injection in ASP.NET Core - Foundation for injecting HttpClients
- Configuration Management That Won't Get You Fired - Managing API endpoints, timeouts via configuration
In the next post: Unit testing with NSubstitute — why over-mocked tests explode the moment you refactor, and how to write tests that verify behavior instead of implementation.