High Availability in Microservices: Designing for Failure from First Outage to Five Nines

The Moment It Breaks

Imagine you are a user. You are standing at a coffee shop counter. You open an app, tap "Pay Now," and nothing happens. No confirmation. No error message. Just the loading spinner, spinning.

So you tap again.

Did the system fail? Or is it just slow?

That question matters more than it seems. If the payment goes through after thirty seconds, the system might technically be working. But to you, standing at the counter, it is broken. The coffee shop staff are waiting. The person behind you is waiting.

At what point does "slow" become "unavailable"?

That question is the heart of high availability in microservices. Availability is more than a simple yes-or-no question: is the server up or down? In distributed systems, the reality is far messier. A system that responds after thirty seconds is, for all practical purposes, unavailable. A system that returns errors half the time is unavailable. A system that works perfectly for everyone except users in one region is partially unavailable.

Availability, therefore, is not binary. It is probabilistic, and it is defined by user-perceived success.

Most systems are designed as if failures are rare events. In production, failure is the default state.

So the right starting point is not success, but failure: understanding why systems become unavailable, and how to design them to continue operating despite it.

What Availability Means in Distributed Systems

Before we go further, let us pin down a definition.

In distributed systems, availability is usually expressed as the proportion of time a system is able to successfully serve requests within an acceptable time limit.

Without a time limit, a system that takes ten minutes to load a page would count as available. That is not useful. So when engineers talk about availability, they always include an implicit or explicit time bound. A request that takes longer than, say, one second might as well be an error.

You will often see availability expressed as a percentage: 99.9%, 99.99%, or 99.999%—sometimes called "five nines." These numbers sound similar, but the difference in real-world downtime is enormous.

A system with 99.9% availability can be down for about eight hours and forty-five minutes per year. That sounds bad, but for many internal tools, it might be acceptable. A system with 99.99% availability can be down for about fifty-two minutes per year. A system with 99.999% availability can be down for about five minutes per year.

This is where two terms become essential.

A Service Level Indicator (SLI) is a measurement of some aspect of your system's behaviour. For availability, the most common SLI is success rate: what proportion of requests received a valid response within your time bound?

A Service Level Objective (SLO) is the target you set for that SLI. For example: "99.9% of payment requests must return a successful response within one second."

The SLO is a formal commitment—to your team, your product, and your users. It turns "the system should be reliable" into something you can measure, track, and defend.

The key insight here is that availability is not a technical target you achieve once. It is a business decision you make repeatedly. And every extra nine costs you something, which we will return to later.

First, we need to understand why distributed systems become unavailable in the first place.

Why Distributed Systems Become Unavailable

In smaller or simpler systems, failures may appear infrequent. In distributed systems, failures are expected and continuous, forming part of the baseline conditions the system must operate within.

Several characteristics of distributed systems shape how failures emerge in practice.

Networks are inherently unreliable. Packets can be dropped, connections can time out, and components within a data centre can fail without warning.

Latency is variable and often unpredictable. A request that takes one millisecond in a development environment may take hundreds of milliseconds in production, and significantly longer under load or contention.

Dependencies introduce additional uncertainty. Databases may reject connections, external services may throttle requests, and upstream systems may become slow or unresponsive.

These conditions are not exceptional; they form part of the normal operating environment of distributed systems.

To reason about availability, it helps to group failures into layers. Each layer behaves differently under stress and requires different design considerations.

The edge layer includes DNS, CDNs, and API gateways. Failures here can prevent requests from reaching your system entirely.

The service layer is where application logic runs. Failures may arise from bugs, resource contention, or overload conditions.

The dependency layer includes databases, caches, message queues, and third-party APIs. These systems introduce external points of failure that are often outside your direct control.

The infrastructure layer includes servers, networking components, power systems, and availability zones. Failures at this level can affect multiple services simultaneously.

Each layer exhibits distinct failure patterns, and designing for availability requires addressing them explicitly.

The Multiplication Problem

There is a subtle assumption that often shows up in system design, even among experienced engineers.

If each individual service is designed to be highly available — say 99.9% — it is natural to expect the overall system to inherit the same level of availability. But in distributed systems, availability does not combine in a linear way. It compounds.

Imagine a simple request flow. A user request passes through Service A, then Service B, then Service C. Each of these services operates at 99.9% availability. The question becomes: what does that mean for the end-to-end system?

The result is not 99.9%. It is the product of each stage: 99.9% × 99.9% × 99.9%, which comes out to approximately 99.7%.

The difference looks small at first glance, but it reflects an important structural reality. As the number of sequential dependencies increases, the overall availability steadily declines. With ten services in a chain, each at 99.9%, the system-level availability drops to around 99.0%. In practical terms, that translates to close to nine hours of downtime per year, even though every individual service still meets a "three nines" target.

This pattern is known as a series system. In a series configuration, components are arranged one after another, and the failure of any single component affects the entire flow. The user experience is ultimately shaped by this chain, not by the reliability of any single service in isolation.

A useful way to think about this is that availability is multiplied across every component on the critical path. The weakest component has the largest influence, but every component matters. The end-to-end experience is the product of everything on the critical route between request and response.

There is also a different structure worth contrasting: a parallel system. In this setup, multiple instances of the same service can handle the same request, and the system only fails if all of them fail at the same time. Assuming failures are independent — which is an important condition we will return to — two instances each at 99.9% availability combine to approximately 99.9999% availability, since both would need to be down simultaneously for the system to fail.

But here is a caveat. In real systems, failures are often not independent. Two instances running in the same data center can both fail during a power outage. Two instances sharing the same database can both go down when that database fails. When failures correlate like this, the theoretical 99.9999% disappears. You are left with something much closer to 99.9%.

This leads to a simple rule. Parallelism only helps if the parallel things can fail separately. That means different servers, different power circuits, different network paths, and ideally different dependencies. If they share a single point of failure, they are not truly parallel.

Most real-world architectures are a mixture of both patterns. Some paths are sequential, forming a strict dependency chain, while others introduce redundancy and parallelism. The end-to-end availability is ultimately governed by the critical path — the sequence of dependencies a request must traverse from entry to response.

Every additional dependency adds weight to that path. And over time, it is the structure of those dependencies, more than the quality of individual services, that determines how resilient the system truly is.

Five Dimensions of Availability

Now that we understand why systems fail, we can talk about how to design against failure. It is helpful to organise techniques into five dimensions. Each dimension exists to answer a specific way systems break.

Redundancy — handles instance failure

This is the simplest idea. Do not have a single copy of anything. Run multiple instances of your service. Place them in different failure domains (what AWS calls availability zones, and other clouds call zones or datacenters). Use multiple regions if you can afford it. Redundancy turns a single point of failure into a survivable event.

Fault isolation — handles blast radius

Redundancy is useless if a failure can spread. Imagine you have two copies of a service running on the same server. A power failure kills both. That is not true redundancy. Fault isolation means building barriers between components. If one service leaks memory, it should not bring down its neighbour. If one database replica corrupts data, the others should be untouched.

Graceful degradation — handles dependency failure

Not every failure can be prevented. Sometimes a dependency goes down. Graceful degradation means that when something fails, the rest of the system continues working as well as it can. If the product recommendation service is down, the checkout should still work. If the review service is slow, the page should load without reviews. Partial functionality can be better than no functionality.

Recovery speed — handles inevitable failure

Availability is not just about how often something fails. It is also about how quickly it recovers. This is measured by two numbers.

Mean Time Between Failures (MTBF) tells you how long the system typically runs without incident.

Mean Time To Recovery (MTTR) tells you how long it takes to get back to normal after a failure. Improving MTTR is often cheaper than improving MTBF. Automating restarts, reducing deployment times, and maintaining good runbooks all help.

Load management — handles overload failure

Systems do not only fail because of bugs. They also fail because of overload. Too many requests arrive at once. The service cannot keep up. Queues grow. Memory fills. Eventually, the service starts timing out or crashing. Load management techniques like rate limiting, backpressure, and load shedding protect the system from itself. It is better to reject some requests quickly than to serve all requests and fail completely.

These five dimensions are not a checklist. They are a system of thinking. When you face a failure, ask yourself: which dimension failed, and which technique would have contained it?

Principles Over Patterns

In the microservices world, you often hear familiar patterns come up again and again — retries, circuit breakers, timeouts, idempotency. These are valuable tools, and they play an important role in building resilient systems. But their real strength comes from the principles they are built on. Once you understand those principles, the patterns become much easier to apply, combine, and adapt to different situations.

The principle of transient failure tolerance says that some failures are temporary. A network packet drops. A database connection pool is briefly exhausted. A service restarts. If you wait a moment and try again, the request may succeed. The pattern for this is retries. But retries must be used carefully. If you retry immediately, you might hit the same transient condition. If you retry too aggressively, you might turn a small blip into a retry storm, where many clients repeatedly retry at the same time, amplifying load on the system. So we add exponential backoff, then jitter to spread retries across time, and finally limits on total retry attempts.

The principle of cascading failure prevention says that one slow or failing component should not drag down everything else. If a downstream service is timing out, your service should stop trying to reach it for a while. The pattern for this is the circuit breaker. It works like an electrical circuit breaker. When failures cross a threshold, the circuit opens, and all subsequent requests fail immediately without attempting the downstream call. After a cooldown period, the circuit half-opens to test recovery. This gives the downstream service time to stabilise without being flooded.

The principle of resource exhaustion avoidance says that no request should be allowed to consume unlimited resources. If a database query takes ten minutes, it should not be allowed to hold a connection for that long. The pattern for this is timeouts. Every network call, every database query, every external API call should have a timeout. Without them, a single slow dependency can exhaust your entire connection pool and bring the whole service down.

The principle of safe retries says that repeating a request should not cause unintended side effects. Imagine a user taps "Pay Now" and the request times out. You retry it. But what if the first request actually succeeded, and the timeout was just a slow response? Now you have charged the user twice. The pattern for this is idempotency. An idempotent operation is one that can be applied multiple times without changing the result beyond the first application. Payment systems often use idempotency keys: the client generates a unique key for each payment attempt, and the server records which keys it has already processed. A retry with the same key is recognised and ignored.

These principles are not independent. They reinforce each other. Retries without idempotency create duplicate charges. Circuit breakers without timeouts leave threads hanging. Learn the principles first. The patterns follow naturally.

Availability Across the Stack

One of the most useful ways to think about availability is to follow a single request from the user's click to the database and back again. Failures look very different depending on where in the stack they occur.

A user opens your payment app. The first thing that happens is a DNS lookup. The domain name must be resolved to an IP address. If your DNS provider has an outage, users cannot reach you at all. Edge failures are rare, but when they occur, they typically result in a complete loss of access for users.

Next, the request hits a CDN or an API gateway. The CDN serves static content. The gateway handles routing, authentication, and rate limiting. If the gateway fails, no request reaches your services, regardless of how healthy those services are.

Then the request enters your service layer. This is where your business logic lives. The service might call other services, which call others. Each hop introduces a new dependency and a new potential failure point.

Finally, the request reaches the data layer. Databases, caches, and object storage. This layer is often the hardest to make highly available because state is difficult to replicate correctly.

Here is the uncomfortable truth: availability rarely breaks in your own business logic. It breaks at the boundaries—where your system meets the network, the database, or someone else's API.

A stateless service with a simple API is relatively easy to make available. Run multiple copies. Put a load balancer in front. Done. But a service that depends on a database, three other microservices, a message queue, and an external payment API is fragile. The chain is long. The parallel paths are few.

This leads to a practical rule of thumb. Invest in edge resilience first. Then invest in dependency resilience. Then worry about your own service. It is the dependencies that take you down most.

Dependencies You Do Not Control

This is the hardest truth in distributed systems. You do not control your dependencies. Your database is managed by a cloud provider. Your payment gateway is a third-party API. Your authentication service might be shared across your entire company.

You inherit their availability. If your payment gateway has 99.9% availability, your payment flows cannot exceed 99.9% availability regardless of what you do on your side. That is a hard upper bound. No amount of clever engineering can exceed the availability of a dependency on your critical path.

Your system is only as available as the least reliable thing on that path.

So what can you do?

The first strategy is timeouts and retries. This is the minimum acceptable baseline. Set a reasonable timeout for every external call. Implement retries with exponential backoff and jitter. This handles many transient failures at low cost.

The second strategy is fallbacks. A fallback is an alternative path when the primary dependency fails. If the payment gateway is slow, maybe you can queue the payment and process it asynchronously. If the product catalogue service is down, maybe you can serve stale data from cache. If the review service is unavailable, maybe you just render no reviews. Partial functionality is always better than a failed page.

The third strategy is asynchronous processing. If a dependency does not need to be called in the request path, do not call it there. Put a message on a queue. Let a background worker process it when it can. This decouples your availability from the dependency's availability entirely. The user gets a response immediately. The work happens when conditions allow.

The fourth strategy is feature degradation. If a critical dependency is genuinely down, be honest about it. Show the user a clear message. Disable the affected feature. Provide a fallback experience. If the authentication service is down, maybe you allow guest checkout. That is better than failing completely.

The important thing is to know which dependencies are truly critical. Very few are. Most can degrade gracefully if you design for it.

The Cost of Availability

At this point, you might be thinking: why not just make everything 99.999% available? Why not run multiple regions, active-active clusters, and automatic failover for everything?

The answer is cost. And not just financial cost, though that is significant. Complexity cost. Operational cost. Cognitive cost.

Every additional nine increases your burden dramatically. Going from 99.9% to 99.99% typically requires multi-region deployment. That means replicating data across geographic distances, which introduces consistency challenges. It means handling split-brain scenarios, where both regions believe they are the primary. It means testing failover regularly, because untested failover is not failover.

Going from 99.99% to 99.999% requires even more. You need active-active clusters, which are extremely difficult to operate correctly. You need automated recovery from every conceivable failure mode. You need to handle network partitions gracefully.

At this level, we run into a deeper constraint in distributed systems. The CAP theorem states that in a distributed system, in the presence of a network partition, you must choose between consistency and availability. You can serve requests and potentially return stale data, or you can refuse requests until consistency is guaranteed. You cannot do both.

Most systems choose availability over consistency for non-critical data. For payments, you might choose consistency. That choice has real availability consequences—and it is the right choice to make deliberately, not by accident.

The right question is not "how can I make this more available?" The right question is "what availability does this feature actually need?"

An internal reporting dashboard can live with 99.9%. A payment system might need 99.99%.

This is where the concept of an error budget becomes valuable. An error budget is the amount of unavailability you are willing to tolerate over a given period. For a 99.9% SLO, your error budget is about eight hours and forty-five minutes per year. As long as your downtime stays under that limit, you are meeting your goal.

This changes the conversation. Instead of asking "how do I prevent all failures?" you ask "how do I stay within my error budget?" That is a far more productive—and honest—question.

Observability: How to Measure Availability with SLIs and SLOs

You cannot improve what you cannot measure. And you cannot measure what you do not define.

We established SLIs and SLOs earlier. Now let us talk about what to actually measure.

For availability, the most common SLI is success rate: what proportion of requests received a successful response within the acceptable time bound?

You should also measure latency percentiles. The p99 latency is the latency that ninety-nine percent of requests are faster than. If your p99 is two seconds, one percent of users are waiting more than two seconds. For availability purposes, a very slow request is effectively unavailable. Many teams count requests that exceed their latency SLO as errors in their availability calculations—which is the right call.

Error rate is another essential metric. But not all errors are equivalent. A 500 internal server error signals something different from a 503 service unavailable, which signals something different from a 429 too many requests. Categorise your errors so you know what is actually failing and where.

Saturation metrics tell you why failures might be starting to happen before they fully arrive. CPU utilisation, memory usage, queue depth, connection pool saturation. When these metrics climb, availability often follows shortly after. Saturation is a leading indicator of failure, and monitoring it gives you time to act.

Here is a more grounded way to think about something that is often overlooked in theoretical discussions. Most outages are not first detected because a server goes down. They surface because users begin to experience failures. In practice, your monitoring is only as effective as its ability to identify user-impacting issues before they reach your users.

And here is a practical rule for alerting that tends to separate healthy systems from noisy ones. You don’t alert on every individual failure. You alert when your error budget is being consumed faster than expected. A single failed request is usually just noise. A sustained trend that would exhaust your error budget within the next hour is something worth acting on.

This change in thinking is important. It shifts you away from a reactive posture—where every error triggers an alert—towards a more measured approach, where small levels of failure are treated as normal operating conditions. In distributed systems, that is exactly what they are.

The goal is not to eliminate all failures. The goal is to operate safely within a defined failure budget.

Where to Go From Here

We have covered a lot of ground. We started with a user clicking "Pay Now" and nothing happening. We defined availability as user-perceived success within a time bound. We explored why distributed systems fail, from misplaced network assumptions to cascading dependencies. We saw how availability multiplies across a chain of services and why the critical path determines your ceiling. We organised techniques into five dimensions that each answer a specific failure mode. We learned the principles behind common patterns. We walked the stack from edge to data layer and saw that availability breaks at boundaries, not in business logic. We faced the hard truth of dependencies we do not control. We acknowledged the cost of chasing nines. We tightened the CAP theorem to its practical meaning. And we ended with measurement, error budgets, and a more mature way to think about alerting.

The next time a user clicks "Pay Now" and nothing happens, that moment is not just a bug. It is the visible edge of your system's availability design—the place where every decision about redundancy, fallbacks, timeouts, and SLOs either paid off or did not.

If you take one thing away from this article, let it be this: availability is not about preventing failure. It is about designing for failure as a first-class citizen. It is about understanding that your system will break, deciding how much breakage you can tolerate, and building the mechanisms to stay within that tolerance.

The best systems are not the ones that never fail. They are the ones that fail gracefully, recover quickly, and never surprise their users.

Systems earn trust not by never failing, but by failing in ways users can understand, recover from, and eventually forget.

Now go design something that breaks well.