Skip to main content
An Open Mindset for High Availability System Design

Β·

An Open Mindset for High Availability System Design

PREFACE

Hello everyone! Here we are again, meeting in the final days of 2022. This time, I'm returning to my usual content: sharing knowledge and experience from my professional journey.
For those who are new here, my articles generally revolve around Backend, Blockchain, and DevOps. Through sharing knowledge with the community, I hope to receive feedback that helps me improve my skills and deepen my expertise. I'll also continue to publish personal reflections on work and life from my own perspective.

THE TOPIC

In today's article, I'd like to share my general perspective on system design. In a way, this is also an opportunity to summarize and reflect on the knowledge I've accumulated about systems over the past year.
I hope this article will encourage you to adopt an open mindset and find your own answer to the following question:
How do we design a system that achieves High Availability (HA)?

FOUNDATIONAL KNOWLEDGE

Before diving into the article, I'd recommend having some basic knowledge of the following topics to get the most out of the discussion:
Now, if you're comfortable with these fundamentals, let's dive in and explore how to design a system with the highest possible level of High Availability (HA).

HIGHLY AVAILABLE SYSTEMS

What?

What is a Highly Available system?
  • A highly available system is designed to operate continuously, 24/7, under virtually any circumstances: server failures, earthquakes, tsunamis, nuclear wars, alien invasions... The goal is to keep annual downtime to just a few hundred or even a few dozen milliseconds.

Why?

Why should we put so much effort into this?
  • If your website serves hundreds of thousands or even millions of users, just a few seconds of downtime can cause unimaginable losses, both for users and for the service provider.
  • In case you haven't heard, there was apparently an outage a few weeks ago involving some "M" telecommunications provider, which suddenly lost connectivity for an entire afternoon. This may have affected the livelihoods and daily operations of many customers who couldn't process important transactions or handle critical matters in time. (?!) (Disclaimer: This information has not been independently verified by the author.)

How?

So, how do we implement it?
  • Once we've established the goal, it's time to get our hands dirty. But where should we start, and how should we approach the implementation? Let's examine and analyze the architecture diagram below.
In the architecture above, apart from the Client layer (where local caching can be applied), I'll divide the system into several layers:
  • DNS Layer
  • Load Balancer Layer
  • Services Layer
  • Database Layer
Have you spotted the core principle behind this architecture yet?
The key to achieving HA is simply to eliminate single points of failure (SPOFs) throughout the system.
In centralized architectures, each layer often has a single point of failure. However, load balancing can be implemented across multiple layers, not just the Services Layer, where we're most accustomed to seeing it.

Where?

Where can we place load balancers?
Now that we understand the principle, let's break down the architecture layer by layer.

DNS Layer

At the Domain Name System (DNS) layer, we can use DNS-based load balancing to direct domains and subdomains to servers hosted by different providers, such as DigitalOcean, AWS, Azure, and GCP.
My go-to DNS provider is Cloudflare. Of course, DNS itself can also become a point of failure, so you may want to explore combining multiple DNS services for greater resilience.
Choosing a Virtual Machine provider is another important consideration, as physical servers are distributed across different locations around the world. Depending on your target market, you should select suitable deployment regions to minimize latency and reduce system risks.
In Southeast Asia, most servers are typically hosted in Singapore, with a smaller number located in Taiwan.

Load Balancer Layer

Once traffic reaches a server, we can introduce another load-balancing layer to distribute requests across different server clusters.
One widely used load-balancing solution is Nginx. However, this layer can itself become a single point of failure. To address this, we can configure a master–standby architecture for the load balancers using Nginx Plus.

Services Layer

Next is the Services Layer, where we commonly run multiple instances of the same service to handle incoming requests concurrently.
There are various tools and approaches to consider at this layer, ranging from relatively simple solutions to more advanced orchestration platforms, such as PM2, Docker Swarm, and Kubernetes.
In the architecture below, I'm using Docker Swarm mode to deploy multiple service instances that can process requests in parallel.

Database Layer

At the bottom of the stack is the Database Layer. Here, we can avoid a single point of failure by deploying multiple database instances and applying appropriate replication and traffic-distribution strategies.
For MongoDB, I'll focus on two approaches: Replication and Sharding.
I'll cover the detailed design, configuration, and implementation of a highly available MongoDB architecture, along with the CQRS pattern, in a more comprehensive article in the future.

CONCLUSION

I hope this article has given you a general overview of designing highly available systems and shown that HA strategies can be applied at different layers of an architecture.
For traditional centralized systems, reducing downtime is possible, but eliminating it entirelyβ€”achieving approximately zero downtimeβ€”is extremely difficult, if not practically impossible.
This is where modern decentralized systems, such as Blockchain, may offer promising new approaches and open up potential opportunities in the Web3.0 space.
Of course, we also need to avoid overengineeringβ€”or, as the saying goes, "using a sledgehammer to crack a nut." We should carefully evaluate the needs of each system and choose an appropriate solution.
Building a massive HA infrastructure introduces additional complexity and operational costs. If the actual workload doesn't justify it, such an architecture may create more problems than it solves.
However, maintaining an open mindset is essential. Having a broad understanding of the available approaches gives us the knowledge and perspective needed to evaluate trade-offs and choose the most suitable solution, ultimately saving considerable time and effort that can be spent on work that truly matters.
I'd also like to reiterate that everything shared in this article comes from my personal knowledge and experience, so there may be gaps or inaccuracies. Please treat it as a reference rather than an absolute authority, and feel free to point out anything I've overlooked.
If you found this article helpful, please leave an upvote and share your thoughts or suggestions in the comments below.
Wishing everyone a happy New Year 2023! I look forward to sharing more articles with you in the future.
β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–„β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–„β–‘β–‘β–‘β–‘
β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–Œβ–’β–ˆβ–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–„β–€β–’β–Œβ–‘β–‘β–‘
β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–Œβ–’β–’β–ˆβ–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–„β–€β–’β–’β–’β–β–‘β–‘β–‘
░░░░░░░▐▄▀▒▒▀▀▀▀▄▄▄▀▒▒▒▒▒▐░░░
β–‘β–‘β–‘β–‘β–‘β–„β–„β–€β–’β–‘β–’β–’β–’β–’β–’β–’β–’β–’β–’β–ˆβ–’β–’β–„β–ˆβ–’β–β–‘β–‘β–‘
β–‘β–‘β–‘β–„β–€β–’β–’β–’β–‘β–‘β–‘β–’β–’β–’β–‘β–‘β–‘β–’β–’β–’β–€β–ˆβ–ˆβ–€β–’β–Œβ–‘β–‘β–‘
β–‘β–‘β–β–’β–’β–’β–„β–„β–’β–’β–’β–’β–‘β–‘β–‘β–’β–’β–’β–’β–’β–’β–’β–€β–„β–’β–’β–Œβ–‘β–‘
β–‘β–‘β–Œβ–‘β–‘β–Œβ–ˆβ–€β–’β–’β–’β–’β–’β–„β–€β–ˆβ–„β–’β–’β–’β–’β–’β–’β–’β–ˆβ–’β–β–‘β–‘
β–‘β–β–‘β–‘β–‘β–’β–’β–’β–’β–’β–’β–’β–’β–Œβ–ˆβ–ˆβ–€β–’β–’β–‘β–‘β–‘β–’β–’β–’β–€β–„β–Œβ–‘
β–‘β–Œβ–‘β–’β–„β–ˆβ–ˆβ–„β–’β–’β–’β–’β–’β–’β–’β–’β–’β–‘β–‘β–‘β–‘β–‘β–‘β–’β–’β–’β–’β–Œβ–‘
β–€β–’β–€β–β–„β–ˆβ–„β–ˆβ–Œβ–„β–‘β–€β–’β–’β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–’β–’β–’β–β–‘
β–β–’β–’β–β–€β–β–€β–’β–‘β–„β–„β–’β–„β–’β–’β–’β–’β–’β–’β–‘β–’β–‘β–’β–‘β–’β–’β–’β–’β–Œ
▐▒▒▒▀▀▄▄▒▒▒▄▒▒▒▒▒▒▒▒░▒░▒░▒▒▐░
β–‘β–Œβ–’β–’β–’β–’β–’β–’β–€β–€β–€β–’β–’β–’β–’β–’β–’β–‘β–’β–‘β–’β–‘β–’β–‘β–’β–’β–’β–Œβ–‘
░▐▒▒▒▒▒▒▒▒▒▒▒▒▒▒░▒░▒░▒▒▄▒▒▐░░
β–‘β–‘β–€β–„β–’β–’β–’β–’β–’β–’β–’β–’β–’β–’β–’β–‘β–’β–‘β–’β–‘β–’β–„β–’β–’β–’β–’β–Œβ–‘β–‘
β–‘β–‘β–‘β–‘β–€β–„β–’β–’β–’β–’β–’β–’β–’β–’β–’β–’β–„β–„β–„β–€β–’β–’β–’β–’β–„β–€β–‘β–‘β–‘
β–‘β–‘β–‘β–‘β–‘β–‘β–€β–„β–„β–„β–„β–„β–„β–€β–€β–€β–’β–’β–’β–’β–’β–„β–„β–€β–‘β–‘β–‘β–‘β–‘
β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–’β–’β–’β–’β–’β–’β–’β–’β–’β–’β–€β–€β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘

Subscribe to the newsletter

Get new posts on Backend, Blockchain, DevOps, and life notes.

Loading Comments...