I got laid off for the first time back in September (surprise!) and it took a giant toll on me emotionally. For the last 10 or so years I’d managed to muscle my way through engineering jobs despite not having a formal degree in CS. I’ve had the immense privilege of working at some really cool companies, working my way from the front of the frontend through the bowels of the stack and (for a brief moment) into the cold heart of bare metal. I’ve worked on product, systems, support tooling, growth/sales pipelines, billing, and miscellaneous developer workflows. In that time, I’ve written code in Ruby, JavaScript, TypeScript, Golang, Elixir and (some) Rust and worked with some crazy intelligent people. I’ve given talks internationally and written blog posts about various deeply technical pursuits. While I’m by no means an expert at all or many of things, I share this to say that I have a chunk of experience under my belt. ...

2024 was the year when I finally transitioned from surviving to thriving. With everything that transpired over the last few years, I got lost in the chaos of my reality and it’s taken me a second to “get back”. I’m not quite who I used to be, but I’ve become a version of myself that I’m quite pleased with, same same but different. The year came and went, for the most part, without incident. And (for once) I have no one to blame but myself for any and all shenanigans I found myself in. I grew, I learnt, and successfully averted any major crisis. In no specific order, here’s everything that happened this year in a nutshell: ...

Change is inevitable, especially so in applications that ebb and flow with system changes and user requirements. Generally, changes are codified in the form of an event; an event like a ddos attack for instance triggers a reaction or a response from one or more nodes, which ultimately mutates the overall state of the universe in a system. Undoubtedly, events rarely happen as one offs. At any given time, a sequence of events flows through the system. Whether it be a cascade of related events or multiple isolated events, the system is consistently responding and adapting. ...

Distributed nodes share neither memory nor a logical clock. In order to effectively keep a system in running order, nodes need some way to communicate with one another to share data and coordinate complex tasks. Interprocess communication (IPC) plays an integral role in this. IPC is the mechanism that enables a server to query a database instance and resolve a request with the associated data successfully. Without it, nodes remain stranded akin to the people in the Tower of Babel, connected by infrastructure but unable to communicate with one another. ...

In a distributed system, failures are to be expected. Whether it be a fault of the network or the node, failure is disruptive to the entire system. Failed processes struggle to recover and receive updates. Healthy nodes that try to connect to a failed node might fail completely or receive stale information, which can jeopardize the integrity and overall (C)onsistency of the system. To mitigate this, nodes have a failure mode, which it codifies in the form of a membership protocol. ...

In a distributed system, communication is crucial to ensuring no node becomes stale. Updates can happen in 2 main ways, through consensus, or through gossip. In a consensus model, nodes attempt to reach quorum by mutually agreeing on the order of updates or periodically voting on and rotating the leader in charge of distributing updates to fellow nodes—I covered the ins and outs of this in an earlier post, if you’re curious to learn more about it. Gossip is a concept analogous to consensus. It similarly keeps nodes updated except it achieves this via epidemic theory; If one node is infected, the entire population of nodes will eventually become infected. ...

In an earlier post, I glossed over a concept that forms a fundamental tenet of systems design; the CAP theorem. To reiterate, the CAP theorem states that of 3 desirable traits in a distributed system, Consistency, Availability and Partition tolerance, only 2 can be prioritized at any given time. It highlights the tradeoffs we make when we design for a distributed setting. The CAP theorem was birthed from the move away from vertical scaling where we simply sized up machines for greater processing capacity to horizontal scaling where machines work in a distributed and parallel fashion. The CAP theorem specifically focuses on databases, since it highlights the difficulty of maintaining state when reads and writes can originate from anywhere and can easily be disrupted in the event of a network failure. ...

What does it mean to be consistent? In an earlier post, we talked about the importance of building consensus when it comes to ensuring consistency in a distributed setting. Loosely, consistency focuses on keeping all nodes in a system up to date on changes as they happen in real time or near real time. This ensures the reliability of the system as a whole, so users are never (fully) let down when parts of the system unexpectedly go down. ...

In a distributed setup where nodes share the load of processing requests and storing/retrieving data, reaching consensus is crucial to ensuring consistency in an application. To return to the concert ticket sales example from the last post, we wouldn’t want 2 users buying the same seat since that violates the condition of assigned seating. Consensus can then be described as ensuring all nodes agree on a specific value or a state in spite of a potential failure or delay; there lies the rub. ...

Demystifying Distributed Systems: the what and why If you deploy an app today, dealing with a distributed system is an inescapable problem. Whether it’s understanding how to balance requests, or synchronizing data replication, at least some aspect of your deployment will require orchestrating and coordinating multiple servers to operate in unison. The goal of “distributedness” here is to have things work across regions and systems in a way that is completely imperceptible to a user. ...