Every system is a set of boxes running programs. Each program runs out of a different resource first. Knowing which one is most of the job.
What system design is
System design is designing a system that meets its requirements within its constraints.
The requirements are what the system must do. Serve this many requests per second. Answer within this latency. Never lose a write. Stay up when a machine dies. The constraints are what you have to do it with: money, machines, people, and time.
Everything else in the discipline follows from that sentence. You make tradeoffs because no design has every property at once. You plan for spikes and outages because the requirements hold on the worst day, not the average one. You decide how data moves and where it rests because those two choices set most of the cost.
Interviewers state the target as a service level objective, or SLO: “p99 latency under 200 ms”, “99.9% of requests succeed”. An SLA is the contract version of the same number with penalties attached. You design to the SLO.
Nodes, edges, and flows
A system diagram has three kinds of things in it.
A node is a job. “The database.” “The cache.” “The API servers.” One node maps to one machine, or to many identical machines doing the same job. When a job needs more capacity, you add machines behind the same name. That is horizontal scaling, and it is why a load balancer exists at all.
An edge is a network call from one node to another.
A flow is the path one request takes across the edges, and what happens to it when a node on that path is slow or gone.
Every node is a box, or a pile of identical boxes. Every box runs a program. And every program wants a different mix of the same four resources.
The four resources
CPU
cores
Does the work.
A core executes one stream of instructions at a time: parsing a request, running a query plan, encrypting bytes, comparing strings. A program that can only use one thread gets one core, no matter how big the box is.
RAM
gigabytes
Holds what is hot.
Reading from RAM takes about 100 nanoseconds, because it is an electrical signal to a chip with no moving parts and no protocol in between. RAM forgets everything when the power goes, so it can never be the only copy of anything that matters.
Disk
gigabytes and IOPS
Holds what must survive.
Two limits, not one. Capacity is how many bytes fit. IOPS is how many separate reads or writes per second the drive can serve. A random read from an SSD takes about 100 microseconds, a thousand times RAM, because the request goes through a storage controller out to flash cells. Appending to the end of a file is far cheaper per byte than random access, because the drive streams instead of seeking.
Network
bandwidth and round trips
Moves bytes between boxes.
Bandwidth is bytes per second through the card. Round trips are how many times a request has to cross to another box and wait for the answer. One round trip inside a datacenter costs about half a millisecond. A request that touches five boxes in sequence pays five of them before it can respond.
The numbers above are not trivia. They are the reason every design move in this course exists. Put them on one ladder and the gaps are the whole story.
How far away each thing is
Read from RAM100 ns
An electrical signal to a chip. No moving parts, no protocol.
×1 · 1 second
Random read from an SSD100 µs
Through the storage controller to flash cells and back.
×1,000 · 17 minutes
Round trip to another box, same datacenter500 µs
Two network stacks, a few switches, and the wait for a reply.
×5,000 · 1.4 hours
Random read from a spinning disk10 ms
The head physically moves and the platter has to spin under it.
×100,000 · 28 hours
Round trip to another region50 to 150 ms
The speed of light over thousands of kilometers, plus every router on the way.
×1,000,000 · 12 days
Each tick on the track is a power of ten, so a bar counts 10x steps above a RAM read, not length. The second number is the same gap on a human clock where a RAM read takes one second: an SSD read is a coffee break, and a round trip to another region is a vacation. Every design move you will learn is a way to avoid paying a rung you do not have to.
There is a fifth quantity, and it is not a resource: money. The four resources are what you buy. Dollars are what you pay. Every tradeoff you will make is one of the four traded for another, or for dollars.
What each box runs out of first
This is the part that separates a senior answer from a list of component names. Anyone can say “put a load balancer in front and a database behind it.” A senior engineer says what each of those boxes will run out of first, because that decides how it fails under load and how you scale it.
Relational database (Postgres)
Runs out of first
RAM
Then
Disk IOPS
Barely uses
Network
Postgres reads rows through a buffer cache in RAM. A page already in memory costs about 100 nanoseconds. The same page on an SSD costs about 100 microseconds. So a database is fast exactly when its hot data, the working set, fits in RAM. Disk sees the misses, plus one write per commit: the write-ahead log must reach the disk before the client hears “done”, so write latency is bounded by the disk. CPU is per-query work. Point lookups are cheap; joins and sorts are not. Network stays small because rows are small.
The naive answer is “a database needs disk.” The senior answer is “a database needs RAM, with disk for durability.”
Cache (Redis)
Runs out of first
RAM
Then
One CPU core
Barely uses
Disk
The whole dataset lives in memory. That is the point. Commands execute on a single thread, so one instance tops out near one core’s worth of commands per second, on the order of 100,000, no matter how big the box is. To go past that you run more instances, not a bigger one. Network carries every request and reply. There is no disk unless you turn on persistence.
API server
Runs out of first
CPU
Then
Network
Barely uses
Disk
Every request gets parsed, authenticated, validated, and serialized. That is all CPU. Nothing persists on the box, so a dead one is replaced by a fresh one and nobody notices. That is why this is the easiest box to scale: add more of them behind the load balancer.
Load balancer
Runs out of first
Network
Then
CPU
Never uses
Disk
Every byte crosses its network card twice, once in and once out. TLS handshakes cost CPU. The table of open connections lives in RAM. It writes nothing to disk, because it stores nothing. If it dies, clients retry against another one. A box with no state is the easiest kind to have many of.
Log or queue (Kafka)
Runs out of first
Disk, sequential
Then
Network
Barely uses
CPU
It appends every message to the end of a file and reads the file back in order. Sequential writes are the cheap kind of disk, so one broker absorbs write rates that would flatten a database doing random writes. The operating system keeps the tail of the file in RAM, so recent reads never touch the disk at all.
Object store (S3)
Runs out of first
Disk capacity
Then
Network egress
Barely uses
CPU
Cheap per byte, slow per request. Nobody computes there. Clients fetch whole objects, so the bill is how much you store and how many bytes leave.
Why this is the first lesson
Every move you will learn later is a resource trade.
A cache in front of a database spends RAM on a second box to save disk IOPS on the first.
Sharding splits one box's disk and CPU across many boxes, at the price of extra network hops and coordination.
Replication spends disk and network on copies to buy availability and read throughput.
A queue spends cheap sequential disk to shield an expensive box from a spike.
When you can name what a box runs out of first, you can predict how it fails under load before you draw it. That is what the grader checks. It is also what an interviewer is listening for when they ask “why is that a separate box?”
Check yourself
A Postgres box has 64 GB of RAM and a 2 TB SSD. The hot working set is 100 GB. What does it run out of first?
Disk IOPS. The working set does not fit in RAM, so a large share of reads miss the buffer cache, and each miss is a 100 microsecond random read. The 2 TB of capacity is irrelevant. The drive’s operations per second is the wall.
Your API servers are at 90% CPU. What do you change?
Add more API servers behind the load balancer. They hold no state, so a new one is as good as an old one. A bigger disk or more RAM on the existing boxes does nothing for a CPU-bound program.
Why does a load balancer need no disk?
It stores nothing. Its only state is the table of open connections, which lives in RAM and is rebuilt by clients reconnecting if the box dies.
A Redis box with 32 cores reports 100% CPU on one core and the others idle. What is happening, and what is the fix?
Redis runs commands on one thread, so one core is saturated and the other 31 cannot help. The fix is more Redis instances, each owning a slice of the keys, not a bigger box.
Try it
Name the bottleneck before the grader does
Open the Key-value store problem. For each box you place, say which resource it will run out of first. Then grade the design. The verdict names the node that saturated, so you can check your guess against the simulation.