One xlarge shard, no sharding theater
Before drawing three shards, I want everyone to actually run the numbers on one big box. db.xlarge holds 4 TB; the dataset is ~3.9 TB. One shard, RF 3, quorum. It fits, it serves the miss stream forty times over, and there is no hash ring, no cross-shard anything, no resharding runbook. The edge eats the traffic like in the top design.
The honest catch, stated with numbers instead of vibes: at +60%/yr you cross 4 TB in about five months. So this design has an expiry date, and I'm posting it anyway, because "shard on day one" and "shard in month five with five months of real access-pattern data" are different engineering decisions and the second one is often better. You'll shard around what's actually hot instead of what you guessed.
Have you measured, or are you sharding because the blog post did?
4 Comments
Sign in to join the discussion.
- Lena Fischer@lena_fischer
the migration plan needs to exist before month four, in writing. moving 4 TB while holding RPO 0 and quorum acks is not a weekend. otherwise this is fine as a v1.
- Split Brain@split_brain
i actually like this one. everyone prices the resharding risk and nobody prices the carrying-two-extra-shards-you-didn't-need risk. deferring with a labeled expiry date is a position, not an accident
- Aisha Khan@aisha_khan
@dan_okafor the work isn't the same work though. month-five you knows the hot set, the real growth rate, and whether this product even survived. day-one you is guessing on all three. i'll take the informed migration
- Daniel Okafor@dan_okafor
five months is a countdown, not a plan. the migration you're deferring lands on 4TB at RF3 with live quorum writes. you saved zero total work and moved it to the worst possible date