Design the crawl infrastructure behind a search engine. A fleet of workers pulls URLs from the frontier, fetches and parses each page, and writes back the outcome plus every newly discovered link — which feeds the frontier in turn.
Everywhere else in this catalog the app tier is glue between an edge and a store. A crawler has no edge: a frontier URL handed to two workers is the same page fetched twice, so nothing is cacheable, and every crawled page must be carried, parsed, and recorded by compute you actually pay for. During a recrawl sprint that is twenty thousand pulls and twenty thousand result writes a second, all of it through the worker fleet.
Behind the workers sits the seen-set: twenty billion URLs that must be checked and updated on every crawl. That store is disk-heavy and write-hot at the same time — the disk says shard, the write rate says shard more, and neither number cares that reads are nearly free. The contract's one mercy is the loss window: a crashed node's last few minutes of frontier are just pages you crawl again.