The hard part of a news feed is not storing posts. It is deciding when, where, and how many times those posts should be materialized. Follower distributions are skewed, ranking changes by request, queues fall behind, and users still expect the first screen in a few hundred milliseconds.
I treat a feed as two related systems with different ownership. Post storage owns the durable content record. Timeline materialization owns an ordered, bounded set of candidate post references for a reader. Conflating them creates a data model that is expensive to mutate, difficult to rank, and almost impossible to recover cleanly.
01Reads are latency sensitivePrecompute where the amplification is affordable.02Writes are skewedOne celebrity can invalidate the average case design.03Recovery is a data pathRetries, replay, and degradation need explicit contracts.1. Requirements before components
The numbers below are working assumptions, not universal truth. Their purpose is to force architectural decisions. A regional professional network and a global consumer social product should not inherit the same topology just because both display a scrolling list.
users300M registered40M daily active; 12M peak hour activecontent60M posts/dayText and metadata in the post store; media in object storage + CDNreads800M feed loads/dayHigh read to write ratio with bursty session startsgraph350 follows median activeLong tail includes accounts with millions of followersfreshness5s normal targetExplicit eventual consistency window for materialized timelinesregions3 serving regionsLocal reads preferred; cross region propagation is asynchronousRough capacity estimation
| Dimension | Average | Design peak | Why it matters |
|---|---|---|---|
| Post writes | ≈ 700/sec | ≈ 7,000/sec | The durable write path is manageable; fan out is the multiplier. |
| Feed reads | ≈ 9,300/sec | ≈ 100,000/sec | Hydration, ranking, and cache misses dominate the latency budget. |
| Fan out references | ≈ 21B/day before filtering | Millions from one post | At 32 bytes/reference, raw logical writes are ≈ 672GB/day before protocol and replication overhead. |
| Post metadata | ≈ 120GB/day raw | 2–4× replicated | Media is excluded; object bytes belong on an object store backed by a CDN. |
| Active timeline cache | ≈ 480GB raw | 1–2TB practical | 40M active users × 500 references × ~24 bytes, then allocator and replica overhead. |
| Fan out network | ≈ 0.7TB/day payload | Several TB/day practical | Serialization, acknowledgements, replication, and traffic across zones amplify the logical payload. |
Example service objectives
feed.readp99 < 200msFirst 20 hydrated items from the reader's serving region.post.ackp99 < 350msAcknowledge after post + outbox commit, not after follower fan out.normal.freshness99% < 5sPosts from normal authors visible in materialized follower timelines.recoveryRPO ≈ 0 eventsCommitted outbox intent survives broker or publisher failure.These targets influence the design. Requiring synchronous visibility to every follower would put fan out in the post acknowledgement path and make celebrity latency unbounded. Accepting a measured propagation window lets the system commit durable intent quickly, then distribute asynchronously.
2. Separate the write path from the read path
A post write is a durable state transition. A feed read is a latency sensitive projection query. They share identifiers, but they should not share a synchronous failure boundary.
The post database remains authoritative for content. The timeline store contains disposable, rebuildable references not full copies of posts.
Commit intent, then propagate
User → gateway → post service → post + outbox transaction → broker → fan out workers.
ack after durable local commitCompose candidates, then hydrate
User → feed API → timeline IDs → celebrity candidates → post hydration → filter → rank.
fallback before timeout budget expiresBudgets are illustrative. The useful property is an explicit deadline and a cheaper fallback at each optional stage.
3. The producer dual write failure and transactional outbox
Publishing PostCreated after committing the post creates an unsafe gap: the database commit can succeed while broker publication fails. The post exists, the author sees success, and no follower timeline is ever updated. Publishing first only reverses the inconsistency. A feed can reference a post that never committed.
Post persistence and event publication are two independent writes.
The post database can provide local atomicity; the broker cannot join that transaction reliably.
Insert the post and an outbox record in one transaction. A relay publishes committed rows.
More writes, retention, relay monitoring, and duplicates when a relay crashes after publish.
The relay retries or publishes twice after an uncertain broker acknowledgement.
Persist the event ID in the outbox; consumers enforce idempotent timeline mutation.
BEGIN;
INSERT INTO posts (post_id, author_id, body_ref, created_at) VALUES (...);
INSERT INTO outbox (event_id, aggregate_id, type, payload)
VALUES (..., post_id, 'PostCreated', ...);
COMMIT;
-- an independent relay publishes pending outbox rows with retry + backoff
The outbox prevents lost publication intent, not duplicate delivery. If the relay publishes and crashes before marking the row sent, it publishes again. That is acceptable only because the downstream effect is keyed by (timelineOwnerId, postId) and implemented with a unique constraint, conditional write, or insert with set semantics.
4. Fan out is where the workload changes shape
Fan out on write converts read latency into write amplification. Fan out on read converts cheap writes into graph traversal and merge work during requests. Neither is universally correct because follower distributions, author activity, and reader activity are not uniform.
| Strategy | Write path | Read path | Primary failure mode | Best fit |
|---|---|---|---|---|
| Fan out on write | Push post ID to follower timelines | Fast bounded lookup | Write explosion and propagation lag | Normal authors, active followers |
| Fan out on read | Store post once | Fetch author streams, merge, rank | Unpredictable latency and graph pressure | Celebrity authors and authors with high amplification |
| Hybrid | Policy routes each author/post | Merge precomputed + pulled candidates | More operational and ranking complexity | Large systems with skewed distributions |
The routing threshold should be dynamic: follower count, author posting rate, active follower ratio, current worker lag, cache pressure, and infrastructure cost all matter.
For an extreme follower set, emit chunk jobs such as (postId, followerPageCursor). Do not let one worker hold a loop across two million users in memory.
5. The social graph is not the timeline
Follower relationships belong to a graph service or an access layer optimized for graph operations. The feed projection consumes that graph; it does not become its source of truth. In practice, adjacency lists often outperform a general graph database for the hot operations: page followers for an author, page followings for a reader, test whether A follows B, and apply follow or unfollow mutations.
- Followers by author: optimized for fan out enumeration; very large sets are partitioned and paged with cursors.
- Followings by reader: optimized for candidate generation and relationship filtering during reads.
- Partitioning: ordinary lists can hash by owner; celebrity lists need subpartitions or page ownership to avoid one hot shard.
- Caching: cache small adjacency lists and membership checks, but do not attempt to keep every list with millions of followers in one hot cache key.
What should happen on follow?
future onlyLowest write costNew posts appear after the relationship commits. Historical content is absent until discovered elsewhere.
Good for early products or weak history expectations.async backfillComplete but expensiveA background job copies historical references into the follower timeline.
Needs bounded depth, idempotency, and protection from follow/unfollow churn.recent windowPractical defaultBackfill the newest 20–100 eligible posts, merge by ordering key, then let future fan out continue.
Useful history without replaying an author's lifetime.Follow/unfollow is eventually reflected in materialized timelines, but the read path should enforce current blocks, mutes, and relationship policy. Unfollow does not require synchronously deleting every reference before returning success; hydration/filtering makes reads safe while asynchronous cleanup converges.
6. The broker moves load through time, but it does not remove it
The broker provides durable buffering, ordering within a defined partition, consumer coordination, and replay. It does not create downstream capacity. If 500k fan out mutations arrive per second and workers can sustain 300k, consumer lag is the honest signal that the freshness SLO is being spent.
Long retention, replay, partition ordering, and high sequential throughput. Operationally valuable when projections must be rebuilt.
Flexible routing, acknowledgement for each message, and direct work distribution. Replay usually needs explicit retention or republishing design.
Throughput matters, but so do replay horizon, ordering scope, operational skill, delivery semantics, and managed service constraints.
- Partition key: partitioning only by
authorIdpreserves author order but can create a hot spot around celebrity authors. A staged event can partition follower chunks byhash(timelineOwnerId)after the original post event is expanded. - Consumer groups: scale independent projections separately, including home timelines, search indexing, moderation, and notifications, so a slow consumer does not block unrelated work.
- Retries: classify transient and permanent failures, use exponential backoff with jitter, and keep attempt metadata.
- Poison events: send bounded failures to a DLQ with event ID, schema version, correlation ID, and sanitized cause. A DLQ needs an owner and replay procedure.
- Ordering: define the scope. Global post ordering is unnecessary; stable ordering keys for each author or timeline are normally sufficient.
Backpressure is a feature. An unbounded queue is delayed failure. Scale consumers only while timeline storage, network, and connection pools have safe headroom.
7. Idempotency is a storage property, not a retry slogan
Exactly once delivery is the wrong mental model once an event crosses the broker into an independent timeline database. The useful invariant is: applying PostCreated repeatedly produces one timeline reference for that owner and post.
timeline_entries
owner_id string
ordering_key int64
post_id string
UNIQUE (owner_id, post_id)
INDEX (owner_id, ordering_key DESC, post_id DESC)A worker crash halfway through a follower chunk is safe when each mutation is conditional or follows set semantics. The chunk can be retried from its start, or checkpointed at a stable follower cursor. Checkpointing reduces duplicate work; the uniqueness invariant preserves correctness when checkpoints are stale.
8. Timeline storage follows the access pattern
A logical timeline can be modeled as timeline:{userId} containing ordered post references. The score may be a creation timestamp for chronological feeds or a stable candidate ordering key. Redis sorted sets, a wide row modeled after Cassandra or Scylla, a partitioned key value model similar to DynamoDB, or a specialized store can all serve this pattern. The decision should follow operations, not brand preference.
insertAdd newest referenceConditional on owner + post identity.readFetch newest NSeek from an ordering cursor, not an offset.deleteRemove / ignore postAsync cleanup plus safety checks during hydration.trimBound the candidate listKeep hundreds or a few thousand useful references.Infinitely growing timelines for each user are unnecessary. The hot feed needs a bounded candidate window, not an archive of every post ever produced. Older pages can be reconstructed from author streams or durable secondary storage when the product actually requires deep history. Bounded timelines cap memory, compaction, migration, and rebuild cost.
Cursor pagination, not page 5,000
Offset pagination scans or skips increasing amounts of data and becomes unstable as new posts arrive. A cursor should encode the last stable sort boundary, such as (createdAt, postId) for chronological feeds or (rankingScore, postId) for a ranking snapshot. The post ID breaks ties deterministically.
GET /v1/feed?limit=20&after=opaque_cursor
decoded cursor = {
rankingSnapshot: "rnk_20260817_4f2",
score: 0.834219,
postId: "post_01K..."
}New content arriving between requests should appear on a refresh boundary, not reshuffle an active pagination session. A temporary ranking snapshot or maximum watermark reduces duplicates and missing items. The cursor stays opaque so the service can evolve its ordering model without exposing storage internals.
9. Timeline construction and ranking are separate concerns
Candidate generation should maximize useful recall within a hard size budget. Ranking then orders those candidates. Mixing the two makes it difficult to know whether a missing post was never generated, filtered by policy, or scored poorly.
- Chronological mode: stable timestamp ordering is cheap, explainable, and a valuable fallback.
- Engagement mode: combine recency, relationship strength, predicted engagement, and content quality without letting one signal dominate indefinitely.
- Policy layer: blocked or muted users, safety decisions, legal restrictions, deduplication, diversity, and ad spacing are authoritative filters, not optional ranking hints.
- Failure behavior: if feature retrieval or personalized ranking exceeds its budget, fall back to cached features or chronological ordering before the feed API deadline.
10. Cache architecture is explicit state
The cache is part of the architecture, not a transparent performance layer. Different objects have different freshness and invalidation rules: posts are mostly immutable, profiles change occasionally, graph relationships affect policy, timeline candidates are derived, and ranking features decay quickly.
post cacheLonger TTLInvalidate or version on edit/delete.profile cacheModerate TTLSmall objects; tolerate bounded staleness.graph cacheMembership focusedProtect correctness with policy checks during reads.timeline cacheUpdated on every writeBound size; rebuild from durable streams.feature cacheShort TTLVersion models and feature schemas.media CDNContent addressedKeep media bytes off feed servers.Request coalescing prevents hundreds of misses for the same hot key from becoming hundreds of reads from the backing store.
Hot content and celebrity author streams need deliberate treatment: replicate keys with heavy read traffic, use local worker caches for immutable post objects, partition large collections, coalesce concurrent loads, and apply admission policies so one viral object does not evict the useful working set. TTL jitter reduces synchronized expiry. Serving stale content while revalidation runs can preserve availability when bounded staleness is acceptable.
11. Partitioning for ownership and skew
Hashing by userId is a useful starting point because timelines have a natural owner, but it is not a complete scaling strategy. The largest users create hot graph partitions, broker imbalance, oversized work units, and cache keys that cannot migrate cheaply.
partition = hash(ownerId) % N keeps one reader's ordered mutations together.
Page huge adjacency lists into stable chunks that can be scheduled, retried, and checkpointed independently.
Preserve the original author event, then repartition chunk jobs by destination ownership to spread writes.
Use virtual shards or indirection so ownership moves gradually; measure migration cost while both read and write paths are active.
PostCreated(postId, authorId)
→ enumerate follower pages
→ FanoutChunk(postId, pageCursor, destinationShard)
→ parallel workers
→ conditional timeline inserts
→ chunk completion metric / retry12. What fails at 3 AM?
Duplicate delivery
The worker performs a conditional insert on (timelineOwnerId, postId). A duplicate is a measured no operation, then the message is acknowledged.
Partial fan out
Retry the stable chunk or resume from a checkpoint. The database uniqueness rule protects owners that were already updated.
Derived store outage
Serve a stale snapshot when safe, reconstruct a smaller chronological feed from recent author streams, disable recommendations, and protect durable storage with strict concurrency.
Optional stage misses budget
Cancel personalized ranking at its deadline and return deterministic chronological candidates. Track fallback rate as an SLO symptom.
Amplification shock
Route it to fan out on read, cache the author stream and post object, coalesce hydration, and protect recommendation capacity from the same hot item.
Stale timeline reference
Hydration sees a tombstone or missing post and omits it. Async cleanup removes the dangling references later.
13. Deletes and edits are distributed mutations
A delete request should mark the authoritative post deleted, write PostDeleted to the outbox in the same transaction, and acknowledge. Cleanup workers remove timeline references asynchronously. Until they finish, the read path hydrates every reference against the post store or post cache and omits tombstoned content.
Delete request
→ Post Service marks post deleted + writes outbox
→ PostDeleted event
→ post-cache invalidation + timeline cleanup
Read before cleanup completes
→ timeline reference exists
→ hydration returns tombstone / missing
→ Feed API omits item safelyTimeline entries normally hold post IDs rather than full post bodies. That keeps edits local to the authoritative post record and its cache. Denormalizing small immutable display fields can reduce hydration work, but every copied field acquires an invalidation and repair obligation.
14. Architecture across multiple regions: local reads, explicit convergence
Feed latency rewards regional serving. Strong global consistency for every timeline mutation would add coordination to the hottest path and reduce availability during network partitions. The practical design usually accepts explicit eventual consistency for derived feeds while keeping post ownership and conflict rules precise.
edge routingNearest healthy regionRoute reads geographically; preserve session affinity only when required.regional cacheLocal timeline + post cacheAvoid calls between regions on normal feed reads.durable stateReplicated post storeDefine a write home, quorum, or conflict model. Do not imply that writes to multiple masters work by magic.propagationAsync event replicationMeasure event age between regions and rebuild projections after failover.Active in every region improves regional availability and latency but demands conflict ownership, event deduplication across regions, and more expensive operations. Primary and standby simplifies writes and recovery reasoning but accepts failover time and potentially higher latency outside the primary region. The correct choice follows recovery objectives, regulatory boundaries, team maturity, and cost, not architecture fashion.
15. Observability should tell us whether feeds are converging
CPU and request counts are necessary but insufficient. Operators need to see the age and completeness of derived state, where the latency budget is spent, and whether skew is concentrating work.
feed_latencyRead p50/p95/p99 by region, client, and fallback modepost_ack_latencyPost + outbox commit latency and error ratefeed_cache_hitHit ratio, coalesced loads, stale responses, hot keysconsumer_lagLag and oldest event age by partition and consumer grouppartition_skewIngress and processing imbalance across broker partitionsfanout_completionp50/p95/p99 time from author post to follower visibilityfanout_throughputFollowers processed/sec, retries, failed writes, chunk agedlq_volumeCount, age, event type, and owned investigation statusranking_latencyFeature, model, policy, and serialization stage budgetsranking_fallbackChronological fallback rate and affected requeststimeline_storageMemory, throttling, compaction, evictions, and hot partitionsnetwork_bytesAmplification across zones and regions per logical post
Propagate trace context and a stable correlation ID across Post API → Outbox → Broker → Fan Out Chunk → Timeline Store. A single trace will not sample every follower mutation for a celebrity post, so combine representative spans with aggregate chunk metrics and searchable event IDs. Author or post identifiers with high cardinality belong in controlled logs and traces, not unbounded metric labels.
16. Cost is an architectural constraint
The dominant bill is rarely the post table. It is fan out writes, replicated cache memory, durable timeline storage, broker throughput, ranking computation, and network traffic between regions. Hybrid fan out is therefore a cost control as much as a latency optimization: it avoids spending millions of writes on followers who may never request the content.
Do not materialize for inactive followers indefinitely; define the reactivation/backfill behavior.
Keep the useful recent window, measure bytes per active user, and use admission policies.
Use staged ranking so expensive inference runs on hundreds of candidates, not every available post.
Replicate compact events and immutable media references; avoid chatty hydration between regions.
Overengineering for a hypothetical workload with a billion users can be more damaging than a design that can evolve. It increases deployment risk, operational support burden, cloud spend, and the number of consistency models the team must understand before product market fit.
17. Architecture evolution by measured pressure
stage 1Early product
Postgres owns posts and follows. Redis caches recent timelines. A background worker performs simple chronological fan out. One region, clear tables, excellent instrumentation.
Scale trigger: DB query and worker latency approach SLO limits.stage 2Growth
Add a durable broker, transactional outbox, dedicated Feed API, materialized timelines, chunked workers, cursor pagination, and storage that understands partition ownership.
Scale trigger: skew and ranking needs dominate average throughput.stage 3Large scale
Adopt hybrid fan out, distributed timeline stores, staged ranking, routing for hot users, projections across multiple regions, replay tooling, and operational cost controls.
Scale trigger: business demand, not a diagram copied from a larger company.18. Final architecture review
Before approving this design, I would ask for evidence that the system remains correct and useful when the happy path stops being representative:
- Requirements, read/write peaks, freshness windows, and pagination semantics are explicit.
- Post storage and timeline materialization have separate ownership and recovery paths.
- Post + outbox commit is atomic; publication is retryable and observable.
- Timeline writes are idempotent at
(ownerId, postId). - Fan out work is chunked, checkpointable, and safe to replay.
- Celebrity classification reacts to cost, activity, lag, and skew.
- Broker partitioning preserves required ordering without creating hot partitions.
- Timelines are bounded and page by stable cursors rather than offsets.
- Ranking has stage budgets, policy enforcement, and a chronological fallback.
- Cache miss, stampede, hot key, and stale read behavior is documented.
- Deletes are safe before asynchronous projection cleanup completes.
- Regional consistency, failover, replay, and recovery objectives are testable.
- Metrics expose propagation age, partition skew, retries, DLQ ownership, and fallback rates.
- Cost per feed read and cost per published post can be measured by author class.
A production feed is a set of controlled inconsistencies: the durable post can exist before every timeline references it; a delete can commit before every projection is clean; regional candidates can converge asynchronously. That is acceptable only when each boundary has an SLO, a safe read behavior, an observable backlog, and a recovery procedure.
A queue does not eliminate load. A cache does not eliminate consistency. A retry does not create safety. The architecture works when load can be delayed without becoming unbounded, stale state can be identified without leaking deleted content, and every repeated operation converges on the same business effect.