Choosing a database, a queue and a container platform is not yet a system design. Start with who reads the data, who writes it, how quickly it must become visible and what should happen when a dependency fails.
One useful distinction is to treat architecture as the constraints and boundaries of a system, and system design as the detailed decisions that satisfy its requirements. The terms overlap in practice. What matters is making the assumptions explicit, not defending a particular vocabulary.
Define the workload before choosing tools
Consider an illustrative telemetry dashboard. The Telemetry project provides the domain context, but the following numbers and architecture are a teaching example, not a description of its deployment.
If 100 devices each send two 200-byte messages per second, the raw payload rate is:
100 devices x 2 messages/second x 200 bytes = 40,000 bytes/second
That is not the complete storage or network requirement. Protocol overhead, indexes, replicas and retention increase the cost. A dashboard requesting a day of data can also be much more expensive than its small number of HTTP requests suggests.
Write down both the normal load and the burst you intend to support. Include acceptable latency, data freshness, retention and a loss policy. Without those bounds, “scalable” has no testable meaning.
Read-heavy systems: cache the right thing
A frequently read object may benefit from a cache, but cache correctness depends on the product. A stale profile photo and a stale payment status have different consequences.
Cache stampedes occur when many requests discover an absent or expired value and all recompute it. Possible protections include combining concurrent requests for the same key, serving a stale value while refreshing, and using leases to control cache fills.
Adding random variation to expiration times helps spread expirations across different keys. It does not, on its own, prevent thousands of requests from recomputing one popular key. The Scaling Memcache at Facebook paper is a useful account of these problems and the use of leases.
For the telemetry example, a precomputed recent-window summary might serve the main dashboard while a separate query handles historical exploration. Measure cache hit rate, origin load and tolerated staleness before claiming an improvement.
Write-heavy systems: queues move the bottleneck
A queue can absorb bursts and allow ingestion to continue while workers catch up. It does not create infinite processing capacity. If the arrival rate stays above the consumption rate, the backlog keeps growing until retention, disk space or latency becomes unacceptable.
Distinguish two acknowledgements:
- Accepted: the event has been durably recorded for later processing.
- Completed: the application has applied the intended state change.
An in-memory handoff is not durable acceptance. Define where persistence occurs and what the caller should retry if acknowledgement is lost. Use bounded buffers, backpressure or a deliberate dropping policy rather than letting an unbounded queue become a delayed outage.
For telemetry, dropping an explicitly designated low-priority sample may be acceptable. For financial records, silently dropping events is not an equivalent trade-off.
Retries require idempotent state changes
A worker can commit a database update and then fail before acknowledging the message. The next delivery is not proof that the first attempt did nothing.
Use a stable event identity, a durable uniqueness constraint and a transaction that couples deduplication with the business change. An application-level “does this exist?” check followed by an insert is vulnerable to concurrent workers.
The guide to idempotent payment notifications walks through the transaction boundary and why outbound effects need their own delivery strategy. It is a narrower, more actionable problem than promising “exactly once” across every component.
Consistency is a requirement, not a slogan
During a network partition, a distributed system cannot guarantee both linearizable consistency and availability for every request under the CAP model. That does not imply that every ordinary database choice is simply “choose two of three,” or that all replica lag is a partition.
Ask what a user must observe after writing. A read replica may be appropriate for a historical chart but unsuitable for immediately confirming a just-completed payment without an explicit read-after-write strategy. Eventual consistency also has no universal “few milliseconds” guarantee.
Sharding adds another set of boundaries: cross-shard queries, transactions, rebalancing and hot keys. First measure queries, indexes, contention and data volume. Do not assume that splitting storage will fix an inefficient access pattern.
Make failure visible and recovery measurable
Timeouts limit how long a caller waits. Bounded retries with backoff and jitter avoid immediate repeated pressure. A circuit breaker can stop calls to a failing dependency, but only if its thresholds and fallback behavior fit the operation.
For the example pipeline, useful signals include event age at consumption, rejected samples, queue depth, processing rate and end-to-end data freshness. Logs and traces help explain failures; metrics and alerts tell you that an operational limit has been crossed.
Infrastructure recovery is a related but separate concern. A Proxmox HA setup can restart a guest without ensuring that an application has processed its backlog or recovered its connections.
A design review checklist
- State the workload, acceptable latency and data-loss tolerance.
- Identify the durable acknowledgement and the authoritative data store.
- Explain how duplicates, stale reads and partial failures are handled.
- Bound resource growth: buffers, retries, retention and connection pools.
- Test a dependency failure and measure recovery at the application boundary.
The best architecture is not the one with the most components. It is the smallest design whose behavior you can explain under the load and failures you actually need to support.
References
- Designing Data-Intensive Applications, Martin Kleppmann.
- Scaling Memcache at Facebook, USENIX NSDI 2013.
- Making retries safe with idempotent APIs, Amazon Builders’ Library.
- Enterprise Integration Patterns, Gregor Hohpe and Bobby Woolf.