Latency and throughput are the two ways we measure performance, and they are not the same thing. Latency is how long one request takes. Throughput is how many requests the system completes per unit of time. Interviewers watch closely for whether you keep these separate, because optimizing one can hurt the other.
Think of a highway. Latency is how long it takes one car to drive from A to B. Throughput is how many cars pass a point per hour. Adding lanes raises throughput without making any single car faster, and raising the speed limit lowers latency without adding capacity. A system can have high throughput and bad latency at the same time, and vice versa.
Averages hide pain. Report p50 (median), p95, p99, and p99.9. The tail matters more than it looks because a single user-facing page often fans out to many backend calls, and the slowest one determines the whole page's latency. If each of 100 calls has a 1% chance of being slow, most page loads hit at least one slow call. This is why big companies obsess over p99 latency, not the average.
These order-of-magnitude numbers let you reason about designs on the spot.
The key takeaway: memory is roughly a million times faster than a cross-continent network round trip, which is why caching and putting data near users matter so much.
A simple, powerful relationship: concurrency = throughput times latency (L equals lambda times W). If each request takes 100 milliseconds and you want 1,000 requests per second, you need about 100 requests in flight at once. The insight for interviews: at a fixed concurrency limit, the only way to raise throughput is to lower latency, and vice versa. It also tells you how many connections or threads to provision.
Batching and buffering raise throughput but add latency, because requests wait to be grouped. And there is a hard rule from queueing theory: as utilization approaches 100 percent, latency grows toward infinity, because there is no slack to absorb bursts. This is why you never run a system at full utilization and why unbounded queues are dangerous: they hide the problem by letting latency balloon instead of rejecting load.
"My latency target is p99 under 200 milliseconds, and I expect about 50,000 requests per second at peak. To hold the p99 I will cache aggressively, keep data in-region, and fan out backend calls in parallel. For throughput I will scale the stateless tier horizontally and put a bounded queue with load shedding in front of the slow path, and I will keep utilization well below 100 percent to protect the tail."