Distributed computing

Distributed computing splits a single computational task across multiple independent machines, nodes in a cluster, that communicate over a network and coordinate to produce one combined result. Instead of one very large machine processing a dataset in full, the dataset is partitioned and each node processes its share, in parallel, before results are combined.

This differs from simply running a program on a more powerful single machine, vertical scaling, which eventually hits a physical ceiling on memory and processing power; distributed systems scale horizontally instead, by adding more machines, which in principle has no fixed ceiling but introduces new problems a single machine never has: network latency between nodes, partial failures where some nodes crash while others keep running, and the coordination overhead of keeping results consistent across all of them.

Distributed computing is what makes engines like Apache Spark and the DataFrame abstractions built on them practical for datasets far too large for one machine's memory, and it underlies most modern batch processing at scale. It matters because data volumes routinely exceed what a single machine can hold or process in reasonable time. The common pitfall is distributing a workload that's actually small enough to run on one machine, paying the coordination and network overhead of a cluster for no real benefit, before confirming a single powerful machine can't do the job.