🚀 Moving from beginner to intermediate PySpark is not about memorizing more functions: it is about understanding how data moves inside a distributed job.
A DataFrame is split into partitions that can be processed in parallel. Too few partitions waste CPU; too many create small tasks and coordination overhead. repartition() redistributes data, while coalesce() reduces partitions with less movement, which is useful for avoiding too many files when writing Parquet.
The central concept is the shuffle: redistributing data between partitions. Operations such as groupBy(), join(), distinct(), orderBy(), and repartition() can trigger one and are usually expensive.
To improve performance, filter and select columns before a join, keep keys in the same data type, and check whether the entire dataset really needs to move. Cache only DataFrames that are reused, and inspect the execution plan before changing settings randomly.
The author offers a practical idea: when a job becomes slow, first look for where information is traveling. The cause is often an unnecessary shuffle, an oversized join, or poorly sized partitions.
💡 Explanation in a nutshell#
Imagine Spark distributing boxes among several workers. If each worker has the right boxes, they work in parallel. If they must exchange boxes to group or sort them, time is lost. Those exchanges are shuffles: sometimes unavoidable, but worth reducing.
More information at the link 👇
