repartition vs coalesce in PySpark: Shuffle, Direction and the coalesce(1) Trap

The short version

The partition count is the parallelism of the next stage: one task per partition, one core per task. repartition(n) pays for a shuffle and can go up or down with even partitions. coalesce(n) skips the shuffle by gluing neighbors together: the cheap way down, and no way up.

This sits in Troubleshooting and Tuning, 10% of the Databricks Certified Associate Developer for Apache Spark exam per the official 30 Oct 2025 exam guide. Questions describe a symptom (idle cores, tiny files) and ask which call fixes it.

repartition: a full shuffle, any count

df.repartition(64)              # exactly 64, rows dealt out evenly
df.repartition(64, "country")   # 64, hashed on country
df.repartition("country")       # hashed; count from spark.sql.shuffle.partitions

With a number only, every row is redistributed so the new partitions come out roughly equal. With columns, rows are hash partitioned so every row with the same key lands in the same partition; with columns and no number, the count is spark.sql.shuffle.partitions, 200 by default. Either way it is a wide transformation: every row is written to shuffle files and read back over the network, and a new stage starts.

coalesce: merge what is already there

df.coalesce(100)   # 1000 partitions in, 100 out, no shuffle

Each of the 100 new partitions claims about 10 of the existing ones. That is a narrow dependency, so nothing crosses the network. Two rules follow from it:

• Down only. df.coalesce(128) on an 8 partition DataFrame leaves it at 8, silently. No error, no change.
• Sizes are inherited. Merged neighbors keep whatever imbalance they had. Coalesce cannot fix uneven partitions; only a shuffle can.

The coalesce(1) trap: it shrinks the upstream work too

Because coalesce adds no stage boundary, the work before it runs in the same stage, in the reduced number of tasks. A heavy parse or join followed by coalesce(1) can end up running entirely on one core. The PySpark API reference says this directly: a drastic coalesce may run your computation on fewer nodes than you like, and calling repartition() instead adds a shuffle step but keeps the upstream work parallel.

So when one output file is genuinely required after heavy work, repartition(1) is the safer call.

repartition by column before a partitionBy write

write.partitionBy("country") lays the output out as one folder per value. If rows for every country are scattered across every partition, every task writes a file into every folder, and the small files multiply. Grouping the rows first means each task writes into one folder:

(df.repartition("country")
   .write.mode("overwrite")
   .partitionBy("country")
   .parquet("/data/output"))

Coalesce cannot do this, because it takes a number only and never regroups rows by key.

Side by side

Sourced from the PySpark API reference for DataFrame.repartition and DataFrame.coalesce and the Apache Spark SQL performance tuning guide.

Decision signals

• Many small partitions, about to write: coalesce(n), as long as the upstream work is light.
• Fewer partitions than cores (for example, a handful of unsplittable gzip files): repartition(n). Coalesce cannot raise the count.
• Partitions of wildly different sizes: repartition(n), since only a shuffle evens them out.
• Rows should be grouped by a key: repartition("key"), typically before partitionBy("key").
• One output file after heavy work: repartition(1), not coalesce(1).

To confirm which one ran, read explain(): an Exchange is a shuffle, a Coalesce node is not. In SQL, the /*+ REPARTITION(c) */ and /*+ COALESCE(n) */ hints do the same.

What this means for the exam

The common distractor for raising parallelism is changing spark.sql.shuffle.partitions. That setting only applies to a shuffle's output, so it does nothing for a stage that reads files and never shuffles. Where partition counts come from (128 MB input splits, 200 shuffle partitions) is in Chapter 07, partitions and shuffles; join strategies are in the companion post on broadcast hash joins versus sort-merge joins.

To check where you stand across all seven sections, take the readiness quiz or read the section weights post.

FAQ