A broadcast hash join copies the small side of a join to every executor as an in-memory hash table, so the large side is never shuffled. A sort-merge join shuffles and sorts both sides by the join key, then merges them. Spark broadcasts automatically when a side is estimated under 10 MB; otherwise sort-merge join is the default.
Last updated October 2026.
An equi-join can only match two rows in the same task, so something has to move. Sort-merge join moves both sides; broadcast hash join moves only the small one. Broadcast is often the biggest speed-up in a job, and the fastest way to crash a driver when the "small" table is not small.
On the Databricks Certified Associate Developer for Apache Spark exam this spans the DataFrame API section (30% in the official exam guide) and Troubleshooting and Tuning (10%).
spark.sql.shuffle.partitions partitions (200 by default).Nothing needs to fit in memory whole, so it handles any size and every join type, including full outer. The price is shuffling and sorting both tables; when it struggles, it fails slowly.
The small side is collected onto the driver, sent once to every executor, and built into a hash table. Each task probes it with its own partition of the large side, in place, so a 2 TB fact table never crosses the network.
The cost is memory: the whole small side must fit on the driver and on every executor. When it does not, the job fails hard, with out-of-memory errors or a broadcast timeout (spark.sql.broadcastTimeout, 300 seconds by default).
from pyspark.sql.functions import broadcast
sales.join(broadcast(stores), "store_id")
sales.join(stores.hint("broadcast"), "store_id")
spark.sql("SELECT /*+ BROADCAST(s) */ * FROM sales f JOIN stores s USING (store_id)")
spark.conf.set("spark.sql.autoBroadcastJoinThreshold", -1) # automatic broadcast offspark.sql.autoBroadcastJoinThreshold is the largest estimated size Spark broadcasts on its own: 10485760 bytes (10 MB) by default. -1 disables automatic broadcasting.ANALYZE TABLE ... COMPUTE STATISTICS also fixes that), dangerous when the table keeps growing.Only the side whose unmatched rows need not be kept can be broadcast:
left_semi, left_anti): only the right side.The how strings themselves are covered in Chapter 21 on joins.
Adaptive Query Execution (AQE) is on by default since Spark 3.2. After a shuffle stage, AQE knows each side's real size; if one is under spark.sql.adaptive.autoBroadcastJoinThreshold (by default the same value as the regular threshold), it swaps the sort-merge join for a broadcast hash join. The sorts are saved, but the shuffle already ran, so a broadcast planned up front is usually faster.
| Broadcast hash join | Sort-merge join | |
|---|---|---|
| Large side shuffled | No | Yes, both sides, plus a sort |
| Memory needed | Whole small side on the driver and every executor | One partition at a time; spills |
| Chosen when | Estimate under 10 MB, a hint, or AQE at runtime | The default for two large sides |
| Join types | No full outer; side limited by type | All equi-joins |
| Failure mode | Out of memory or broadcast timeout | Slow: network, sort and spill |
| In explain() | BroadcastHashJoin, BroadcastExchange | SortMergeJoin, Exchange hashpartitioning and Sort on both sides |
Sourced from the Apache Spark SQL performance tuning guide (join strategy hints, AQE) and the PySpark API reference for broadcast.
Call explain() on the joined DataFrame. BroadcastHashJoin ... BuildRight means the right side was broadcast, with a BroadcastExchange on it and no Exchange above the large scan. SortMergeJoin comes with an Exchange hashpartitioning(key, 200) and a Sort under each side. With AQE the root is AdaptiveSparkPlan, and once isFinalPlan=true the final plan can show a broadcast join where the initial plan had a sort-merge join.
Exam tip: "a few MB lookup table" and "avoid shuffling the large table" points at broadcast(). "Both tables are large" or "the broadcast table grew and the driver runs out of memory" points at removing the hint and letting sort-merge join (and AQE) do the work.
Two distractors recur. Raising spark.sql.shuffle.partitions still shuffles every byte of the large side. Raising the threshold to match an oversized table changes nothing when a hint is already there. More in Chapter 31 on AQE and the companion post on repartition versus coalesce; to find weak spots, try the readiness quiz or the practice hub.
A broadcast hash join copies the small side to every executor and joins the large side in place, with no shuffle of it. A sort-merge join shuffles and sorts both sides by the join key, which works for any size and every join type.
The default is 10485760 bytes, which is 10 MB. Spark broadcasts a join side automatically when its estimated size is under that value, and setting it to -1 disables automatic broadcasting.
Yes. A side marked with broadcast() or a BROADCAST hint is broadcast regardless of the threshold, as long as the join type allows that side to be broadcast. That is why a hint on a table that keeps growing can eventually cause out-of-memory errors.
No. A full outer join must keep unmatched rows from both sides, so neither side can be broadcast, even with a hint. Spark plans a sort-merge join instead.
Broadcast joins is a trade-off chapter in Unit 3 of the Spark Developer Associate course, with the plans, the threshold, the hints and exam-style questions on when each strategy is right.
Open the chapter →