Spark Deploy Modes: Local vs Client vs Cluster, and Where Spark Connect Fits

The short version

Local mode for a laptop, unit tests and CI. Client mode for shells, notebooks and anything where you read output in your console. Cluster mode for scheduled production jobs, where the machine that submitted the job may go away. The application code is identical in all three; only the launch settings change.

Three roles, one question

Every Spark application has a driver (holds the SparkSession, plans the job, schedules tasks, receives results such as collect()), executors (run the tasks, hold cached data, and must reach the driver for the whole run) and a cluster manager (Standalone, YARN or Kubernetes, named by --master). The deploy mode answers one question: where does the driver run? To choose, ask what happens if the submitting machine goes away.

Local mode: everything in one JVM

spark-submit --master "local[*]" etl_job.py

# or in code
spark = SparkSession.builder.master("local[4]").getOrCreate()

The driver and the executor run in a single JVM on one machine, and tasks run on threads: local is one thread, local[4] four, local[*] one per logical core. There is no cluster manager and nothing to provision, which makes it right for development, unit tests and small samples. The ceiling is one machine's memory and cores. Strictly, local is a --master value rather than a --deploy-mode value, but the exam guide lists it alongside client and cluster.

Client mode: the default, the driver stays with you

spark-submit --master yarn --deploy-mode client etl_job.py

The driver runs inside the spark-submit process on the submitting machine, and the executors run on the cluster. Leave out --deploy-mode and this is what you get.

The catch: the submitting machine must stay up and connected for the whole run. If it sleeps or loses the network, the driver dies and the application with it. Submit from a gateway machine close to the workers, not across a VPN.

Cluster mode: the driver moves in

spark-submit --master yarn --deploy-mode cluster etl_job.py

The cluster manager launches the driver on a node inside the cluster (a YARN container, a Kubernetes driver pod), next to its executors. The submitter can disconnect once the job is accepted, and the driver talks to executors over the cluster's own network instead of across a VPN. The cost is visibility: there is no console, so output and driver logs live in the Spark UI and the cluster manager's logs.

Side by side

Sourced from the Submitting Applications and Cluster Mode Overview pages of the Apache Spark documentation.

Shells, notebooks and Python on Standalone

The pyspark shell and notebooks need their driver attached to the console, so they run in client mode, and cluster deploy mode is not available to shells at all. On a gateway node the driver sits on the gateway while the heavy work still runs on the cluster's executors.

One restriction the exam likes: on a Standalone cluster, Python applications cannot use cluster mode. The documented options are client mode from a gateway, or YARN or Kubernetes when the driver must live inside the cluster.

How Spark Connect changes the picture

Deploy modes describe classic Spark, where your application is the driver. Spark Connect splits them: a thin client builds an unresolved logical plan and sends it over gRPC to a Spark Connect server, which analyzes and runs it. The driver lives with the server, on the cluster side, while the client can be a laptop IDE.

from pyspark.sql import SparkSession

spark = SparkSession.builder.remote("sc://host:15002").getOrCreate()

The pyspark-client package is pure Python with no JVM, a client crash takes down only its own process, and --deploy-mode, a classic spark-submit setting, does not apply. The trade: Connect clients have no RDD API and no SparkContext, so code that needs them stays in classic mode. Spark 4.0 adds spark.api.mode (classic or connect) to switch an application between the two.

Decision signals for the exam

Deployment modes and Spark Connect together form "Using Spark Connect to Deploy Applications", 5% of the Databricks Certified Associate Developer for Apache Spark exam in the official guide. These signals cover it:

• Interactive shell, results in the console: client, from a gateway near the workers.
• Scheduled job, submitter may go away: cluster.
• Python job on a Standalone cluster: client from a gateway, or move to YARN or Kubernetes for cluster mode.
• IDE on a laptop with no Java, shared remote cluster: Spark Connect, not a deploy mode.

FAQ