Auto Loader Schema Inference and Evolution: Modes, Hints and the Rescued Data Column

What this deep dive adds

Chapter 7 of the course covers what Auto Loader is: the cloudFiles source, file discovery, the checkpoint and the idea of schema evolution. This post is the reference behind that card: a full pipeline, every option it uses, the five evolution modes side by side, and what lands in _rescued_data.

Auto Loader sits in Data Ingestion and Loading, the second of the seven sections in the May 2026 version of the Databricks Certified Data Engineer Associate exam. Questions usually describe a pipeline that broke, or did not break, when the files changed shape.

The full pipeline, line by line

checkpoint = "/Volumes/main/bronze/checkpoints/events"

(spark.readStream
    .format("cloudFiles")
    .option("cloudFiles.format", "json")
    .option("cloudFiles.schemaLocation", checkpoint)
    .option("cloudFiles.inferColumnTypes", "true")
    .option("cloudFiles.schemaHints", "event_ts timestamp, amount decimal(10,2)")
    .load("/Volumes/main/landing/events/")
  .writeStream
    .option("checkpointLocation", checkpoint)
    .option("mergeSchema", "true")
    .trigger(availableNow=True)
    .toTable("main.bronze.events"))

Step 1: how the schema is inferred

On the first run Auto Loader samples up to 50 GB or 1,000 files, whichever limit it reaches first, and writes the result to a _schemas directory under the schema location. Later runs read the stored schema instead of sampling again, which is why the schema location is required.

• JSON, CSV and XML infer every column as a string, nested JSON fields included. Strings never fail to parse, so nothing is lost when a value changes type between files. Set cloudFiles.inferColumnTypes to true to get real types instead.
• Parquet and Avro carry their own types, and Auto Loader uses them. When Parquet files disagree, it picks the widest type.
• CSV headers are assumed. For headerless files set .option("header", "false").
• Case: abc, Abc and ABC are treated as one column during inference.

Step 2: schema hints for the columns you know

.option("cloudFiles.schemaHints", "tags map<string,string>, version int")

A hint overrides the inferred type of one column and leaves the rest to inference. Hints can also name a column that has not appeared yet, so it exists with the right type from the first batch. They apply only when you have not supplied a full schema with .schema(...), and they work with inferColumnTypes on or off.

Step 3: a new column arrives

This is the part the exam tests. cloudFiles.schemaEvolutionMode takes one of five values:

Sourced from the Databricks documentation page "Configure schema inference and evolution in Auto Loader".

Why the default stops the stream on purpose

With addNewColumns, Auto Loader runs inference on the latest micro-batch, appends the new columns to the stored schema and then fails the stream. A running stream cannot change its schema in place, so the failure is how the new version takes effect. Databricks recommends running the stream as a Lakeflow Job, whose retry restarts it without anyone stepping in. A question that says "the stream failed after the files gained a field, and succeeded on retry" is describing exactly this.

Supplying your own schema

(spark.readStream
    .format("cloudFiles")
    .option("cloudFiles.format", "json")
    .schema("id bigint, event_ts timestamp, amount decimal(10,2)")
    .option("rescuedDataColumn", "_rescued_data")
    .load("/Volumes/main/landing/events/"))

With .schema(...) the default mode becomes none, and addNewColumns is not allowed at all. The line above adds the rescued data column explicitly, so unexpected fields are kept rather than silently dropped. If you want evolution with known types, use schema hints instead of a full schema.

The rescued data column

When inference is on, Auto Loader adds a _rescued_data column. Instead of failing or dropping a record that does not fit the schema, it stores the parts that did not fit as a JSON string, together with the path of the file they came from. Three things end up there:

• a field that is not in the schema (in rescue mode, or in none mode with the column set);
• a value with the wrong type, such as "twelve" in an int column;
• a field whose case differs from the schema, such as Amount against amount.

SELECT _rescued_data, count(*) AS records
FROM main.bronze.events
WHERE _rescued_data IS NOT NULL
GROUP BY _rescued_data;

A query like this in Silver is the usual way to notice drift without stopping Bronze. The column also changes how parser modes behave: with it set, a type mismatch no longer drops a record in DROPMALFORMED or fails the read in FAILFAST. Only records that are actually corrupt do.

Two details that are easy to miss

• Partition columns do not evolve the schema. Auto Loader reads Hive-style paths such as event=click/date=2026-10-01/, but a new directory key is not picked up automatically. List the keys in cloudFiles.partitionColumns, for example "event,date".
• Pipelines manage the location for you. In Lakeflow Spark Declarative Pipelines, Databricks manages the schema location and checkpoint, so you never set cloudFiles.schemaLocation yourself.

What this means for the exam

A common distractor puts schema handling on the checkpoint or on file notification mode. The checkpoint tracks which files were processed, and file notification only changes how new files are found. Neither touches the schema. When to choose Auto Loader over COPY INTO at all is covered in the COPY INTO vs Auto Loader post and in Chapter 8 of the course.

FAQ