This would test a different behavior, chiefly use built-in Spark SQL expressions where possible instead of a Python UDF. Native expressions benefit from Spark optimization and avoid expensive Python boundary serialization for simple operations. The specific evidence needed here is that identical fixtures produce matching normalized outputs when the shared function runs in both jobs.

This would test a different behavior, chiefly make the transformation deterministic and side-effect-free instead of depending on Python global state. Spark task retries and parallel execution do not preserve a single shared mutable Python process state. The specific evidence needed here is that identical fixtures produce matching normalized outputs when the shared function runs in both jobs.

This would test a different behavior, chiefly use assertSchemaEqual or equivalent schema assertions for transformation output. A schema-specific test detects incompatible types and field structure that row-count checks cannot detect. The specific evidence needed here is that identical fixtures produce matching normalized outputs when the shared function runs in both jobs.

The checkpoint determines previously discovered input files and prevents unnecessary reprocessing on recovery. That option implements preserve the intended Auto Loader checkpoint and schema state across permitted restarts. It addresses a different problem; here the distinguishing need is to configure the Auto Loader schema evolution strategy appropriate to the source and production contract.

Auto Loader tracks discovered files and incrementally processes newly arrived objects without costly full-directory enumeration. That option implements configure Auto Loader with an appropriate cloud-file source and durable schema/checkpoint location. It addresses a different problem; here the distinguishing need is to configure the Auto Loader schema evolution strategy appropriate to the source and production contract.

A rescued data column or quarantine path preserves unreadable fields instead of silently dropping evidence. That option implements retain rescued or quarantined source records with ingestion provenance for targeted remediation. It addresses a different problem; here the distinguishing need is to configure the Auto Loader schema evolution strategy appropriate to the source and production contract.

This would test a different behavior, chiefly audit Unity Catalog privilege inheritance and move broad grants to the intended narrow scope. Privileges at higher securable levels can affect descendant objects and must match the access design. The specific evidence needed here is that the governance group can manage the schema and the departed identity no longer retains owner rights.

This would test a different behavior, chiefly grant only the necessary CREATE or USE privileges on the intended securable hierarchy. Read privileges do not imply object creation; Unity Catalog uses scoped privileges for different operations. The specific evidence needed here is that the governance group can manage the schema and the departed identity no longer retains owner rights.

This would test a different behavior, chiefly separate ownership and policy administration through appropriate Unity Catalog roles and policy scopes. Governance separation of duties prevents ordinary data stewards from bypassing central data protection. The specific evidence needed here is that the governance group can manage the schema and the departed identity no longer retains owner rights.

Shared validated entity rules prevent duplicated conflicting cleansing decisions in separate marts. That option implements build a shared conformed silver entity model before downstream domain-specific gold datasets. It addresses a different problem; here the distinguishing need is to model the dimension with historical validity intervals and join facts at the correct event time.

Medallion layers preserve source evidence while progressively applying quality, business rules and publication contracts. That option implements separate bronze ingestion, validated silver transformations, and governed gold reporting tables. It addresses a different problem; here the distinguishing need is to model the dimension with historical validity intervals and join facts at the correct event time.

Separating evolving raw input from curated contracts protects consumers from frequent source changes. That option implements retain flexible raw attributes in bronze and publish a validated typed schema in silver or gold. It addresses a different problem; here the distinguishing need is to model the dimension with historical validity intervals and join facts at the correct event time.

Compute sizing should follow measured spill and shuffle behavior rather than blind worker-count increases. That option implements profile memory pressure and select a better memory/worker configuration alongside join optimization. It addresses a different problem; here the distinguishing need is to design idempotency protection for external side effects and explicitly configure safe retry behavior.

Ephemeral job compute avoids charging for long idle interactive sessions when runs are infrequent. That option implements use appropriately sized job or serverless compute with automated lifecycle for the scheduled workload. It addresses a different problem; here the distinguishing need is to design idempotency protection for external side effects and explicitly configure safe retry behavior.

Compute and configuration choices should match whether runtime is dominated by CPU, memory, I/O or skew. That option implements use workload profiling to identify the bottleneck before adjusting compute size or execution mode. It addresses a different problem; here the distinguishing need is to design idempotency protection for external side effects and explicitly configure safe retry behavior.

This would test a different behavior, chiefly configure event-time watermarks and bounded join-state conditions for the business lateness tolerance. Watermarks allow Spark to evict expired streaming state without discarding permissible late events. The specific evidence needed here is that peak arrival tests show the backlog draining and processing rate staying above ingress rate.

This would test a different behavior, chiefly apply appropriate streaming write settings and compaction or managed optimization to control file size. Large numbers of small files add metadata and I/O overhead to downstream scans. The specific evidence needed here is that peak arrival tests show the backlog draining and processing rate staying above ingress rate.

This would test a different behavior, chiefly use supported incremental stateful aggregation with checkpointed progress instead of repeated full historical reads. Incremental processing updates state for new events rather than repeatedly transforming all prior source data. The specific evidence needed here is that peak arrival tests show the backlog draining and processing rate staying above ingress rate.

Operator-level profiling distinguishes inefficient data access from downstream aggregation work. That option implements inspect the SQL query profile for scan volumes, shuffles, joins and spill information. It addresses a different problem; here the distinguishing need is to query the supported Jobs REST API or SDK for run and task state.

Task-level run metadata pinpoints failed attempts and environmental errors hidden by a summary status. That option implements inspect task-run details and corresponding compute logs through Jobs UI or supported API. It addresses a different problem; here the distinguishing need is to query the supported Jobs REST API or SDK for run and task state.

Execution profiling reveals the costly operators and task distribution that are not evident from business output rows. That option implements open Spark UI and query profile to examine stages, shuffle and task skew. It addresses a different problem; here the distinguishing need is to query the supported Jobs REST API or SDK for run and task state.

Repair runs can preserve eligible successful upstream work while rerunning selected failed tasks. That option implements use Lakeflow Jobs repair to retry failed downstream tasks and review the relevant failure details. It addresses a different problem; here the distinguishing need is to inspect task-value references and If/Else condition inputs in the Lakeflow Jobs run details.

Interactive and automated runs can differ in principal and dependency context even with identical code. That option implements compare execution identity, secret permissions and job-task environment for the failing run. It addresses a different problem; here the distinguishing need is to inspect task-value references and If/Else condition inputs in the Lakeflow Jobs run details.

Catalog access failures depend on the run-as principal and required hierarchy privileges, not notebook ownership alone. That option implements inspect the failed task's execution identity and effective Unity Catalog privileges. It addresses a different problem; here the distinguishing need is to inspect task-value references and If/Else condition inputs in the Lakeflow Jobs run details.

Clustering eligibility differs by table type and compute runtime; unsupported combinations must be upgraded or redesigned. That option implements validate table format and runtime support before enabling automatic liquid clustering. It addresses a different problem; here the distinguishing need is to choose one supported clustering layout strategy and do not combine ZORDER with liquid clustering.

Co-locating values of selective columns improves file-level statistics and data skipping. That option implements cluster by the frequently filtered columns and run an appropriate optimize cycle. It addresses a different problem; here the distinguishing need is to choose one supported clustering layout strategy and do not combine ZORDER with liquid clustering.

Changing clustering metadata alone does not rewrite existing files into the new layout. That option implements run an appropriate OPTIMIZE operation after modifying the liquid-clustering definition. It addresses a different problem; here the distinguishing need is to choose one supported clustering layout strategy and do not combine ZORDER with liquid clustering.

Business-key deduplication makes repeated deliveries harmless and preserves distinct legitimate events. That option implements deduplicate with a stable event or transaction key before appending to trusted Delta tables. It addresses a different problem; here the distinguishing need is to use a Delta MERGE with stable matching keys and explicit matched/update and not-matched/insert logic.

Delete markers must map to target removals instead of being inserted as ordinary rows. That option implements handle matched deletes in a keyed MERGE or CDC processing step according to the source contract. It addresses a different problem; here the distinguishing need is to use a Delta MERGE with stable matching keys and explicit matched/update and not-matched/insert logic.

Arrival time can differ from source transaction order, so state reconciliation needs source ordering. That option implements choose the target record using source sequence ordering rather than ingestion arrival order. It addresses a different problem; here the distinguishing need is to use a Delta MERGE with stable matching keys and explicit matched/update and not-matched/insert logic.

Message-bus at-least-once delivery can repeat events and requires explicit business-key deduplication. That option implements apply event-key-based deduplication and checkpointed ingestion before publishing trusted target records. It addresses a different problem; here the distinguishing need is to use a supported Structured Streaming message-bus connector with checkpoints and Delta writes.

Schema registration and explicit decoding prevent interpreting changed binary event layouts as arbitrary bytes. That option implements use the appropriate schema-aware deserialization and controlled schema compatibility validation. It addresses a different problem; here the distinguishing need is to use a supported Structured Streaming message-bus connector with checkpoints and Delta writes.

Keeping raw events provides replay and audit capability independent of the mutable downstream representation. That option implements append raw events to bronze Delta first, then apply keyed changes in a separate silver processing step. It addresses a different problem; here the distinguishing need is to use a supported Structured Streaming message-bus connector with checkpoints and Delta writes.