An accidental Cartesian product can multiply intermediate rows and dominate cost. That option implements inspect the physical plan and query profile to find a missing or incorrect join predicate. It addresses a different problem; here the distinguishing need is to inspect Spark UI memory, shuffle spill and executor logs to choose a join or memory fix.

Disproportionate task durations and shuffle bytes reveal skew rather than general CPU scarcity. That option implements use Spark UI task timelines and shuffle metrics to isolate skewed partitions before applying a targeted fix. It addresses a different problem; here the distinguishing need is to inspect Spark UI memory, shuffle spill and executor logs to choose a join or memory fix.

Task-level diagnostics provide specific executor exceptions and data locations hidden by the top-level job status. That option implements inspect executor logs and Spark UI failed task details to isolate the offending partition and data. It addresses a different problem; here the distinguishing need is to inspect Spark UI memory, shuffle spill and executor logs to choose a join or memory fix.

This would test a different behavior, chiefly use a supported serverless or compatible runtime/access configuration for ABAC policy enforcement. Fine-grained ABAC restrictions have compute prerequisites and cannot be assumed to work on all older runtimes. The specific evidence needed here is that test principals see only their permitted regional rows and centrally administered policy cannot be bypassed by table owners.

This would test a different behavior, chiefly create Unity Catalog ABAC masking policies scoped to governed tags at the appropriate catalog or schema. Tag-conditioned ABAC policies provide centralized reusable protection as new matching columns appear. The specific evidence needed here is that test principals see only their permitted regional rows and centrally administered policy cannot be bypassed by table owners.

This would test a different behavior, chiefly define governed tags with limited assignment permissions before relying on tag-scoped ABAC. ABAC policy selection depends on governed tags that are controlled independently of ordinary write access. The specific evidence needed here is that test principals see only their permitted regional rows and centrally administered policy cannot be bypassed by table owners.

This would test a different behavior, chiefly run or enable the supported statistics collection and optimization process for the managed table. File statistics and layout information help the engine eliminate irrelevant data during queries. The specific evidence needed here is that freshness is met by pipeline updates and optimization history is assessed separately for scan performance.

This would test a different behavior, chiefly use an evidence-based maintenance cadence or platform-managed optimization instead of constant manual full rewrites. Excessive maintenance can contend with ingestion and cost more than the resulting scan benefit. The specific evidence needed here is that freshness is met by pipeline updates and optimization history is assessed separately for scan performance.

This would test a different behavior, chiefly enable predictive optimization for eligible Unity Catalog managed tables where supported. Predictive optimization manages operations such as optimization and cleanup using observed table needs. The specific evidence needed here is that freshness is met by pipeline updates and optimization history is assessed separately for scan performance.

This would test a different behavior, chiefly use assertSchemaEqual or equivalent schema assertions for transformation output. A schema-specific test detects incompatible types and field structure that row-count checks cannot detect. The specific evidence needed here is that a test deliberately injecting duplicate rows fails while the reference transformation passes.

This would test a different behavior, chiefly use built-in Spark SQL expressions where possible instead of a Python UDF. Native expressions benefit from Spark optimization and avoid expensive Python boundary serialization for simple operations. The specific evidence needed here is that a test deliberately injecting duplicate rows fails while the reference transformation passes.

This would test a different behavior, chiefly implement a pure DataFrame-to-DataFrame function and apply it with DataFrame.transform. Pure DataFrame transformations are reusable and can be tested independently of notebook state. The specific evidence needed here is that a test deliberately injecting duplicate rows fails while the reference transformation passes.

An accidental Cartesian product can multiply intermediate rows and dominate cost. That option implements inspect the physical plan and query profile to find a missing or incorrect join predicate. It addresses a different problem; here the distinguishing need is to use query profile scan metrics to identify predicate pushdown or data-skipping regressions.

Disproportionate task durations and shuffle bytes reveal skew rather than general CPU scarcity. That option implements use Spark UI task timelines and shuffle metrics to isolate skewed partitions before applying a targeted fix. It addresses a different problem; here the distinguishing need is to use query profile scan metrics to identify predicate pushdown or data-skipping regressions.

Execution metrics identify whether memory pressure comes from wide shuffles, skew or configuration. That option implements inspect Spark UI memory, shuffle spill and executor logs to choose a join or memory fix. It addresses a different problem; here the distinguishing need is to use query profile scan metrics to identify predicate pushdown or data-skipping regressions.

Scan-level evidence distinguishes inefficient reads from subsequent aggregation performance. That option implements use query profile scan metrics to identify predicate pushdown or data-skipping regressions. It addresses a different problem; here the distinguishing need is to inspect the physical plan and query profile to find a missing or incorrect join predicate.

Disproportionate task durations and shuffle bytes reveal skew rather than general CPU scarcity. That option implements use Spark UI task timelines and shuffle metrics to isolate skewed partitions before applying a targeted fix. It addresses a different problem; here the distinguishing need is to inspect the physical plan and query profile to find a missing or incorrect join predicate.

Task-level diagnostics provide specific executor exceptions and data locations hidden by the top-level job status. That option implements inspect executor logs and Spark UI failed task details to isolate the offending partition and data. It addresses a different problem; here the distinguishing need is to inspect the physical plan and query profile to find a missing or incorrect join predicate.

This would test a different behavior, chiefly use AUTO CDC configured for SCD Type 2 with the appropriate sequence column. SCD Type 2 preserves prior versions with validity periods while maintaining the current version. The specific evidence needed here is that a delete event removes or closes the correct key and unrelated keys remain unchanged.

This would test a different behavior, chiefly use AUTO CDC with stable keys, sequence ordering, and delete handling for SCD Type 1. AUTO CDC applies CDC events to target keys while sequencing resolves update order and delete semantics. The specific evidence needed here is that a delete event removes or closes the correct key and unrelated keys remain unchanged.

This would test a different behavior, chiefly use AUTO CDC FROM SNAPSHOT with deterministic snapshot ordering for the supported pipeline pattern. Snapshot-oriented AUTO CDC compares successive complete versions to derive changes without a transaction log. The specific evidence needed here is that a delete event removes or closes the correct key and unrelated keys remain unchanged.

This would test a different behavior, chiefly choose a supported managed table format or compatibility feature matching the required Delta and Iceberg readers. Table interoperability requires compatible table metadata and readers rather than renaming data files. The specific evidence needed here is that concurrent supported writes yield valid committed snapshots and readers never observe uncommitted partial files.

This would test a different behavior, chiefly define schema and quality handling with rescued data or quarantine for incompatible values. Explicit schema enforcement and exception capture prevent undetected corruption when source types vary. The specific evidence needed here is that concurrent supported writes yield valid committed snapshots and readers never observe uncommitted partial files.

This would test a different behavior, chiefly ingest new Parquet batches into a Delta bronze table using incremental file discovery. An incremental bronze append pattern avoids losing history and reduces unnecessary input rescans. The specific evidence needed here is that concurrent supported writes yield valid committed snapshots and readers never observe uncommitted partial files.

This would test a different behavior, chiefly use a Delta MERGE with stable matching keys and explicit matched/update and not-matched/insert logic. MERGE applies conditional updates and inserts transactionally against identified target rows. The specific evidence needed here is that a fixture with repeated customer updates produces the latest correct value and one target key.

This would test a different behavior, chiefly choose the target record using source sequence ordering rather than ingestion arrival order. Arrival time can differ from source transaction order, so state reconciliation needs source ordering. The specific evidence needed here is that a fixture with repeated customer updates produces the latest correct value and one target key.

This would test a different behavior, chiefly handle matched deletes in a keyed MERGE or CDC processing step according to the source contract. Delete markers must map to target removals instead of being inserted as ordinary rows. The specific evidence needed here is that a fixture with repeated customer updates produces the latest correct value and one target key.

Streaming tables process new source changes incrementally and track pipeline state across updates. That option implements define a Lakeflow streaming table reading an incremental streaming source. It addresses a different problem; here the distinguishing need is to separate controlled backfill processing from the live streaming checkpoint and reconcile outputs by stable keys.

A materialized view represents a query result refreshed from upstream dependencies rather than simply appending stream records. That option implements define a Lakeflow materialized view for the maintained derived query. It addresses a different problem; here the distinguishing need is to separate controlled backfill processing from the live streaming checkpoint and reconcile outputs by stable keys.

Expectations define data quality handling and surface invalid-record metrics rather than silently swallowing failures. That option implements use Lakeflow expectations with explicit drop or fail behavior and capture expectation metrics. It addresses a different problem; here the distinguishing need is to separate controlled backfill processing from the live streaming checkpoint and reconcile outputs by stable keys.

Automatic liquid clustering selects keys from usage patterns and works with predictive optimization. That option implements enable automatic liquid clustering with CLUSTER BY AUTO on an eligible managed table. It addresses a different problem; here the distinguishing need is to validate table format and runtime support before enabling automatic liquid clustering.

Co-locating values of selective columns improves file-level statistics and data skipping. That option implements cluster by the frequently filtered columns and run an appropriate optimize cycle. It addresses a different problem; here the distinguishing need is to validate table format and runtime support before enabling automatic liquid clustering.

Liquid clustering is a replacement layout option rather than an add-on to ZORDER on the same table. That option implements choose one supported clustering layout strategy and do not combine ZORDER with liquid clustering. It addresses a different problem; here the distinguishing need is to validate table format and runtime support before enabling automatic liquid clustering.