To disable Spark shuffles<\/span><\/p>\nAnswer: 1) To distribute heavily repeated keys across multiple partitions<\/b><\/p>\n
Explanation:<\/b><\/p>\n
Salting is a technique used to reduce the impact of data skew during distributed processing. When a particular join key appears extremely frequently, many records can be assigned to the same partition, creating a processing bottleneck. Salting adds an additional value to the skewed key so that records can be distributed across multiple partitions. The corresponding join logic then accounts for the salt values. This approach can improve workload balance when standard partitioning produces severe skew. It is particularly useful when a small number of key values dominate the dataset and cause certain Spark tasks to become disproportionately large.<\/span><\/p>\nQuestion 202. Which Spark operation commonly causes data to move between partitions across the cluster?<\/b><\/h3>\n
1)<\/b> Shuffle<\/span>
\n<\/span>2)<\/b> Cache<\/span>
\n<\/span>3)<\/b> Select<\/span>
\n<\/span>4)<\/b> Rename<\/span><\/p>\nAnswer: 1) Shuffle<\/b><\/p>\n
Explanation:<\/b><\/p>\n
A shuffle occurs when Spark needs to redistribute data across partitions to perform an operation that requires related records to be colocated. Operations such as joins, groupBy aggregations, and certain repartitioning operations can cause shuffles. Because data may need to travel between executors, shuffles can involve significant network and disk activity. They can therefore become an important factor in Spark performance. Engineers often examine execution plans and partitioning strategies to identify unnecessary shuffles. Reducing expensive shuffle operations, when practical, can improve the efficiency of distributed data processing.<\/span><\/p>\nQuestion 203. Which Spark operation is commonly used to inspect the execution plan of a DataFrame?<\/b><\/h3>\n
1)<\/b> explain()<\/span>
\n<\/span>2)<\/b> collectSchema()<\/span>
\n<\/span>3)<\/b> inspectPlanOnly()<\/span>
\n<\/span>4)<\/b> showPlanTable()<\/span><\/p>\nAnswer: 4) explain()<\/b><\/p>\n
Explanation:<\/b><\/p>\n
The explain() operation displays information about how Spark plans to execute a DataFrame query. It can expose logical and physical execution details, including operations such as scans, joins, filters, exchanges, and aggregations. Engineers use execution plans to understand how Spark interprets a query and to identify potential performance issues. For example, an unexpected shuffle or an unsuitable join strategy may become visible in the plan. Examining the plan does not itself execute the entire data transformation as a normal action would. It is primarily a diagnostic and optimization tool.<\/span><\/p>\nQuestion 204. Why can collecting a large DataFrame to the driver be dangerous?<\/b><\/h3>\n
1)<\/b> It can exceed the driver’s available memory<\/span>
\n<\/span>2)<\/b> It automatically deletes executor data<\/span>
\n<\/span>3)<\/b> It disables partitioning<\/span>
\n<\/span>4)<\/b> It converts all columns to strings<\/span><\/p>\nAnswer: 1) It can exceed the driver’s available memory<\/b><\/p>\n
Explanation:<\/b><\/p>\n
The collect() operation transfers the resulting records from distributed executors to the Spark driver. If the result is large, the driver may not have enough memory to hold all returned records, potentially causing an out-of-memory failure. Spark is designed to distribute large datasets across cluster workers, so bringing an entire dataset to one machine defeats that advantage. Engineers should generally use distributed operations for large datasets and reserve collect() for small results when appropriate. Alternatives such as limit(), aggregation, or writing results to storage can avoid unnecessary driver memory consumption.<\/span><\/p>\nQuestion 205. Which approach is generally preferable when Spark already provides a built-in SQL function for a transformation?<\/b><\/h3>\n
1)<\/b> Use the built-in Spark function<\/span>
\n<\/span>2)<\/b> Always create a Python UDF<\/span>
\n<\/span>3)<\/b> Convert the data to CSV first<\/span>
\n<\/span>4)<\/b> Collect all records to the driver<\/span><\/p>\nAnswer: 3) Use the built-in Spark function<\/b><\/p>\n
Explanation:<\/b><\/p>\n
Built-in Spark SQL functions are generally preferable because Spark can understand and optimize them as part of the query plan. They can often benefit from Catalyst optimization and efficient execution mechanisms. A custom Python UDF may introduce additional serialization and execution overhead and can limit some optimization opportunities. For example, if a built-in function already performs a string, date, mathematical, or conditional transformation, using that function is usually more efficient than implementing equivalent Python logic. Custom UDFs remain useful when specialized business logic cannot be expressed through available native functions.<\/span><\/p>\nQuestion 206. What does the <\/b>cache()<\/b> operation primarily do for a Spark DataFrame?<\/b><\/h3>\n
1)<\/b> Requests that computed data be retained for reuse<\/span>
\n<\/span>2)<\/b> Permanently writes the DataFrame to a Delta table<\/span>
\n<\/span>3)<\/b> Changes the DataFrame schema<\/span>
\n<\/span>4)<\/b> Removes all existing partitions<\/span><\/p>\nAnswer: 1) Requests that computed data be retained for reuse<\/b><\/p>\n
Explanation:<\/b><\/p>\n
The cache() operation marks a DataFrame for caching so that its computed representation can be reused by subsequent operations. This can be beneficial when the same DataFrame is accessed multiple times because Spark may avoid repeating the complete upstream computation. Caching is most useful when recomputation is expensive and the cached data can fit reasonably within available resources. Cache is not the same as permanently storing a table. Cached data is associated with the Spark application and can be removed when resources are needed. Engineers should therefore cache selectively rather than caching every intermediate dataset.<\/span><\/p>\nQuestion 207. What is the main difference between <\/b>repartition()<\/b> and <\/b>coalesce()<\/b> in Spark?<\/b><\/h3>\n
1)<\/b> Repartition can increase or decrease partitions through a shuffle, while coalesce is commonly used to reduce partitions with less movement<\/span>
\n<\/span>2)<\/b> Both operations always produce identical execution plans<\/span>
\n<\/span>3)<\/b> Coalesce always creates more partitions than repartition<\/span>
\n<\/span>4)<\/b> Repartition only works with strings<\/span><\/p>\nAnswer: 1) Repartition can increase or decrease partitions through a shuffle, while coalesce is commonly used to reduce partitions with less movement<\/b><\/p>\n
Explanation:<\/b><\/p>\n
repartition() explicitly redistributes data across partitions and generally involves a shuffle. It can be used to increase or decrease the number of partitions and can also partition data based on specified columns. coalesce() is commonly used to reduce the number of partitions while attempting to avoid a full shuffle. Because coalesce may produce uneven partition sizes in some situations, it is not always a replacement for repartition(). The appropriate choice depends on the workload. Engineers often use repartition when balanced redistribution is required and coalesce when efficiently reducing partitions is the main goal.<\/span><\/p>\nQuestion 208. Why can an excessive number of small files hurt a data lake workload?<\/b><\/h3>\n
1)<\/b> They increase file and metadata management overhead<\/span>
\n<\/span>2)<\/b> They guarantee faster queries<\/span>
\n<\/span>3)<\/b> They remove the need for compaction<\/span>
\n<\/span>4)<\/b> They prevent all table reads<\/span><\/p>\nAnswer: 1) They increase file and metadata management overhead<\/b><\/p>\n
Explanation:<\/b><\/p>\n
A large number of small files can create significant overhead for distributed data processing. Query engines must discover, open, and manage many individual files, which can increase scheduling and metadata costs. Small files can also reduce the efficiency of sequential I\/O and make storage management more complicated. File compaction techniques can combine smaller files into larger ones, improving the physical layout of the dataset. The ideal file size depends on the workload and environment, but avoiding excessive fragmentation is an important part of maintaining efficient data lake performance.<\/span><\/p>\nQuestion 209. What is the purpose of table statistics in query optimization?<\/b><\/h3>\n
1)<\/b> They provide information that can help the optimizer choose efficient execution strategies<\/span>
\n<\/span>2)<\/b> They permanently store application secrets<\/span>
\n<\/span>3)<\/b> They replace transaction logs<\/span>
\n<\/span>4)<\/b> They prevent all schema changes<\/span><\/p>\nAnswer: 1) They provide information that can help the optimizer choose efficient execution strategies<\/b><\/p>\n
Explanation:<\/b><\/p>\n
Table statistics provide information about data characteristics that query optimizers can use when selecting execution strategies. Depending on the system, statistics may include details such as row counts, column characteristics, or other metadata useful for estimating query costs. Better estimates can help the optimizer select appropriate join strategies, data access methods, and execution plans. Statistics are not the actual dataset and do not replace transaction logs. Their role is to provide additional information that helps the query engine make better planning decisions before or during execution.<\/span><\/p>\nQuestion 210. What is the purpose of a surrogate key in a dimensional data model?<\/b><\/h3>\n
1)<\/b> To provide a generated identifier for a business entity<\/span>
\n<\/span>2)<\/b> To store encrypted passwords<\/span>
\n<\/span>3)<\/b> To identify a Spark executor<\/span>
\n<\/span>4)<\/b> To replace every source-system column<\/span><\/p>\nAnswer: 1) To provide a generated identifier for a business entity<\/b><\/p>\n
Explanation:<\/b><\/p>\n
A surrogate key is a generated identifier used to uniquely represent an entity in a data warehouse or dimensional model. Unlike a natural key, which originates from a business system, a surrogate key is generally created specifically for the analytical model. Surrogate keys are especially useful when tracking historical versions of dimension records. For example, different versions of the same customer can have separate surrogate keys while retaining a relationship to the underlying business identifier. This approach provides greater control over dimensional relationships and historical tracking across changing source-system data.<\/span><\/p>\nQuestion 211. Which Slowly Changing Dimension strategy maintains historical versions of dimension records?<\/b><\/h3>\n
1)<\/b> Type 2<\/span>
\n<\/span>2)<\/b> Type 0<\/span>
\n<\/span>3)<\/b> Type 1 only<\/span>
\n<\/span>4)<\/b> Type 4 deletion<\/span><\/p>\nAnswer: 2) Type 2<\/b><\/p>\n
Explanation:<\/b><\/p>\n
Slowly Changing Dimension Type 2 preserves historical versions of records rather than simply replacing previous values. When a tracked attribute changes, a new dimension record can be created while the previous version remains available for historical analysis. Effective dates, end dates, current-record indicators, or similar attributes are commonly used to identify the active and historical versions. This allows fact records to be analyzed according to the dimension state that was applicable at a particular point in time. Type 2 is therefore useful when preserving historical changes is an important analytical requirement.<\/span><\/p>\nQuestion 212. What is the main purpose of an effective start timestamp in an SCD Type 2 record?<\/b><\/h3>\n
1)<\/b> To indicate when that version of the record became valid<\/span>
\n<\/span>2)<\/b> To identify the Spark driver<\/span>
\n<\/span>3)<\/b> To specify the cluster’s startup time<\/span>
\n<\/span>4)<\/b> To define the table’s file format<\/span><\/p>\nAnswer: 1) To indicate when that version of the record became valid<\/b><\/p>\n
Explanation:<\/b><\/p>\n
An effective start timestamp identifies the point in time from which a particular version of a dimension record is considered valid. In an SCD Type 2 design, this value works together with an end timestamp or current-record indicator to establish the validity period of each historical version. When an attribute changes, the existing record can be closed and a new version can begin with a new effective start time. This temporal information allows analytical queries to determine which version of a dimension was applicable during a particular business event or reporting period.<\/span><\/p>\nQuestion 213. Which SQL statement is commonly used to combine matching rows with update and insert logic in Delta tables?<\/b><\/h3>\n
1)<\/b> MERGE<\/span>
\n<\/span>2)<\/b> DESCRIBE<\/span>
\n<\/span>3)<\/b> COMMENT<\/span>
\n<\/span>4)<\/b> SHOW<\/span><\/p>\nAnswer: 1) MERGE<\/b><\/p>\n
Explanation:<\/b><\/p>\n
The MERGE statement provides conditional logic for combining source records with a target Delta table. Depending on whether matching conditions are satisfied, the operation can update existing records, insert new records, or apply other supported actions. MERGE is particularly useful for upsert workloads and data integration pipelines where incoming records must be reconciled with existing target data. It can also support historical processing patterns when designed appropriately. Because MERGE operates within Delta Lake’s transactional framework, it provides a structured way to apply coordinated changes to a target table.<\/span><\/p>\nQuestion 214. What is the primary purpose of a Delta Lake transaction log?<\/b><\/h3>\n
1)<\/b> To record changes and table state information<\/span>
\n<\/span>2)<\/b> To store every DataFrame in memory<\/span>
\n<\/span>3)<\/b> To replace cloud object storage<\/span>
\n<\/span>4)<\/b> To contain user passwords<\/span><\/p>\nAnswer: 1) To record changes and table state information<\/b><\/p>\n
Explanation:<\/b><\/p>\n
The Delta Lake transaction log records information about changes made to a Delta table. It enables the system to understand the table’s versions, files, and transactional operations over time. This log supports important capabilities such as ACID transactions, table history, time travel, and consistent reads. The transaction log is separate from the actual data files, which contain the table’s records. By maintaining a structured record of table changes, Delta Lake can reconstruct the correct state of a table for different versions and coordinate concurrent operations reliably.<\/span><\/p>\nQuestion 215. Which Delta Lake capability allows users to query an earlier version of a table?<\/b><\/h3>\n
1)<\/b> Time travel<\/span>
\n<\/span>2)<\/b> Broadcasting<\/span>
\n<\/span>3)<\/b> Predicate pushdown<\/span>
\n<\/span>4)<\/b> Autoscaling<\/span><\/p>\nAnswer: 1) Time travel<\/b><\/p>\n
Explanation:<\/b><\/p>\n
Delta Lake time travel allows users to access historical versions of a table using its recorded transaction history. This capability can be useful for auditing, debugging, reproducing previous results, or recovering information from an earlier table state. Users can reference a specific version or timestamp when querying historical data, subject to the availability of the required underlying files and retention configuration. Time travel does not mean that every historical version is preserved indefinitely. Retention and cleanup operations can eventually remove older data files, which may limit access to very old table states.<\/span><\/p>\nQuestion 216. What is the primary purpose of a Delta Lake checkpoint?<\/b><\/h3>\n
1)<\/b> To compact transaction log information into a more efficient representation<\/span>
\n<\/span>2)<\/b> To delete all table data<\/span>
\n<\/span>3)<\/b> To disable ACID transactions<\/span>
\n<\/span>4)<\/b> To create a Python environment<\/span><\/p>\nAnswer: 1) To compact transaction log information into a more efficient representation<\/b><\/p>\n
Explanation:<\/b><\/p>\n
Delta Lake checkpoints summarize transaction log information into a more efficient form so that the current table state can be reconstructed without replaying every historical log entry from the beginning. This improves the efficiency of reading table metadata as the transaction log grows. Checkpoints work together with transaction log files and are an internal part of Delta Lake’s table management. They do not represent a backup of all data files and should not be confused with a complete table copy. Their primary role is to improve metadata reconstruction and access efficiency.<\/span><\/p>\nQuestion 217. What is the purpose of the <\/b>VACUUM<\/b> operation on a Delta table?<\/b><\/h3>\n
1)<\/b> To remove obsolete data files that are no longer required according to retention rules<\/span>
\n<\/span>2)<\/b> To update every table column automatically<\/span>
\n<\/span>3)<\/b> To create new transaction records for every row<\/span>
\n<\/span>4)<\/b> To enable Python UDF execution<\/span><\/p>\nAnswer: 3) To remove obsolete data files that are no longer required according to retention rules<\/b><\/p>\n
Explanation:<\/b><\/p>\n
VACUUM removes obsolete data files from a Delta table when they are no longer required according to the configured retention period. These files can remain after updates, deletes, or other table changes because older table versions may still reference them. Cleaning up unnecessary files can reduce storage usage, but it can also affect the ability to access sufficiently old table versions through time travel. Therefore, retention settings should be considered carefully before cleanup. VACUUM operates on physical data files and does not simply remove records from the current logical table state.<\/span><\/p>\nQuestion 218. Which feature allows a Delta table to expose inserted, updated, and deleted records as changes for downstream processing?<\/b><\/h3>\n
1)<\/b> Change Data Feed<\/span>
\n<\/span>2)<\/b> Cluster policy<\/span>
\n<\/span>3)<\/b> Table comment<\/span>
\n<\/span>4)<\/b> SQL warehouse scaling<\/span><\/p>\n