RESTORE<\/span><\/p>\nAnswer: 2) OPTIMIZE<\/b><\/p>\n
Explanation:<\/b><\/p>\n
The <\/span>OPTIMIZE<\/span> command reorganizes files in a Delta table to improve data layout and reduce the performance impact of having many small files. Small-file accumulation can increase metadata overhead and require more file operations during queries. OPTIMIZE can compact files and, when applicable, use data layout techniques to improve query efficiency. <\/span>VACUUM<\/span> removes obsolete files rather than compacting active ones. <\/span>DESCRIBE HISTORY<\/span> reports table operations, while <\/span>RESTORE<\/span> returns a table to a previous state. Therefore, OPTIMIZE is the operation associated with file compaction and data-layout optimization.<\/span><\/p>\nQuestion 22. Which Delta Lake capability allows a table to be restored to an earlier version?<\/b><\/h3>\n
1)<\/b> RESTORE<\/span>
\n<\/span>2)<\/b> CLONE<\/span>
\n<\/span>3)<\/b> OPTIMIZE<\/span>
\n<\/span>4)<\/b> VACUUM<\/span><\/p>\nAnswer: 1) RESTORE<\/b><\/p>\n
Explanation:<\/b><\/p>\n
The Delta Lake <\/span>RESTORE<\/span> operation can return a table to a previous version or timestamp. This capability is useful when an incorrect update, delete, or other modification needs to be reversed. The transaction history identifies available versions that can be restored. <\/span>OPTIMIZE<\/span> reorganizes table files, while <\/span>VACUUM<\/span> removes obsolete files according to retention settings. <\/span>CLONE<\/span> creates a copy of a table rather than directly reverting the existing table. Restore operations should be planned carefully because downstream consumers may be affected when the table’s current state changes.<\/span><\/p>\nQuestion 23. Which SQL operation is commonly used to create a temporary view from a DataFrame?<\/b><\/h3>\n
1)<\/b> createOrReplaceTempView()<\/span>
\n<\/span>2)<\/b> saveAsTable()<\/span>
\n<\/span>3)<\/b> DROP VIEW<\/span>
\n<\/span>4)<\/b> DESCRIBE TABLE<\/span><\/p>\nAnswer: 1) <\/b>createOrReplaceTempView()<\/b><\/p>\n
Explanation:<\/b><\/p>\n
The <\/span>createOrReplaceTempView()<\/span> method registers a DataFrame as a temporary SQL view within the current Spark session. Once registered, SQL queries can reference the view by name. This is useful when data engineers want to combine DataFrame-based transformations with SQL operations without permanently storing the result as a table. <\/span>saveAsTable()<\/span> persists data as a table, while <\/span>DROP VIEW<\/span> removes a view and <\/span>DESCRIBE TABLE<\/span> returns metadata. Temporary views are generally session-scoped, making them suitable for intermediate processing and exploratory transformations.<\/span><\/p>\nQuestion 24. Which Spark transformation is commonly used to combine rows from two DataFrames based on matching columns?<\/b><\/h3>\n
1)<\/b> join()<\/span>
\n<\/span>2)<\/b> cache()<\/span>
\n<\/span>3)<\/b> count()<\/span>
\n<\/span>4)<\/b> collect()<\/span><\/p>\nAnswer: 1) <\/b>join()<\/b><\/p>\n
Explanation:<\/b><\/p>\n
The <\/span>join()<\/span> transformation combines records from two DataFrames according to a specified join condition. Common join types include inner, left, right, full, and cross joins. Joins are fundamental to data engineering because information is often distributed across multiple datasets that need to be combined using keys. <\/span>cache()<\/span> controls persistence behavior, while <\/span>count()<\/span> and <\/span>collect()<\/span> are actions that trigger computation. Choosing an appropriate join type and condition is important because an inefficient join can create excessive data movement and increase processing time in distributed Spark workloads.<\/span><\/p>\nQuestion 25. Which Spark operation can be used to remove duplicate rows from a DataFrame?<\/b><\/h3>\n
1)<\/b> distinct()<\/span>
\n<\/span>2)<\/b> explode()<\/span>
\n<\/span>3)<\/b> union()<\/span>
\n<\/span>4)<\/b> pivot()<\/span><\/p>\nAnswer: 1) <\/b>distinct()<\/b><\/p>\n
Explanation:<\/b><\/p>\n
The <\/span>distinct()<\/span> transformation returns a DataFrame containing unique rows, removing duplicate records based on the complete row contents. It is useful when duplicate records have entered a pipeline and the business requirement is to retain only unique combinations of column values. <\/span>explode()<\/span> expands array or map elements into separate rows, while <\/span>union()<\/span> combines compatible DataFrames. <\/span>pivot()<\/span> reshapes grouped data into columns. Because <\/span>distinct()<\/span> can require data redistribution across partitions, data engineers should consider its computational cost when processing very large datasets.<\/span><\/p>\nQuestion 26. Which Spark function is useful for replacing null values with specified values?<\/b><\/h3>\n
1)<\/b> fillna()<\/span>
\n<\/span>2)<\/b> explode()<\/span>
\n<\/span>3)<\/b> pivot()<\/span>
\n<\/span>4)<\/b> repartition()<\/span><\/p>\nAnswer: 1) <\/b>fillna()<\/b><\/p>\n
Explanation:<\/b><\/p>\n
The <\/span>fillna()<\/span> DataFrame operation can replace null values with specified replacement values. This is commonly used during data-cleaning pipelines when missing values have a defined business treatment. Replacement values can be supplied for particular columns or according to supported data types. <\/span>explode()<\/span> is used to expand arrays or maps, <\/span>pivot()<\/span> reshapes grouped data, and <\/span>repartition()<\/span> changes the number or distribution of partitions. Null handling should be based on the meaning of the missing data rather than automatically replacing every null with an arbitrary value.<\/span><\/p>\nQuestion 27. What is the primary purpose of repartitioning a Spark DataFrame?<\/b><\/h3>\n
1)<\/b> To change the distribution or number of partitions<\/span>
\n<\/span>2)<\/b> To permanently delete duplicate records<\/span>
\n<\/span>3)<\/b> To encrypt the DataFrame<\/span>
\n<\/span>4)<\/b> To create a database schema<\/span><\/p>\nAnswer: 1) To change the distribution or number of partitions<\/b><\/p>\n
Explanation:<\/b><\/p>\n
Repartitioning changes how data is distributed across Spark partitions. The <\/span>repartition()<\/span> operation can increase or decrease the number of partitions and can also redistribute records according to specified columns. This can be useful for balancing workloads or preparing data for operations that benefit from a particular partitioning strategy. Repartitioning generally involves a shuffle, which can be expensive for large datasets. It does not encrypt data, create database schemas, or automatically remove duplicates. Partition strategy should therefore be selected based on workload characteristics and downstream processing requirements.<\/span><\/p>\nQuestion 28. Which operation can reduce the number of partitions without requiring a full shuffle in common cases?<\/b><\/h3>\n
1)<\/b> coalesce()<\/span>
\n<\/span>2)<\/b> join()<\/span>
\n<\/span>3)<\/b> groupBy()<\/span>
\n<\/span>4)<\/b> distinct()<\/span><\/p>\nAnswer: 1) <\/b>coalesce()<\/b><\/p>\n
Explanation:<\/b><\/p>\n
The <\/span>coalesce()<\/span> operation can reduce the number of partitions while generally avoiding a full shuffle. This makes it useful when a DataFrame has more partitions than necessary, such as after filtering removes a substantial amount of data. Because it can reduce partition count with less data movement, it may be more efficient than using <\/span>repartition()<\/span> solely for decreasing partitions. However, it may result in uneven partition sizes depending on the data. Join, groupBy, and distinct operations serve different purposes and commonly involve data redistribution.<\/span><\/p>\nQuestion 29. Which Spark function is commonly used to expand each element of an array into a separate row?<\/b><\/h3>\n
1)<\/b> explode()<\/span>
\n<\/span>2)<\/b> collect()<\/span>
\n<\/span>3)<\/b> count()<\/span>
\n<\/span>4)<\/b> coalesce()<\/span><\/p>\nAnswer: 1) <\/b>explode()<\/b><\/p>\n
Explanation:<\/b><\/p>\n
The <\/span>explode()<\/span> function transforms an array or map column so that its elements can be represented as separate rows. For example, if one record contains an array of product identifiers, <\/span>explode()<\/span> can produce one output row for each product. This is particularly useful when semi-structured data contains nested collections that need to be normalized for further analysis. <\/span>collect()<\/span> retrieves results to the driver, <\/span>count()<\/span> returns a record count, and <\/span>coalesce()<\/span> changes partition behavior. Therefore, <\/span>explode()<\/span> is the appropriate function for expanding array elements into individual rows.<\/span><\/p>\nQuestion 30. Which Spark operation is generally used to combine two DataFrames with compatible schemas by appending their rows?<\/b><\/h3>\n
1)<\/b> union()<\/span>
\n<\/span>2)<\/b> join()<\/span>
\n<\/span>3)<\/b> intersect()<\/span>
\n<\/span>4)<\/b> subtract()<\/span><\/p>\nAnswer: 1) <\/b>union()<\/b><\/p>\n
Explanation:<\/b><\/p>\n
The <\/span>union()<\/span> operation combines the rows of two DataFrames when their schemas are compatible. Instead of matching records through a key, union appends the rows from one dataset to the rows of another. This is useful when multiple datasets have the same logical structure and need to be processed as a single dataset. <\/span>join()<\/span> combines columns based on matching conditions, while <\/span>intersect()<\/span> and <\/span>subtract()<\/span> perform set-oriented comparisons. Data engineers should verify column order and compatible data types before using union to avoid unexpected results.<\/span><\/p>\nQuestion 31. Which Databricks feature is commonly used to define reusable, declarative data transformation pipelines?<\/b><\/h3>\n
1)<\/b> Delta Live Tables<\/span>
\n<\/span>2)<\/b> Secret Scope<\/span>
\n<\/span>3)<\/b> SQL Warehouse<\/span>
\n<\/span>4)<\/b> Cluster Policy<\/span><\/p>\nAnswer: 1) Delta Live Tables<\/b><\/p>\n
Explanation:<\/b><\/p>\n
Delta Live Tables, now part of Databricks Lakeflow Declarative Pipelines terminology, provides a framework for defining data transformation pipelines declaratively. Engineers specify the desired datasets and transformation logic while the platform manages aspects of pipeline execution, dependencies, and operational processing. This approach can simplify the development and maintenance of reliable data pipelines. Secret scopes focus on credentials, SQL Warehouses provide SQL compute, and cluster policies govern compute configurations. Declarative pipeline frameworks are particularly useful when teams want data dependencies and transformation logic to be represented clearly within managed pipeline definitions.<\/span><\/p>\nQuestion 32. What is the purpose of defining dependencies between tasks in a Databricks workflow?<\/b><\/h3>\n
1)<\/b> To control task execution order<\/span>
\n<\/span>2)<\/b> To encrypt task output<\/span>
\n<\/span>3)<\/b> To increase database storage automatically<\/span>
\n<\/span>4)<\/b> To remove Delta transaction logs<\/span><\/p>\nAnswer: 1) To control task execution order<\/b><\/p>\n
Explanation:<\/b><\/p>\n
Task dependencies define relationships between workflow tasks so that downstream tasks can execute after required upstream tasks have completed successfully or according to configured conditions. For example, an ingestion task can be completed before a transformation task begins, followed by a validation or reporting task. This dependency structure helps organize complex data pipelines and prevents tasks from running before their required inputs are available. Dependencies do not encrypt output, manage storage directly, or remove transaction logs. They are primarily an orchestration mechanism for controlling workflow execution.<\/span><\/p>\nQuestion 33. Which feature allows a Databricks job to retry a failed task automatically?<\/b><\/h3>\n
1)<\/b> Task retry configuration<\/span>
\n<\/span>2)<\/b> Delta time travel<\/span>