View Full Databricks Certified Data Engineer Professional Exam Dumps and Practice Test Dumps

 

Question 361. Which function sorts the elements of an array in ascending order in Spark SQL?

1) array_sort()

2) array_join()

3) array_except()

4) arrays_zip()

Answer: 1) array_sort()

Explanation:

The array_sort() function sorts the elements within an array and returns a new array containing the elements in sorted order. It is useful when array values need to be normalized before comparison, reporting, or downstream processing. The function can work with arrays containing comparable data types and can also be used with custom comparator logic in supported Spark versions. Sorting is performed on the array itself rather than changing the order of rows in the DataFrame. This makes array_sort() particularly useful when working with nested or semi-structured data where each row contains an array that must be consistently ordered.

Question 362. Which function sorts an array using Spark SQL array sorting semantics?

1) sort_array()

2) array_union()

3) flatten()

4) sequence()

Answer: 1) sort_array()

Explanation:

The sort_array() function sorts the elements of an array in ascending order by default. It also supports descending order through an optional Boolean parameter. This function is useful when an array needs deterministic ordering for analysis or comparison. For example, an array containing [5, 2, 8] can be sorted into [2, 5, 8]. Unlike DataFrame-level sorting, sort_array() operates independently on the array value within each row. It is therefore appropriate when the requirement concerns the internal ordering of array elements rather than the ordering of records returned by a query.

Question 363. Which function determines whether two arrays have at least one common element?

1) arrays_overlap()

2) array_intersect()

3) array_except()

4) array_join()

Answer: 1) arrays_overlap()

Explanation:

The arrays_overlap() function checks whether two arrays contain at least one common element. It returns a Boolean result when the overlap can be determined. This is useful for scenarios such as checking whether two users share an interest, whether two products have common categories, or whether two records contain matching tags. The function compares array contents rather than requiring the arrays to have the same length or ordering. Using arrays_overlap() can simplify logic that would otherwise require exploding both arrays and performing additional joins or aggregations.

Question 364. Which function returns the common elements between two arrays?

1) array_intersect()

2) array_union()

3) array_except()

4) arrays_overlap()

Answer: 1) array_intersect()

Explanation:

The array_intersect() function returns an array containing elements that occur in both input arrays. It is useful when processing nested data where two sets of values need to be compared directly. For example, if one array contains customer interests and another contains campaign categories, array_intersect() can identify the shared values. The result represents the intersection rather than simply indicating whether an intersection exists. This differs from arrays_overlap(), which primarily answers the yes-or-no question of whether common elements are present.

Question 365. Which function combines the distinct elements from two arrays?

1) array_union()

2) array_intersect()

3) array_except()

4) array_join()

Answer: 1) array_union()

Explanation:

The array_union() function combines the elements from two arrays and removes duplicate values from the resulting array. It is useful when two collections of categories, identifiers, permissions, or tags need to be combined into one distinct set. The function operates at the array level, so there is no need to explode the arrays into individual rows simply to combine their values. Because duplicate elements are eliminated, the resulting array represents the union of the input collections rather than a simple concatenation that preserves repeated values.

Question 366. Which function returns elements that exist in one array but not in another?

1) array_except()

2) array_union()

3) array_intersect()

4) arrays_overlap()

Answer: 1) array_except()

Explanation:

The array_except() function returns the distinct elements that are present in the first array but absent from the second array. It is useful for identifying differences between collections. For example, it can determine which permissions a user has that are not included in a required permission set. The operation is directional, meaning changing the order of the two input arrays can change the result. This makes array_except() useful for comparison tasks involving categories, tags, identifiers, or other repeated values stored as arrays.

Question 367. Which function combines array elements into a single string using a delimiter?

1) array_join()

2) array_sort()

3) flatten()

4) arrays_overlap()

Answer: 1) array_join()

Explanation:

The array_join() function converts the elements of an array into a single string, placing a specified delimiter between the elements. For example, an array containing [“red”, “blue”, “green”] can be converted into a string such as red,blue,green. An optional replacement string can also be provided for null elements. This function is useful when nested array data must be displayed in reports, exported to text-based formats, or combined into a human-readable representation without first exploding the array into multiple rows.

Question 368. Which function converts an array of arrays into a single flattened array?

1) flatten()

2) array_join()

3) sequence()

4) array_union()

Answer: 1) flatten()

Explanation:

The flatten() function reduces one level of nested arrays by combining the inner arrays into a single array. For example, an input such as [[1, 2], [3, 4]] becomes [1, 2, 3, 4]. This is useful when source data contains nested collections and downstream processing expects a simpler array structure. flatten() works on array-of-array structures and avoids manually exploding and reconstructing the data. It should be distinguished from functions that combine two independent arrays, because its primary purpose is removing one layer of array nesting.

Question 369. Which function generates an array containing a sequence of values between specified boundaries?

1) sequence()

2) flatten()

3) array_sort()

4) array_join()

Answer: 1) sequence()

Explanation:

The sequence() function generates an array containing a sequence of values between a starting value and an ending value. It can work with numeric values and date or timestamp values, with an optional step controlling the increment. This makes it useful for generating date ranges, numeric ranges, or other regularly spaced values directly inside Spark SQL expressions. For example, a sequence can support calendar-based processing where each row needs an array representing a range of dates. This approach can reduce the need to construct such sequences outside Spark.

Question 370. Which higher-order function applies an expression to every element of an array?

1) transform()

2) filter()

3) exists()

4) forall()

Answer: 1) transform()

Explanation:

The transform() higher-order function applies a specified lambda expression to every element of an array and returns a new array containing the transformed values. It is useful when nested array data needs to be modified without exploding the array into rows. For example, numeric values can be multiplied by a factor, strings can be converted to another form, or fields within complex elements can be extracted. Because the operation remains within the array structure, transform() provides a concise way to perform element-level transformations while preserving one output array for each input row.

Question 371. Which higher-order function returns only array elements that satisfy a specified condition?

1) filter()

2) transform()

3) exists()

4) aggregate()

Answer: 1) filter()

Explanation:

The higher-order filter() function evaluates a condition against each element of an array and returns a new array containing only the elements that satisfy that condition. This should not be confused with the DataFrame filter() operation, which filters rows. The higher-order version operates inside an array value. It is useful for removing unwanted nested values, such as retaining only positive numbers or selecting tags that meet a particular condition. This allows nested data to be filtered without first converting every array element into a separate DataFrame row.

Question 372. Which higher-order function checks whether at least one array element satisfies a condition?

1) exists()

2) forall()

3) filter()

4) transform()

Answer: 1) exists()

Explanation:

The exists() higher-order function tests an array using a Boolean condition and determines whether at least one element satisfies that condition. The result is a Boolean value rather than a filtered array. This is useful when a pipeline needs to make a decision based on nested data, such as checking whether a transaction list contains a high-value transaction or whether a collection includes a particular category. exists() avoids unnecessary array expansion and aggregation because it directly expresses the requirement that at least one matching element must be present.

Question 373. Which higher-order function verifies that every array element satisfies a condition?

1) forall()

2) exists()

3) filter()

4) transform()

Answer: 1) forall()

Explanation:

The forall() higher-order function evaluates a condition for every element in an array and determines whether all elements satisfy that condition. It returns a Boolean result. This is useful for validation scenarios involving nested data, such as verifying that all recorded measurements are within an acceptable range or that every item in an array meets a required rule. Unlike exists(), which succeeds when at least one element matches, forall() requires the condition to hold across the complete collection. It therefore provides a concise way to express array-wide validation logic.

Question 374. Which higher-order function can aggregate the elements of an array into a single result?

1) aggregate()

2) transform()

3) exists()

4) array_union()

Answer: 1) aggregate()

Explanation:

The higher-order aggregate() function processes the elements of an array and combines them into a single accumulated result. It uses an initial value and an aggregation expression to update the accumulator as each element is processed. An optional finishing expression can then transform the accumulated value. This is useful when an array contains nested values that need custom calculations without being exploded into rows. For example, an array can be processed to calculate a custom total, build a structured result, or apply logic that is more specialized than a standard aggregation function.

Question 375. Which function transforms the keys of a map while preserving its map structure?

1) transform_keys()

2) transform_values()

3) map_filter()

4) map_zip_with()

Answer: 1) transform_keys()

Explanation:

The transform_keys() higher-order function applies an expression to the keys of a map and returns a new map with transformed keys. The associated values remain associated with their corresponding entries. This is useful when map keys need normalization, formatting, or conversion before further processing. For example, string keys can be changed to lowercase or transformed according to a business rule. Because the operation works directly on the map structure, it avoids manually converting map entries into rows and reconstructing the map afterward.

Question 376. Which function transforms the values of a map while retaining the original keys?

1) transform_values()

2) transform_keys()

3) map_filter()

4) map_zip_with()

Answer: 1) transform_values()

Explanation:

The transform_values() function applies a transformation to each value in a map while preserving its keys. This is useful when map values require cleaning, conversion, calculation, or standardization. For example, numeric values can be multiplied by a factor or string values can be normalized while the original identifiers remain unchanged. The function is particularly helpful with nested and semi-structured datasets because it modifies map contents directly. It avoids the additional work of converting map entries into rows, applying a transformation, and rebuilding the map afterward.

Question 377. Which higher-order function removes map entries that do not satisfy a condition?

1) map_filter()

2) transform_keys()

3) transform_values()

4) zip_with()

Answer: 1) map_filter()

Explanation:

The map_filter() function evaluates a condition against the key-value pairs of a map and returns a new map containing only entries that satisfy the condition. This is useful when nested map data contains values that should be retained only when they meet a business or quality rule. For example, a map can be filtered to retain only entries with values above a specified threshold. Since the operation works directly on the map, it provides a compact alternative to exploding map entries, filtering rows, and reconstructing the nested structure.

Question 378. Which function combines two arrays element by element using a lambda expression?

1) zip_with()

2) arrays_zip()

3) array_union()

4) array_intersect()

Answer: 1) zip_with()

Explanation:

The zip_with() function combines two arrays element by element and applies a specified lambda expression to corresponding elements. It is useful when two arrays contain related values that must be processed together. For example, one array might contain quantities while another contains prices, allowing a lambda expression to calculate a value for each corresponding pair. The function produces an array containing the results of those pairwise operations. This makes zip_with() particularly useful for custom calculations involving parallel arrays without requiring the arrays to be exploded into rows.

Question 379. Which function combines two arrays into an array of paired values without requiring a custom lambda expression?

1) arrays_zip()

2) zip_with()

3) array_intersect()

4) flatten()

Answer: 1) arrays_zip()

Explanation:

The arrays_zip() function combines multiple arrays by position and returns an array of structs containing the corresponding elements. It is useful when several arrays represent parallel attributes that need to remain aligned. For example, arrays of product IDs and prices can be zipped so that each resulting struct contains the ID and corresponding price. Unlike zip_with(), it does not require a custom lambda expression to define how corresponding elements should be combined. This makes arrays_zip() convenient when the primary requirement is preserving positional relationships between multiple arrays.

Question 380. Which function is most appropriate for applying a custom expression to corresponding elements from two arrays?

1) zip_with()

2) arrays_zip()

3) array_sort()

4) array_except()

**Answer: 1) zip_with()

Explanation:

zip_with() is designed for element-by-element processing of two arrays using a custom lambda expression. It is especially useful when corresponding positions represent related data and the desired output requires calculation or conditional logic. For example, quantities from one array can be multiplied by prices from another array to produce an array of line totals. arrays_zip() can also align arrays, but it returns structs rather than directly applying a custom operation to each pair. Therefore, when the requirement explicitly involves custom pairwise processing, zip_with() is the appropriate choice.