mergeByName()<\/span><\/p>\nAnswer: 2) <\/b>unionByName()<\/b><\/p>\n
Explanation:<\/b><\/p>\n
The <\/span>unionByName()<\/span> function combines two DataFrames by matching columns according to their names instead of relying on their physical positions. This is especially useful when datasets have the same logical structure but their columns appear in different orders. By comparison, <\/span>union()<\/span> matches columns by position, which can produce incorrect results when schemas are arranged differently. <\/span>unionByName()<\/span> therefore provides a safer approach when integrating independently generated DataFrames. Data types should still be compatible between corresponding columns. When building production pipelines, using <\/span>unionByName()<\/span> can reduce schema-order-related errors and make data integration logic more predictable.<\/span><\/p>\nQuestion 242. Which option allows <\/b>unionByName()<\/b> to combine DataFrames when some columns are missing from one DataFrame?<\/b><\/h3>\n
1)<\/b> allowMissingColumns=True<\/span><\/p>\n2)<\/b> ignoreMissingColumns=True<\/span><\/p>\n3)<\/b> fillMissingColumns=True<\/span><\/p>\n4)<\/b> resolveMissingColumns=True<\/span><\/p>\nAnswer: 1) <\/b>allowMissingColumns=True<\/b><\/p>\n
Explanation:<\/b><\/p>\n
The <\/span>allowMissingColumns=True<\/span> parameter allows <\/span>unionByName()<\/span> to combine DataFrames even when one DataFrame does not contain every column present in the other. Missing columns are added to the resulting DataFrame and populated with null values where appropriate. This is useful when schemas evolve over time or when different data sources provide slightly different sets of attributes. Without this option, the schemas generally need to contain matching column names. This feature can simplify ingestion pipelines because developers do not always need to manually add missing columns before performing the union operation.<\/span><\/p>\nQuestion 243. What is the primary purpose of the PySpark <\/b>withColumn()<\/b> function?<\/b><\/h3>\n
1)<\/b> Delete an entire DataFrame<\/span><\/p>\n2)<\/b> Add or replace a column<\/span><\/p>\n3)<\/b> Create a database<\/span><\/p>\n4)<\/b> Start a streaming query<\/span><\/p>\nAnswer: 2) Add or replace a column<\/b><\/p>\n
Explanation:<\/b><\/p>\n
The <\/span>withColumn()<\/span> function is used to add a new column to a DataFrame or replace an existing column with a transformed expression. For example, a pipeline can use <\/span>withColumn()<\/span> to calculate a derived value, convert a data type, or apply conditional logic. The function returns a new DataFrame rather than modifying the original DataFrame in place. This makes it useful for constructing transformation pipelines where several column-level operations are applied sequentially. Developers should avoid creating extremely large chains of <\/span>withColumn()<\/span> calls unnecessarily because complex transformation plans can become difficult to optimize and maintain.<\/span><\/p>\nQuestion 244. What is the main purpose of <\/b>selectExpr()<\/b> in PySpark?<\/b><\/h3>\n
1)<\/b> Execute SQL-style expressions while selecting columns<\/span><\/p>\n2)<\/b> Delete selected columns permanently<\/span><\/p>\n3)<\/b> Create a Spark cluster<\/span><\/p>\n4)<\/b> Register a Unity Catalog schema<\/span><\/p>\nAnswer: 1) Execute SQL-style expressions while selecting columns<\/b><\/p>\n
Explanation:<\/b><\/p>\n
selectExpr()<\/span> allows developers to select columns while using SQL expression syntax within a DataFrame transformation. It can perform calculations, aliases, casts, and other SQL-style expressions in a concise form. For example, an expression can rename a column or calculate a new value without requiring separate transformation statements. This makes <\/span>selectExpr()<\/span> convenient when developers are comfortable with SQL syntax but are working within PySpark DataFrame code. It returns a new DataFrame containing the requested expressions. It is particularly useful for compact transformations where SQL expressions clearly communicate the intended operation.<\/span><\/p>\nQuestion 245. Which PySpark expression is commonly used to implement conditional logic with multiple conditions?<\/b><\/h3>\n
1)<\/b> when()<\/span> with <\/span>otherwise()<\/span><\/p>\n2)<\/b> ifElse()<\/span><\/p>\n3)<\/b> conditionCase()<\/span><\/p>\n4)<\/b> switchColumn()<\/span><\/p>\nAnswer: 1) <\/b>when()<\/b> with <\/b>otherwise()<\/b><\/p>\n
Explanation:<\/b><\/p>\n
PySpark uses the <\/span>when()<\/span> expression to define conditional logic and <\/span>otherwise()<\/span> to specify the value returned when none of the defined conditions are satisfied. Multiple <\/span>when()<\/span> clauses can be chained to implement several business rules. For example, a spending column could be converted into categories such as low, medium, or high based on threshold values. This approach keeps conditional transformations inside Spark’s expression framework, allowing Spark to optimize them as part of the query plan. It is generally preferable to collecting data to Python simply to apply ordinary row-level conditional logic.<\/span><\/p>\nQuestion 246. Which PySpark operation is commonly used to convert a column to a different data type?<\/b><\/h3>\n
1)<\/b> convert()<\/span><\/p>\n2)<\/b> cast()<\/span><\/p>\n3)<\/b> datatype()<\/span><\/p>\n4)<\/b> transformType()<\/span><\/p>\nAnswer: 2) <\/b>cast()<\/b><\/p>\n
Explanation:<\/b><\/p>\n
The <\/span>cast()<\/span> operation converts a column expression from one data type to another. For example, a string containing numeric values can be converted into an integer or decimal type before calculations are performed. Explicit casting is important in data engineering because incoming source data may not use the types required by downstream transformations. Developers should choose compatible target types and understand how invalid values are handled. Applying <\/span>cast()<\/span> as part of a DataFrame transformation keeps the conversion inside Spark’s execution plan. Proper data type management can also improve reliability when writing data into strongly typed tables.<\/span><\/p>\nQuestion 247. Which PySpark function is designed to parse a JSON string into a structured column?<\/b><\/h3>\n
1)<\/b> from_json()<\/span><\/p>\n2)<\/b> parse_json_text()<\/span><\/p>\n3)<\/b> json_decode()<\/span><\/p>\n4)<\/b> read_json_column()<\/span><\/p>\nAnswer: 1) <\/b>from_json()<\/b><\/p>\n
Explanation:<\/b><\/p>\n
The <\/span>from_json()<\/span> function parses a JSON-formatted string column and converts it into a structured Spark column such as a struct, array, or map, depending on the supplied schema. This is useful when semi-structured information has been ingested into a table as raw text. Once parsed, individual fields can be referenced using normal DataFrame or SQL expressions. Providing an appropriate schema makes the resulting structure predictable and easier to process. This approach is particularly useful in data engineering pipelines where application logs, API responses, or event payloads are initially stored as JSON strings.<\/span><\/p>\nQuestion 248. What does the PySpark <\/b>to_json()<\/b> function generally do?<\/b><\/h3>\n
1)<\/b> Converts a structured column into a JSON string<\/span><\/p>\n2)<\/b> Converts JSON into a table automatically<\/span><\/p>\n3)<\/b> Deletes JSON records<\/span><\/p>\n4)<\/b> Creates a JSON file without a DataFrame<\/span><\/p>\nAnswer: 1) Converts a structured column into a JSON string<\/b><\/p>\n
Explanation:<\/b><\/p>\n
The <\/span>to_json()<\/span> function serializes a Spark complex data type, such as a struct, array, or map, into a JSON-formatted string. This can be useful when a pipeline needs to send structured information to a system that expects JSON text. For example, nested columns can be converted into a single JSON representation before being passed to an external application or stored in a compatible destination. The function performs serialization as part of Spark’s expression processing. Developers should ensure that the source structure contains values that can be represented appropriately in JSON.<\/span><\/p>\nQuestion 249. Which function can convert Unix epoch time into a human-readable timestamp string in Spark SQL?<\/b><\/h3>\n
1)<\/b> from_unixtime()<\/span><\/p>\n2)<\/b> epoch_to_text()<\/span><\/p>\n3)<\/b> unix_to_date()<\/span><\/p>\n4)<\/b> timestamp_decode()<\/span><\/p>\nAnswer: 1) <\/b>from_unixtime()<\/b><\/p>\n
Explanation:<\/b><\/p>\n
The <\/span>from_unixtime()<\/span> function converts a Unix epoch value, typically representing seconds since January 1, 1970 UTC, into a formatted timestamp string. It is useful when event systems store timestamps as numeric epoch values instead of standard timestamp representations. Converting these values makes them easier to interpret and use in date-based transformations. Data engineers should verify the unit of the source value because some systems provide epoch milliseconds rather than seconds. When necessary, the numeric value must first be adjusted before conversion. Correct timestamp interpretation is essential for accurate filtering, aggregation, and time-based analysis.<\/span><\/p>\nQuestion 250. What is a common purpose of the <\/b>date_trunc()<\/b> function in Spark SQL?<\/b><\/h3>\n
1)<\/b> Remove duplicate rows<\/span><\/p>\n2)<\/b> Truncate a timestamp to a specified time unit<\/span><\/p>\n3)<\/b> Convert JSON into a struct<\/span><\/p>\n4)<\/b> Repartition a DataFrame<\/span><\/p>\nAnswer: 2) Truncate a timestamp to a specified time unit<\/b><\/p>\n
Explanation:<\/b><\/p>\n
The <\/span>date_trunc()<\/span> function truncates a timestamp to a specified time unit, such as hour, day, month, or year. This is useful when timestamps need to be grouped into consistent time periods. For example, transaction timestamps can be truncated to the month so that monthly aggregates can be calculated efficiently. Unlike simply formatting a timestamp for display, truncation produces a value representing the beginning of the selected time period. This makes it useful for grouping, filtering, and joining data based on standardized temporal boundaries within Spark SQL transformations.<\/span><\/p>\nQuestion 251. Which PySpark function can extract a specific pattern from a string using a regular expression?<\/b><\/h3>\n
1)<\/b> regexp_extract()<\/span><\/p>\n2)<\/b> regex_find_column()<\/span><\/p>\n3)<\/b> pattern_select()<\/span><\/p>\n4)<\/b> extract_pattern()<\/span><\/p>\nAnswer: 1) <\/b>regexp_extract()<\/b><\/p>\n
Explanation:<\/b><\/p>\n
The <\/span>regexp_extract()<\/span> function uses a regular expression to extract a matching portion of a string column. It is useful when source data contains embedded values that need to be separated or identified based on predictable text patterns. For example, a data engineer could extract a product code from a larger identifier using a suitable regular expression. The function can return a selected capture group from the pattern. Because regular expressions can become complex, they should be tested against representative data before being deployed in production pipelines. Clear patterns help maintain reliable and understandable transformations.<\/span><\/p>\nQuestion 252. Which Spark SQL operator provides null-safe equality comparison?<\/b><\/h3>\n
1)<\/b> =<\/span><\/p>\n2)<\/b> ==<\/span><\/p>\n3)<\/b> <=><\/span><\/p>\n4)<\/b> ===<\/span><\/p>\nAnswer: 3) <\/b><=><\/b><\/p>\n
Explanation:<\/b><\/p>\n
The <\/span><=><\/span> operator provides null-safe equality comparison in Spark SQL. Ordinary equality comparisons involving null generally do not return true because null represents an unknown value. With <\/span><=><\/span>, two null values are considered equal, while a null compared with a non-null value is considered unequal. This behavior is useful when data quality rules or joins need explicit handling of missing values. Using null-safe comparison can prevent unexpected results in expressions where null values are legitimate and must be treated consistently rather than being propagated as unknown comparison results.<\/span><\/p>\nQuestion 253. Which DataFrame methods are commonly used to test whether a column contains null or non-null values?<\/b><\/h3>\n
1)<\/b> isNull()<\/span> and <\/span>isNotNull()<\/span><\/p>\n2)<\/b> checkNull()<\/span> and <\/span>checkValue()<\/span><\/p>\n3)<\/b> nullTest()<\/span> and <\/span>notNullTest()<\/span><\/p>\n4)<\/b> existsNull()<\/span> and <\/span>existsValue()<\/span><\/p>\nAnswer: 1) <\/b>isNull()<\/b> and <\/b>isNotNull()<\/b><\/p>\n
Explanation:<\/b><\/p>\n
PySpark provides <\/span>isNull()<\/span> and <\/span>isNotNull()<\/span> for identifying missing values in DataFrame columns. These expressions are commonly used with <\/span>filter()<\/span> or <\/span>where()<\/span> when a pipeline needs to retain or remove records based on null status. For example, a data quality process may identify rows where a required customer identifier is missing. These methods explicitly represent null checking and are preferable to ordinary equality comparisons because null does not behave like a normal value. They can also be combined with other conditions to create more detailed validation and cleansing logic.<\/span><\/p>\nQuestion 254. What does the SQL <\/b>COALESCE<\/b> function return?<\/b><\/h3>\n
1)<\/b> The largest value<\/span><\/p>\n2)<\/b> The first non-null value from its arguments<\/span><\/p>\n