Pass Your Exam With 100% Verified Associate-Developer-Apache-Spark Exam Questions [Q20-Q44]

Q20. Which of the following statements about the differences between actions and transformations is correct?

Actions are evaluated lazily, while transformations are not evaluated lazily.

Actions generate RDDs, while transformations do not.

Actions do not send results to the driver, while transformations do.

Actions can be queued for delayed execution, while transformations can only be processed immediately.

Actions can trigger Adaptive Query Execution, while transformation cannot.

Q21. Which of the following statements about Spark’s execution hierarchy is correct?

In Spark’s execution hierarchy, a job may reach over multiple stage boundaries.

In Spark’s execution hierarchy, manifests are one layer above jobs.

In Spark’s execution hierarchy, a stage comprises multiple jobs.

In Spark’s execution hierarchy, executors are the smallest unit.

In Spark’s execution hierarchy, tasks are one layer above slots.

Q22. Which of the following code blocks creates a new DataFrame with two columns season and wind_speed_ms where column season is of data type string and column wind_speed_ms is of data type double?

spark.DataFrame({“season”: [“winter”,”summer”], “wind_speed_ms”: [4.5, 7.5]})

spark.createDataFrame([(“summer”, 4.5), (“winter”, 7.5)], [“season”, “wind_speed_ms”])

1. from pyspark.sql import types as T
2. spark.createDataFrame(((“summer”, 4.5), (“winter”, 7.5)), T.StructType([T.StructField(“season”,

CharType()), T.StructField(“season”, T.DoubleType())]))

spark.newDataFrame([(“summer”, 4.5), (“winter”, 7.5)], [“season”, “wind_speed_ms”])

spark.createDataFrame({“season”: [“winter”,”summer”], “wind_speed_ms”: [4.5, 7.5]})

Explanation
spark.createDataFrame([(“summer”, 4.5), (“winter”, 7.5)], [“season”, “wind_speed_ms”]) Correct. This command uses the Spark Session’s createDataFrame method to create a new DataFrame. Notice how rows, columns, and column names are passed in here: The rows are specified as a Python list. Every entry in the list is a new row. Columns are specified as Python tuples (for example (“summer”, 4.5)). Every column is one entry in the tuple.
The column names are specified as the second argument to createDataFrame(). The documentation (link below) shows that “when schema is a list of column names, the type of each column will be inferred from data” (the first argument). Since values 4.5 and 7.5 are both float variables, Spark will correctly infer the double type for column wind_speed_ms. Given that all values in column
“season” contain only strings, Spark will cast the column appropriately as string.
Find out more about SparkSession.createDataFrame() via the link below.
spark.newDataFrame([(“summer”, 4.5), (“winter”, 7.5)], [“season”, “wind_speed_ms”]) No, the SparkSession does not have a newDataFrame method.
from pyspark.sql import types as T
spark.createDataFrame(((“summer”, 4.5), (“winter”, 7.5)), T.StructType([T.StructField(“season”,
T.CharType()), T.StructField(“season”, T.DoubleType())]))
No. pyspark.sql.types does not have a CharType type. See link below for available data types in Spark.
spark.createDataFrame({“season”: [“winter”,”summer”], “wind_speed_ms”: [4.5, 7.5]}) No, this is not correct Spark syntax. If you have considered this option to be correct, you may have some experience with Python’s pandas package, in which this would be correct syntax. To create a Spark DataFrame from a Pandas DataFrame, you can simply use spark.createDataFrame(pandasDf) where pandasDf is the Pandas DataFrame.
Find out more about Spark syntax options using the examples in the documentation for SparkSession.createDataFrame linked below.
spark.DataFrame({“season”: [“winter”,”summer”], “wind_speed_ms”: [4.5, 7.5]}) No, the Spark Session (indicated by spark in the code above) does not have a DataFrame method.
More info: pyspark.sql.SparkSession.createDataFrame – PySpark 3.1.1 documentation and Data Types – Spark 3.1.2 Documentation Static notebook | Dynamic notebook: See test 1

Q23. Which of the following describes a way for resizing a DataFrame from 16 to 8 partitions in the most efficient way?

Use operation DataFrame.repartition(8) to shuffle the DataFrame and reduce the number of partitions.

Use operation DataFrame.coalesce(8) to fully shuffle the DataFrame and reduce the number of partitions.

Use a narrow transformation to reduce the number of partitions.

Use a wide transformation to reduce the number of partitions.
Use operation DataFrame.coalesce(0.5) to halve the number of partitions in the DataFrame.

Q24. Which of the following describes slots?

Slots are dynamically created and destroyed in accordance with an executor’s workload.

To optimize I/O performance, Spark stores data on disk in multiple slots.

A Java Virtual Machine (JVM) working as an executor can be considered as a pool of slots for task execution.

A slot is always limited to a single core.
Slots are the communication interface for executors and are used for receiving commands and sending results to the driver.

Q25. Which of the following code blocks displays various aggregated statistics of all columns in DataFrame transactionsDf, including the standard deviation and minimum of values in each column?

transactionsDf.summary()

transactionsDf.agg(“count”, “mean”, “stddev”, “25%”, “50%”, “75%”, “min”)

transactionsDf.summary(“count”, “mean”, “stddev”, “25%”, “50%”, “75%”, “max”).show()

transactionsDf.agg(“count”, “mean”, “stddev”, “25%”, “50%”, “75%”, “min”).show()

transactionsDf.summary().show()

Q26. Which of the following code blocks returns a new DataFrame with the same columns as DataFrame transactionsDf, except for columns predError and value which should be removed?

transactionsDf.drop([“predError”, “value”])

transactionsDf.drop(“predError”, “value”)

transactionsDf.drop(col(“predError”), col(“value”))

transactionsDf.drop(predError, value)

transactionsDf.drop(“predError & value”)

Q27. Which of the following describes Spark’s way of managing memory?

Spark uses a subset of the reserved system memory.

Storage memory is used for caching partitions derived from DataFrames.

As a general rule for garbage collection, Spark performs better on many small objects than few big objects.

Disabling serialization potentially greatly reduces the memory footprint of a Spark application.

Spark’s memory usage can be divided into three categories: Execution, transaction, and storage.

Q28. Which of the following code blocks returns a single row from DataFrame transactionsDf?
Full DataFrame transactionsDf:
1.+————-+———+—–+——-+———+—-+
2.|transactionId|predError|value|storeId|productId| f|
3.+————-+———+—–+——-+———+—-+
4.| 1| 3| 4| 25| 1|null|
5.| 2| 6| 7| 2| 2|null|
6.| 3| 3| null| 25| 3|null|
7.| 4| null| null| 3| 2|null|
8.| 5| null| null| null| 2|null|
9.| 6| 3| 2| 25| 2|null|
10.+————-+———+—–+——-+———+—-+

transactionsDf.where(col(“storeId”).between(3,25))

transactionsDf.filter((col(“storeId”)!=25) | (col(“productId”)==2))

transactionsDf.filter(col(“storeId”)==25).select(“predError”,”storeId”).distinct()

transactionsDf.select(“productId”, “storeId”).where(“storeId == 2 OR storeId != 25”)

transactionsDf.where(col(“value”).isNull()).select(“productId”, “storeId”).distinct()

Q29. The code block displayed below contains an error. The code block is intended to join DataFrame itemsDf with the larger DataFrame transactionsDf on column itemId. Find the error.
Code block:
transactionsDf.join(itemsDf, “itemId”, how=”broadcast”)

The syntax is wrong, how= should be removed from the code block.

The join method should be replaced by the broadcast method.

Spark will only perform the broadcast operation if this behavior has been enabled on the Spark cluster.

The larger DataFrame transactionsDf is being broadcasted, rather than the smaller DataFrame itemsDf.

broadcast is not a valid join type.

Q30. The code block displayed below contains an error. When the code block below has executed, it should have divided DataFrame transactionsDf into 14 parts, based on columns storeId and transactionDate (in this order). Find the error.
Code block:
transactionsDf.coalesce(14, (“storeId”, “transactionDate”))

The parentheses around the column names need to be removed and .select() needs to be appended to the code block.

Operator coalesce needs to be replaced by repartition, the parentheses around the column names need to be removed, and .count() needs to be appended to the code block.
(Correct)

Operator coalesce needs to be replaced by repartition, the parentheses around the column names need to be removed, and .select() needs to be appended to the code block.

Operator coalesce needs to be replaced by repartition and the parentheses around the column names need to be replaced by square brackets.

Operator coalesce needs to be replaced by repartition.

Q31. The code block shown below should read all files with the file ending .png in directory path into Spark.
Choose the answer that correctly fills the blanks in the code block to accomplish this.
spark.__1__.__2__(__3__).option(__4__, “*.png”).__5__(path)

1. read()
2. format
3. “binaryFile”
4. “recursiveFileLookup”
5. load

1. read
2. format
3. “binaryFile”
4. “pathGlobFilter”
5. load

1. read
2. format
3. binaryFile
4. pathGlobFilter
5. load

1. open
2. format
3. “image”
4. “fileType”
5. open

1. open
2. as
3. “binaryFile”
4. “pathGlobFilter”
5. load

Q32. The code block displayed below contains an error. The code block is intended to return all columns of DataFrame transactionsDf except for columns predError, productId, and value. Find the error.
Excerpt of DataFrame transactionsDf:
transactionsDf.select(~col(“predError”), ~col(“productId”), ~col(“value”))

The select operator should be replaced by the drop operator and the arguments to the drop operator should be column names predError, productId and value wrapped in the col operator so they should be expressed like drop(col(predError), col(productId), col(value)).

The select operator should be replaced with the deselect operator.

The column names in the select operator should not be strings and wrapped in the col operator, so they should be expressed like select(~col(predError), ~col(productId), ~col(value)).

The select operator should be replaced by the drop operator.

The select operator should be replaced by the drop operator and the arguments to the drop operator should be column names predError, productId and value as strings.
(Correct)

Q33. Which of the following describes characteristics of the Spark driver?

The Spark driver requests the transformation of operations into DAG computations from the worker nodes.

If set in the Spark configuration, Spark scales the Spark driver horizontally to improve parallel processing performance.

The Spark driver processes partitions in an optimized, distributed fashion.

In a non-interactive Spark application, the Spark driver automatically creates the SparkSession object.

The Spark driver’s responsibility includes scheduling queries for execution on worker nodes.

Q34. Which of the following options describes the responsibility of the executors in Spark?

The executors accept jobs from the driver, analyze those jobs, and return results to the driver.

The executors accept tasks from the driver, execute those tasks, and return results to the cluster manager.

The executors accept tasks from the driver, execute those tasks, and return results to the driver.

The executors accept tasks from the cluster manager, execute those tasks, and return results to the driver.

The executors accept jobs from the driver, plan those jobs, and return results to the cluster manager.

Q35. The code block shown below should return a copy of DataFrame transactionsDf with an added column cos.
This column should have the values in column value converted to degrees and having the cosine of those converted values taken, rounded to two decimals. Choose the answer that correctly fills the blanks in the code block to accomplish this.
Code block:
transactionsDf.__1__(__2__, round(__3__(__4__(__5__)),2))

1. withColumn
2. col(“cos”)
3. cos
4. degrees
5. transactionsDf.value

1. withColumnRenamed
2. “cos”
3. cos
4. degrees
5. “transactionsDf.value”

1. withColumn
2. “cos”
3. cos
4. degrees
5. transactionsDf.value

1. withColumn
2. col(“cos”)
3. cos
4. degrees
5. col(“value”)
E
. 1. withColumn
2. “cos”
3. degrees
4. cos
5. col(“value”)

Q36. The code block shown below should store DataFrame transactionsDf on two different executors, utilizing the executors’ memory as much as possible, but not writing anything to disk. Choose the answer that correctly fills the blanks in the code block to accomplish this.
1.from pyspark import StorageLevel
2.transactionsDf.__1__(StorageLevel.__2__).__3__

1. cache
2. MEMORY_ONLY_2
3. count()

1. persist
2. DISK_ONLY_2
3. count()

1. persist
2. MEMORY_ONLY_2
3. select()

1. cache
2. DISK_ONLY_2
3. count()

1. persist
2. MEMORY_ONLY_2
3. count()

Q37. The code block displayed below contains an error. The code block should count the number of rows that have a predError of either 3 or 6. Find the error.
Code block:
transactionsDf.filter(col(‘predError’).in([3, 6])).count()

The number of rows cannot be determined with the count() operator.

Instead of filter, the select method should be used.

The method used on column predError is incorrect.

Instead of a list, the values need to be passed as single arguments to the in operator.

Numbers 3 and 6 need to be passed as string variables.

Q38. In which order should the code blocks shown below be run in order to assign articlesDf a DataFrame that lists all items in column attributes ordered by the number of times these items occur, from most to least often?
Sample of DataFrame articlesDf:
1.+——+—————————–+——————-+
2.|itemId|attributes |supplier |
3.+——+—————————–+——————-+
4.|1 |[blue, winter, cozy] |Sports Company Inc.|
5.|2 |[red, summer, fresh, cooling]|YetiX |
6.|3 |[green, summer, travel] |Sports Company Inc.|
7.+——+—————————–+——————-+

1. articlesDf = articlesDf.groupby(“col”)
2. articlesDf = articlesDf.select(explode(col(“attributes”)))
3. articlesDf = articlesDf.orderBy(“count”).select(“col”)
4. articlesDf = articlesDf.sort(“count”,ascending=False).select(“col”)
5. articlesDf = articlesDf.groupby(“col”).count()

4, 5

2, 5, 3

5, 2

2, 3, 4

2, 5, 4

Q39. The code block shown below should write DataFrame transactionsDf to disk at path csvPath as a single CSV file, using tabs (t characters) as separators between columns, expressing missing values as string n/a, and omitting a header row with column names. Choose the answer that correctly fills the blanks in the code block to accomplish this.
transactionsDf.__1__.write.__2__(__3__, ” “).__4__.__5__(csvPath)

1. coalesce(1)
2. option
3. “sep”
4. option(“header”, True)
5. path

1. coalesce(1)
2. option
3. “colsep”
4. option(“nullValue”, “n/a”)
5. path

1. repartition(1)
2. option
3. “sep”
4. option(“nullValue”, “n/a”)
5. csv
(Correct)

1. csv
2. option
3. “sep”
4. option(“emptyValue”, “n/a”)
5. path
* 1. repartition(1)
2. mode
3. “sep”
4. mode(“nullValue”, “n/a”)
5. csv

Q40. Which of the following is a viable way to improve Spark’s performance when dealing with large amounts of data, given that there is only a single application running on the cluster?

Increase values for the properties spark.default.parallelism and spark.sql.shuffle.partitions

Decrease values for the properties spark.default.parallelism and spark.sql.partitions

Increase values for the properties spark.sql.parallelism and spark.sql.partitions

Increase values for the properties spark.sql.parallelism and spark.sql.shuffle.partitions

Increase values for the properties spark.dynamicAllocation.maxExecutors, spark.default.parallelism, and spark.sql.shuffle.partitions

Q41. Which of the following describes the conversion of a computational query into an execution plan in Spark?

Spark uses the catalog to resolve the optimized logical plan.

The catalog assigns specific resources to the optimized memory plan.

The executed physical plan depends on a cost optimization from a previous stage.

Depending on whether DataFrame API or SQL API are used, the physical plan may differ.

The catalog assigns specific resources to the physical plan.

Q42. Which of the following is a characteristic of the cluster manager?

Each cluster manager works on a single partition of data.

The cluster manager receives input from the driver through the SparkContext.

The cluster manager does not exist in standalone mode.

The cluster manager transforms jobs into DAGs.

In client mode, the cluster manager runs on the edge node.

Q43. Which of the following code blocks returns all unique values of column storeId in DataFrame transactionsDf?

transactionsDf[“storeId”].distinct()

transactionsDf.select(“storeId”).distinct()
(Correct)

transactionsDf.filter(“storeId”).distinct()

transactionsDf.select(col(“storeId”).distinct())

transactionsDf.distinct(“storeId”)

Q44. Which of the following statements about broadcast variables is correct?

Broadcast variables are serialized with every single task.

Broadcast variables are commonly used for tables that do not fit into memory.

Broadcast variables are immutable.

Broadcast variables are occasionally dynamically updated on a per-task basis.

Broadcast variables are local to the worker node and not shared across the cluster.

Free valid test braindumps

Pass Your Exam With 100% Verified Associate-Developer-Apache-Spark Exam Questions [Q20-Q44]

Leave a Reply Cancel reply